linux/fs/squashfs
Sean Purcell 87bf54bb43 squashfs: Add zstd support
Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
while decompressing twice as fast as zlib. For SquashFS in particular,
it can decompress as fast as lzo and lz4. It also has the flexibility
to turn down the compression ratio for faster compression times.

The compression benchmark is run on the file tree from the SquashFS archive
found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the
default block size (128 KB) and and various compression algorithms/levels.
xz and zstd are also benchmarked with 256 KB blocks. The decompression
benchmark times how long it takes to `tar` the file tree into `/dev/null`.
See the benchmark file in the upstream zstd source repository located under
`contrib/linux-kernel/squashfs-benchmark.sh` [2] for details.

I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of RAM, and a SSD.

| Method         | Ratio | Compression MB/s | Decompression MB/s |
|----------------|-------|------------------|--------------------|
| gzip           |  2.92 |               15 |                128 |
| lzo            |  2.64 |              9.5 |                217 |
| lz4            |  2.12 |               94 |                218 |
| xz             |  3.43 |              5.5 |                 35 |
| xz 256 KB      |  3.53 |              5.4 |                 40 |
| zstd 1         |  2.71 |               96 |                210 |
| zstd 5         |  2.93 |               69 |                198 |
| zstd 10        |  3.01 |               41 |                225 |
| zstd 15        |  3.13 |             11.4 |                224 |
| zstd 16 256 KB |  3.24 |              8.1 |                210 |

This patch was written by Sean Purcell <me@seanp.xyz>, but I will be
taking over the submission process.

[1] http://releases.ubuntu.com/16.10/
[2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh

zstd source repository: https://github.com/facebook/zstd

Signed-off-by: Sean Purcell <me@seanp.xyz>
Signed-off-by: Nick Terrell <terrelln@fb.com>
Signed-off-by: Chris Mason <clm@fb.com>
Acked-by: Phillip Lougher <phillip@squashfs.org.uk>
2017-09-08 19:33:25 -07:00
..
block.c block,fs: untangle fs.h and blk_types.h 2016-11-01 09:43:26 -06:00
cache.c mm, fs: remove remaining PAGE_CACHE_* and page_cache_{get,release} usage 2016-04-04 10:41:08 -07:00
decompressor_multi_percpu.c Squashfs: Generalise paging handling in the decompressors 2013-11-20 03:59:01 +00:00
decompressor_multi.c Squashfs: Check stream is not NULL in decompressor_multi.c 2013-11-20 03:59:20 +00:00
decompressor_single.c Squashfs: Generalise paging handling in the decompressors 2013-11-20 03:59:01 +00:00
decompressor.c squashfs: Add zstd support 2017-09-08 19:33:25 -07:00
decompressor.h squashfs: Add zstd support 2017-09-08 19:33:25 -07:00
dir.c romfs, squashfs: switch to ->iterate_shared() 2016-05-09 11:41:15 -04:00
export.c VFS: normal filesystems (and lustre): d_inode() annotations 2015-04-15 15:06:57 -04:00
file_cache.c Squashfs: Restructure squashfs_readpage() 2013-11-20 03:59:07 +00:00
file_direct.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
file.c mm, fs: remove remaining PAGE_CACHE_* and page_cache_{get,release} usage 2016-04-04 10:41:08 -07:00
fragment.c Squashfs: Fix sanity check patches on big-endian systems 2011-05-29 10:03:09 +01:00
id.c Squashfs: Fix sanity check patches on big-endian systems 2011-05-29 10:03:09 +01:00
inode.c vfs: Remove {get,set,remove}xattr inode operations 2016-10-07 21:48:36 -04:00
Kconfig squashfs: Add zstd support 2017-09-08 19:33:25 -07:00
lz4_wrapper.c fs/pstore: fs/squashfs: change usage of LZ4 to work with new LZ4 version 2017-02-24 17:46:57 -08:00
lzo_wrapper.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
Makefile squashfs: Add zstd support 2017-09-08 19:33:25 -07:00
namei.c vfs: Remove {get,set,remove}xattr inode operations 2016-10-07 21:48:36 -04:00
page_actor.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
page_actor.h mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
squashfs_fs_i.h fs: cleanup slight list_entry abuse 2015-06-23 18:01:59 -04:00
squashfs_fs_sb.h Squashfs: Generalise paging handling in the decompressors 2013-11-20 03:59:01 +00:00
squashfs_fs.h squashfs: Add zstd support 2017-09-08 19:33:25 -07:00
squashfs.h fs/squashfs/squashfs.h: replace pr_warning by pr_warn 2014-06-04 16:53:52 -07:00
super.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
symlink.c vfs: remove ".readlink = generic_readlink" assignments 2016-12-09 16:45:04 +01:00
xattr_id.c Squashfs: update email address 2011-05-26 10:49:11 +01:00
xattr.c xattr_handler: pass dentry and inode as separate arguments of ->get() 2016-04-10 20:48:24 -04:00
xattr.h vfs: Remove {get,set,remove}xattr inode operations 2016-10-07 21:48:36 -04:00
xz_wrapper.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
zlib_wrapper.c mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros 2016-04-04 10:41:08 -07:00
zstd_wrapper.c squashfs: Add zstd support 2017-09-08 19:33:25 -07:00