| SQUASHFS 4.3 - A squashed read-only filesystem for Linux | |
| Copyright 2002-2014 Phillip Lougher <phillip@lougher.demon.co.uk> | |
| Released under the GPL licence (version 2 or later). | |
| Welcome to Squashfs version 4.3. Please read the README-4.3 and CHANGES files | |
| for details of changes. | |
| Squashfs is a highly compressed read-only filesystem for Linux. | |
| It uses either gzip/xz/lzo/lz4 compression to compress both files, inodes | |
| and directories. Inodes in the system are very small and all blocks are | |
| packed to minimise data overhead. Block sizes greater than 4K are supported | |
| up to a maximum of 1Mbytes (default block size 128K). | |
| Squashfs is intended for general read-only filesystem use, for archival | |
| use (i.e. in cases where a .tar.gz file may be used), and in constrained | |
| block device/memory systems (e.g. embedded systems) where low overhead is | |
| needed. | |
| 1. SQUASHFS OVERVIEW | |
| -------------------- | |
| 1. Data, inodes and directories are compressed. | |
| 2. Squashfs stores full uid/gids (32 bits), and file creation time. | |
| 3. In theory files up to 2^64 bytes are supported. In theory filesystems can | |
| be up to 2^64 bytes. | |
| 4. Inode and directory data are highly compacted, and packed on byte | |
| boundaries. Each compressed inode is on average 8 bytes in length | |
| (the exact length varies on file type, i.e. regular file, directory, | |
| symbolic link, and block/char device inodes have different sizes). | |
| 5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K). | |
| Using 128K blocks achieves greater compression ratios than the normal | |
| 4K block size. | |
| 6. File duplicates are detected and removed. | |
| 7. Filesystems can be compressed with gzip, xz (lzma2), lzo or lz4 | |
| compression algorithms. | |
| 1.1 Extended attributes (xattrs) | |
| -------------------------------- | |
| Squashfs filesystems now have extended attribute support. The | |
| extended attribute implementation has the following features: | |
| 1. Layout can store up to 2^48 bytes of compressed xattr data. | |
| 2. Number of xattrs per inode unlimited. | |
| 3. Total size of xattr data per inode 2^48 bytes of compressed data. | |
| 4. Up to 4 Gbytes of data per xattr value. | |
| 5. Inline and out-of-line xattr values supported for higher performance | |
| in xattr scanning (listxattr & getxattr), and to allow xattr value | |
| de-duplication. | |
| 6. Both whole inode xattr duplicate detection and individual xattr value | |
| duplicate detection supported. These can obviously nest, file C's | |
| xattrs can be a complete duplicate of file B, and file B's xattrs | |
| can be a partial duplicate of file A. | |
| 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." | |
| etc. characters to be eliminated and more concisely stored. | |
| 8. Support for files, directories, symbolic links, device nodes, fifos | |
| and sockets. | |
| Extended attribute support is in 2.6.35 and later kernels. Filesystems | |
| with extended attributes can be mounted on 2.6.29 and later kernels, the | |
| extended attributes will be ignored with a warning. | |
| 2. USING SQUASHFS | |
| ----------------- | |
| Squashfs filesystems should be mounted with 'mount' with the filesystem type | |
| 'squashfs'. If the filesystem is on a block device, the filesystem can be | |
| mounted directly, e.g. | |
| %mount -t squashfs /dev/sda1 /mnt | |
| Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt". | |
| If the squashfs filesystem has been written to a file, the loopback device | |
| can be used to mount it (loopback support must be in the kernel), e.g. | |
| %mount -t squashfs image /mnt -o loop | |
| Will mount the squashfs filesystem in the file "image" under | |
| the directory "/mnt". | |
| 3. MKSQUASHFS | |
| ------------- | |
| 3.1 Mksquashfs options and overview | |
| ----------------------------------- | |
| As squashfs is a read-only filesystem, the mksquashfs program must be used to | |
| create populated squashfs filesystems. | |
| SYNTAX:./mksquashfs source1 source2 ... dest [options] [-e list of exclude | |
| dirs/files] | |
| Filesystem build options: | |
| -comp <comp> select <comp> compression | |
| Compressors available: | |
| gzip (default) | |
| lzo | |
| lz4 | |
| xz | |
| -b <block_size> set data block to <block_size>. Default 128 Kbytes | |
| Optionally a suffix of K or M can be given to specify | |
| Kbytes or Mbytes respectively | |
| -no-exports don't make the filesystem exportable via NFS | |
| -no-sparse don't detect sparse files | |
| -no-xattrs don't store extended attributes | |
| -xattrs store extended attributes (default) | |
| -noI do not compress inode table | |
| -noD do not compress data blocks | |
| -noF do not compress fragment blocks | |
| -noX do not compress extended attributes | |
| -no-fragments do not use fragments | |
| -always-use-fragments use fragment blocks for files larger than block size | |
| -no-duplicates do not perform duplicate checking | |
| -all-root make all files owned by root | |
| -force-uid uid set all file uids to uid | |
| -force-gid gid set all file gids to gid | |
| -nopad do not pad filesystem to a multiple of 4K | |
| -keep-as-directory if one source directory is specified, create a root | |
| directory containing that directory, rather than the | |
| contents of the directory | |
| Filesystem filter options: | |
| -p <pseudo-definition> Add pseudo file definition | |
| -pf <pseudo-file> Add list of pseudo file definitions | |
| -sort <sort_file> sort files according to priorities in <sort_file>. One | |
| file or dir with priority per line. Priority -32768 to | |
| 32767, default priority 0 | |
| -ef <exclude_file> list of exclude dirs/files. One per line | |
| -wildcards Allow extended shell wildcards (globbing) to be used in | |
| exclude dirs/files | |
| -regex Allow POSIX regular expressions to be used in exclude | |
| dirs/files | |
| Filesystem append options: | |
| -noappend do not append to existing filesystem | |
| -root-becomes <name> when appending source files/directories, make the | |
| original root become a subdirectory in the new root | |
| called <name>, rather than adding the new source items | |
| to the original root | |
| Mksquashfs runtime options: | |
| -version print version, licence and copyright message | |
| -exit-on-error treat normally ignored errors as fatal | |
| -recover <name> recover filesystem data using recovery file <name> | |
| -no-recovery don't generate a recovery file | |
| -info print files written to filesystem | |
| -no-progress don't display the progress bar | |
| -progress display progress bar when using the -info option | |
| -processors <number> Use <number> processors. By default will use number of | |
| processors available | |
| -mem <size> Use <size> physical memory. Currently set to 1922M | |
| Optionally a suffix of K, M or G can be given to specify | |
| Kbytes, Mbytes or Gbytes respectively | |
| Miscellaneous options: | |
| -root-owned alternative name for -all-root | |
| -noInodeCompression alternative name for -noI | |
| -noDataCompression alternative name for -noD | |
| -noFragmentCompression alternative name for -noF | |
| -noXattrCompression alternative name for -noX | |
| -Xhelp print compressor options for selected compressor | |
| Compressors available and compressor specific options: | |
| gzip (default) | |
| -Xcompression-level <compression-level> | |
| <compression-level> should be 1 .. 9 (default 9) | |
| -Xwindow-size <window-size> | |
| <window-size> should be 8 .. 15 (default 15) | |
| -Xstrategy strategy1,strategy2,...,strategyN | |
| Compress using strategy1,strategy2,...,strategyN in turn | |
| and choose the best compression. | |
| Available strategies: default, filtered, huffman_only, | |
| run_length_encoded and fixed | |
| lzo | |
| -Xalgorithm <algorithm> | |
| Where <algorithm> is one of: | |
| lzo1x_1 | |
| lzo1x_1_11 | |
| lzo1x_1_12 | |
| lzo1x_1_15 | |
| lzo1x_999 (default) | |
| -Xcompression-level <compression-level> | |
| <compression-level> should be 1 .. 9 (default 8) | |
| Only applies to lzo1x_999 algorithm | |
| lz4 | |
| -Xhc | |
| Compress using LZ4 High Compression | |
| xz | |
| -Xbcj filter1,filter2,...,filterN | |
| Compress using filter1,filter2,...,filterN in turn | |
| (in addition to no filter), and choose the best compression. | |
| Available filters: x86, arm, armthumb, powerpc, sparc, ia64 | |
| -Xdict-size <dict-size> | |
| Use <dict-size> as the XZ dictionary size. The dictionary size | |
| can be specified as a percentage of the block size, or as an | |
| absolute value. The dictionary size must be less than or equal | |
| to the block size and 8192 bytes or larger. It must also be | |
| storable in the xz header as either 2^n or as 2^n+2^(n+1). | |
| Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K | |
| etc. | |
| Source1 source2 ... are the source directories/files containing the | |
| files/directories that will form the squashfs filesystem. If a single | |
| directory is specified (i.e. mksquashfs source output_fs) the squashfs | |
| filesystem will consist of that directory, with the top-level root | |
| directory corresponding to the source directory. | |
| If multiple source directories or files are specified, mksquashfs will merge | |
| the specified sources into a single filesystem, with the root directory | |
| containing each of the source files/directories. The name of each directory | |
| entry will be the basename of the source path. If more than one source | |
| entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where | |
| xxx is the original name. | |
| To make this clear, take two example directories. Source directory | |
| "/home/phillip/test" contains "file1", "file2" and "dir1". | |
| Source directory "goodies" contains "goodies1", "goodies2" and "goodies3". | |
| usage example 1: | |
| %mksquashfs /home/phillip/test output_fs | |
| This will generate a squashfs filesystem with root entries | |
| "file1", "file2" and "dir1". | |
| example 2: | |
| %mksquashfs /home/phillip/test goodies output_fs | |
| This will create a squashfs filesystem with the root containing | |
| entries "test" and "goodies" corresponding to the source | |
| directories "/home/phillip/test" and "goodies". | |
| example 3: | |
| %mksquashfs /home/phillip/test goodies test output_fs | |
| This is the same as the previous example, except a third | |
| source directory "test" has been specified. This conflicts | |
| with the first directory named "test" and will be renamed "test_1". | |
| Multiple sources allow filesystems to be generated without needing to | |
| copy all source files into a common directory. This simplifies creating | |
| filesystems. | |
| The -keep-as-directory option can be used when only one source directory | |
| is specified, and you wish the root to contain that directory, rather than | |
| the contents of the directory. For example: | |
| example 4: | |
| %mksquashfs /home/phillip/test output_fs -keep-as-directory | |
| This is the same as example 1, except for -keep-as-directory. | |
| This will generate a root directory containing directory "test", | |
| rather than the "test" directory contents "file1", "file2" and "dir1". | |
| The Dest argument is the destination where the squashfs filesystem will be | |
| written. This can either be a conventional file or a block device. If the file | |
| doesn't exist it will be created, if it does exist and a squashfs | |
| filesystem exists on it, mksquashfs will append. The -noappend option will | |
| write a new filesystem irrespective of whether an existing filesystem is | |
| present. | |
| 3.2 Changing compression algorithm and compression specific options | |
| ------------------------------------------------------------------- | |
| By default Mksquashfs will compress using the gzip compression | |
| algorithm. This algorithm offers a good trade-off between compression | |
| ratio, and memory and time taken to decompress. | |
| Squashfs also supports LZ4, LZO and XZ (LZMA2) compression. LZO offers worse | |
| compression ratio than gzip, but is faster to decompress. XZ offers better | |
| compression ratio than gzip, but at the expense of greater memory and time | |
| to decompress (and significantly more time to compress). LZ4 is similar | |
| to LZO, but, support for it is not yet in the mainline kernel, and so | |
| its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs | |
| as an archival system like tar. | |
| If you're not building the squashfs-tools and kernel from source, then | |
| the tools and kernel may or may not have been built with support for LZ4, LZO or | |
| XZ compression. The compression algorithms supported by the build of | |
| Mksquashfs can be found by typing mksquashfs without any arguments. The | |
| compressors available are displayed at the end of the help message, e.g. | |
| Compressors available and compressor specific options: | |
| gzip (default) | |
| -Xcompression-level <compression-level> | |
| <compression-level> should be 1 .. 9 (default 9) | |
| -Xwindow-size <window-size> | |
| <window-size> should be 8 .. 15 (default 15) | |
| -Xstrategy strategy1,strategy2,...,strategyN | |
| Compress using strategy1,strategy2,...,strategyN in turn | |
| and choose the best compression. | |
| Available strategies: default, filtered, huffman_only, | |
| run_length_encoded and fixed | |
| lzo | |
| -Xalgorithm <algorithm> | |
| Where <algorithm> is one of: | |
| lzo1x_1 | |
| lzo1x_1_11 | |
| lzo1x_1_12 | |
| lzo1x_1_15 | |
| lzo1x_999 (default) | |
| -Xcompression-level <compression-level> | |
| <compression-level> should be 1 .. 9 (default 8) | |
| Only applies to lzo1x_999 algorithm | |
| lz4 | |
| -Xhc | |
| Compress using LZ4 High Compression | |
| xz | |
| -Xbcj filter1,filter2,...,filterN | |
| Compress using filter1,filter2,...,filterN in turn | |
| (in addition to no filter), and choose the best compression. | |
| Available filters: x86, arm, armthumb, powerpc, sparc, ia64 | |
| -Xdict-size <dict-size> | |
| Use <dict-size> as the XZ dictionary size. The dictionary size | |
| can be specified as a percentage of the block size, or as an | |
| absolute value. The dictionary size must be less than or equal | |
| to the block size and 8192 bytes or larger. It must also be | |
| storable in the xz header as either 2^n or as 2^n+2^(n+1). | |
| Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K | |
| etc. | |
| If the compressor offers compression specific options (all the compressors now | |
| have compression specific options except the deprecated lzma1 compressor) | |
| then these options are also displayed (.i.e. in the above XZ is shown with two | |
| compression specific options). The compression specific options are, obviously, | |
| specific to the compressor in question, and the compressor documentation and | |
| web sites should be consulted to understand their behaviour. In general | |
| the Mksquashfs compression defaults for each compressor are optimised to | |
| give the best performance for each compressor, where what constitutes | |
| best depends on the compressor. For gzip/xz best means highest compression, | |
| for LZO/LZ4 best means a tradeoff between compression and (de)-compression | |
| overhead (LZO/LZ4 by definition are intended for weaker processors). | |
| 3.3 Changing global compression defaults used in mksquashfs | |
| ----------------------------------------------------------- | |
| There are a large number of options that can be used to control the | |
| compression in mksquashfs. By and large the defaults are the most | |
| optimum settings and should only be changed in exceptional circumstances! | |
| Note, this does not apply to the block size, increasing the block size | |
| from the default of 128Kbytes will increase compression (especially | |
| for the xz compressor) and should increase I/O performance too. However, | |
| a block size of greater than 128Kbytes may increase latency in certain | |
| cases (where the filesystem contains lots of fragments, and no locality | |
| of reference is observed). For this reason the block size default is | |
| configured to the less optimal 128Kbytes. Users should experiment | |
| with 256Kbyte sizes or above. | |
| The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression | |
| and -noFragmentCompression) can be used to force mksquashfs to not compress | |
| inodes/directories, data and fragments respectively. Giving all options | |
| generates an uncompressed filesystem. | |
| The -no-fragments tells mksquashfs to not generate fragment blocks, and rather | |
| generate a filesystem similar to a Squashfs 1.x filesystem. It will of course | |
| still be a Squashfs 4.0 filesystem but without fragments, and so it won't be | |
| mountable on a Squashfs 1.x system. | |
| The -always-use-fragments option tells mksquashfs to always generate | |
| fragments for files irrespective of the file length. By default only small | |
| files less than the block size are packed into fragment blocks. The ends of | |
| files which do not fit fully into a block, are NOT by default packed into | |
| fragments. To illustrate this, a 100K file has an initial 64K block and a 36K | |
| remainder. This 36K remainder is not packed into a fragment by default. This | |
| is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a | |
| disk head seek is needed to seek to the initial file data and another disk seek | |
| is need to seek to the fragment block. Specify this option if you want file | |
| remainders to be packed into fragment blocks. Doing so may increase the | |
| compression obtained BUT at the expense of I/O speed. | |
| The -no-duplicates option tells mksquashfs to not check the files being | |
| added to the filesystem for duplicates. This can result in quicker filesystem | |
| generation and appending although obviously compression will suffer badly if | |
| there is a lot of duplicate files. | |
| The -b option allows the block size to be selected, both "K" and "M" postfixes | |
| are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or | |
| 1M bytes. | |
| 3.4 Specifying the UIDs/GIDs used in the filesystem | |
| --------------------------------------------------- | |
| By default files in the generated filesystem inherit the UID and GID ownership | |
| of the original file. However, mksquashfs provides a number of options which | |
| can be used to override the ownership. | |
| The options -all-root and -root-owned (both do exactly the same thing) force all | |
| file uids/gids in the generated Squashfs filesystem to be root. This allows | |
| root owned filesystems to be built without root access on the host machine. | |
| The "-force-uid uid" option forces all files in the generated Squashfs | |
| filesystem to be owned by the specified uid. The uid can be specified either by | |
| name (i.e. "root") or by number. | |
| The "-force-gid gid" option forces all files in the generated Squashfs | |
| filesystem to be group owned by the specified gid. The gid can be specified | |
| either by name (i.e. "root") or by number. | |
| 3.5 Excluding files from the filesystem | |
| --------------------------------------- | |
| The -e and -ef options allow files/directories to be specified which are | |
| excluded from the output filesystem. The -e option takes the exclude | |
| files/directories from the command line, the -ef option takes the | |
| exlude files/directories from the specified exclude file, one file/directory | |
| per line. | |
| Two styles of exclude file matching are supported: basic exclude matching, and | |
| extended wildcard matching. Basic exclude matching is a legacy feature | |
| retained for backwards compatibility with earlier versions of Mksquashfs. | |
| Extended wildcard matching should be used in preference. | |
| 3.5.1 Basic exclude matching | |
| ---------------------------- | |
| Each exclude file is treated as an exact match of a file/directory in | |
| the source directories. If an exclude file/directory is absolute (i.e. | |
| prefixed with /, ../, or ./) the entry is treated as absolute, however, if an | |
| exclude file/directory is relative, it is treated as being relative to each of | |
| the sources in turn, i.e. | |
| %mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3 | |
| Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1, | |
| /tmp/source1/out/ex3 and source2/out/ex3. | |
| 3.5.2 Extended exclude file handling | |
| ------------------------------------ | |
| Extended exclude file matching treats each exclude file as a wildcard or | |
| regex expression. To enable wildcard matching specify the -wildcards | |
| option, and to enable regex matching specify the -regex option. In most | |
| cases the -wildcards option should be used rather than -regex because wildcard | |
| matching behaviour is significantly easier to understand! | |
| In addition to wildcards/regex expressions, exclude files can be "anchored" or | |
| "non-anchored". An anchored exclude is one which matches from the root of the | |
| directory and nowhere else, a non-anchored exclude matches anywhere. For | |
| example given the directory hierarchy "a/b/c/a/b", the anchored exclude | |
| "a/b" will match "a/b" at the root of the directory hierarchy, but | |
| it will not match the "/a/b" sub-directory within directory "c", whereas a | |
| non-anchored exclude would. | |
| A couple of examples should make this clearer. | |
| Anchored excludes | |
| 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' | |
| Exclude all files matching "*.gz" in the top level directory "test". | |
| 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' | |
| Exclude all files beginning with "example" inside directories called | |
| "Test" or "test", that occur inside any top level directory. | |
| Using extended wildcards, negative matching is also possible. | |
| 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' | |
| Exclude all files matching "*.gz" in top level directory "test", | |
| except those with "data" in the name. | |
| Non-anchored excludes | |
| By default excludes match from the top level directory, but it is | |
| often useful to exclude a file matching anywhere in the source directories. | |
| For this non-anchored excludes can be used, specified by pre-fixing the | |
| exclude with "...". | |
| Examples: | |
| 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' | |
| Exclude files matching "*.gz" anywhere in the source directories. | |
| For example this will match "example.gz", "test/example.gz", and | |
| "test/test/example.gz". | |
| 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' | |
| Exclude files matching "*.gz" inside directories called "Test" or | |
| "test" that occur anywhere in the source directories. | |
| Again, using extended wildcards, negative matching is also possible. | |
| 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' | |
| Exclude all files matching "*.gz" anywhere in the source directories, | |
| except those with "data" in the name. | |
| 3.5.3 Exclude files summary | |
| --------------------------- | |
| The -e and -ef exclude options are usefully used in archiving the entire | |
| filesystem, where it is wished to avoid archiving /proc, and the filesystem | |
| being generated, i.e. | |
| %mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh | |
| Multiple -ef options can be specified on the command line, and the -ef | |
| option can be used in conjuction with the -e option. | |
| 3.6 Appending to squashfs filesystems | |
| ------------------------------------- | |
| Running squashfs with the destination directory containing an existing | |
| filesystem will add the source items to the existing filesystem. By default, | |
| the source items are added to the existing root directory. | |
| To make this clear... An existing filesystem "image" contains root entries | |
| "old1", and "old2". Source directory "/home/phillip/test" contains "file1", | |
| "file2" and "dir1". | |
| example 1: | |
| %mksquashfs /home/phillip/test image | |
| Will create a new "image" with root entries "old1", "old2", "file1", "file2" and | |
| "dir1" | |
| example 2: | |
| %mksquashfs /home/phillip/test image -keep-as-directory | |
| Will create a new "image" with root entries "old1", "old2", and "test". | |
| As shown in the previous section, for single source directories | |
| '-keep-as-directory' adds the source directory rather than the | |
| contents of the directory. | |
| example 3: | |
| %mksquashfs /home/phillip/test image -keep-as-directory -root-becomes | |
| original-root | |
| Will create a new "image" with root entries "original-root", and "test". The | |
| '-root-becomes' option specifies that the original root becomes a subdirectory | |
| in the new root, with the specified name. | |
| The append option with file duplicate detection, means squashfs can be | |
| used as a simple versioning archiving filesystem. A squashfs filesystem can | |
| be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 | |
| source will create a filesystem with the two source trees, but only the | |
| changed files will take extra room, the unchanged files will be detected as | |
| duplicates. | |
| 3.7 Appending recovery file feature | |
| ----------------------------------- | |
| Recovery files are created when appending to existing Squashfs | |
| filesystems. This allows the original filesystem to be recovered | |
| if Mksquashfs aborts unexpectedly (i.e. power failure). | |
| The recovery files are called squashfs_recovery_xxx_yyy, where | |
| "xxx" is the name of the filesystem being appended to, and "yyy" is a | |
| number to guarantee filename uniqueness (the PID of the parent Mksquashfs | |
| process). | |
| Normally if Mksquashfs exits correctly the recovery file is deleted to | |
| avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" | |
| option can be used to recover the filesystem, giving the previously | |
| created recovery file as a parameter, i.e. | |
| mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 | |
| The writing of the recovery file can be disabled by specifying the | |
| "-no-recovery" option. | |
| 3.8 Pseudo file support | |
| ----------------------- | |
| Mksquashfs supports pseudo files, these allow fake files, directories, character | |
| and block devices to be specified and added to the Squashfs filesystem being | |
| built, rather than requiring them to be present in the source directories. | |
| This, for example, allows device nodes to be added to the filesystem without | |
| requiring root access. | |
| Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation. | |
| Dynamic pseudo files allow files to be dynamically created when Mksquashfs | |
| is run, their contents being the result of running a command or piece of | |
| shell script. The modifiy operation allows the mode/uid/gid of an existing | |
| file in the source filesystem to be modified. | |
| Two Mksquashfs options are supported, -p allows one pseudo file to be specified | |
| on the command line, and -pf allows a pseudo file to be specified containing a | |
| list of pseduo definitions, one per line. | |
| 3.8.1. Creating a dynamic file | |
| ------------------------------ | |
| Pseudo definition | |
| Filename f mode uid gid command | |
| mode is the octal mode specifier, similar to that expected by chmod. | |
| uid and gid can be either specified as a decimal number, or by name. | |
| command can be an executable or a piece of shell script, and it is executed | |
| by running "/bin/sh -c command". The stdout becomes the contents of | |
| "Filename". | |
| Examples: | |
| Running a basic command | |
| ----------------------- | |
| /somedir/dmesg f 444 root root dmesg | |
| creates a file "/somedir/dmesg" containing the output from dmesg. | |
| Executing shell script | |
| ---------------------- | |
| RELEASE f 444 root root \ | |
| if [ ! -e /tmp/ver ]; then \ | |
| echo 0 > /tmp/ver; \ | |
| fi; \ | |
| ver=`cat /tmp/ver`; \ | |
| ver=$((ver +1)); \ | |
| echo $ver > /tmp/ver; \ | |
| echo -n `cat /tmp/release`; \ | |
| echo "-dev #"$ver `date` "Build host" `hostname` | |
| Creates a file RELEASE containing the release name, date, build host, and | |
| an incrementing version number. The incrementing version is a side-effect | |
| of executing the shell script, and ensures every time Mksquashfs is run a | |
| new version number is used without requiring any other shell scripting. | |
| The above example also shows that commands can be split across multiple lines | |
| using "\". Obviously as the script will be presented to the shell as a single | |
| line, a semicolon is need to separate individual shell commands within the | |
| shell script. | |
| Reading from a device (or fifo/named socket) | |
| -------------------------------------------- | |
| input f 444 root root dd if=/dev/sda1 bs=1024 count=10 | |
| Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs | |
| given a device, fifo, or named socket will place that special file within the | |
| Squashfs filesystem, the above allows input from these special files to be | |
| captured and placed in the Squashfs filesystem. | |
| 3.8.2. Creating a block or character device | |
| ------------------------------------------- | |
| Pseudo definition | |
| Filename type mode uid gid major minor | |
| Where type is either | |
| b - for block devices, and | |
| c - for character devices | |
| mode is the octal mode specifier, similar to that expected by chmod. | |
| uid and gid can be either specified as a decimal number, or by name. | |
| For example: | |
| /dev/chr_dev c 666 root root 100 1 | |
| /dev/blk_dev b 666 0 0 200 200 | |
| creates a character device "/dev/chr_dev" with major:minor 100:1 and | |
| a block device "/dev/blk_dev" with major:minor 200:200, both with root | |
| uid/gid and a mode of rw-rw-rw. | |
| 3.8.3. Creating a directory | |
| --------------------------- | |
| Pseudo definition | |
| Filename d mode uid gid | |
| mode is the octal mode specifier, similar to that expected by chmod. | |
| uid and gid can be either specified as a decimal number, or by name. | |
| For example: | |
| /pseudo_dir d 666 root root | |
| creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. | |
| 3.8.4. Modifying attributes of an existing file | |
| ----------------------------------------------- | |
| Pseudo definition | |
| Filename m mode uid gid | |
| mode is the octal mode specifier, similar to that expected by chmod. | |
| uid and gid can be either specified as a decimal number, or by name. | |
| For example: | |
| dmesg m 666 root root | |
| Changes the attributes of the file "dmesg" in the filesystem to have | |
| root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained | |
| from the source filesystem. | |
| 3.9 Miscellaneous options | |
| ------------------------- | |
| The -info option displays the files/directories as they are compressed and | |
| added to the filesystem. The original uncompressed size of each file | |
| is printed, along with DUPLICATE if the file is a duplicate of a | |
| file in the filesystem. | |
| The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple. | |
| This is performed by default to enable the output filesystem file to be mounted | |
| by loopback, which requires files to be a 4K multiple. If the filesystem is | |
| being written to a block device, or is to be stored in a bootimage, the extra | |
| pad bytes are not needed. | |
| 4. UNSQUASHFS | |
| ------------- | |
| Unsquashfs allows you to decompress and extract a Squashfs filesystem without | |
| mounting it. It can extract the entire filesystem, or a specific | |
| file or directory. | |
| The Unsquashfs usage info is: | |
| SYNTAX: ./unsquashfs [options] filesystem [directories or files to extract] | |
| -v[ersion] print version, licence and copyright information | |
| -d[est] <pathname> unsquash to <pathname>, default "squashfs-root" | |
| -n[o-progress] don't display the progress bar | |
| -no[-xattrs] don't extract xattrs in file system | |
| -x[attrs] extract xattrs in file system (default) | |
| -u[ser-xattrs] only extract user xattrs in file system. | |
| Enables extracting xattrs | |
| -p[rocessors] <number> use <number> processors. By default will use | |
| number of processors available | |
| -i[nfo] print files as they are unsquashed | |
| -li[nfo] print files as they are unsquashed with file | |
| attributes (like ls -l output) | |
| -l[s] list filesystem, but don't unsquash | |
| -ll[s] list filesystem with file attributes (like | |
| ls -l output), but don't unsquash | |
| -f[orce] if file already exists then overwrite | |
| -s[tat] display filesystem superblock information | |
| -e[f] <extract file> list of directories or files to extract. | |
| One per line | |
| -da[ta-queue] <size> Set data queue to <size> Mbytes. Default 256 | |
| Mbytes | |
| -fr[ag-queue] <size> Set fragment queue to <size> Mbytes. Default | |
| 256 Mbytes | |
| -r[egex] treat extract names as POSIX regular expressions | |
| rather than use the default shell wildcard | |
| expansion (globbing) | |
| Decompressors available: | |
| gzip | |
| lzo | |
| lz4 | |
| xz | |
| To extract a subset of the filesystem, the filenames or directory | |
| trees that are to be extracted can be specified on the command line. The | |
| files/directories should be specified using the full path to the | |
| files/directories as they appear within the Squashfs filesystem. The | |
| files/directories will also be extracted to those positions within the specified | |
| destination directory. | |
| The extract files can also be given in a file using the "-e[f]" option. | |
| Similarly to Mksquashfs, wildcard matching is performed on the extract | |
| files. Wildcard matching is enabled by default. | |
| Examples: | |
| 1. unsquashfs image.sqsh 'test/*.gz' | |
| Extract all files matching "*.gz" in the top level directory "test". | |
| 2. unsquashfs image.sqsh '[Tt]est/example*' | |
| Extract all files beginning with "example" inside top level directories | |
| called "Test" or "test". | |
| Using extended wildcards, negative matching is also possible. | |
| 3. unsquashfs image.sqsh 'test/!(*data*).gz' | |
| Extract all files matching "*.gz" in top level directory "test", | |
| except those with "data" in the name. | |
| 4.1 Unsquashfs options | |
| ---------------------- | |
| The "-ls" option can be used to list the contents of a filesystem without | |
| decompressing the filesystem data itself. The "-lls" option is similar | |
| but it also displays file attributes (ls -l style output). | |
| The "-info" option forces Unsquashfs to print each file as it is decompressed. | |
| The -"linfo" is similar but it also displays file attributes. | |
| The "-dest" option specifies the directory that is used to decompress | |
| the filesystem data. If this option is not given then the filesystem is | |
| decompressed to the directory "squashfs-root" in the current working directory. | |
| The "-force" option forces Unsquashfs to output to the destination | |
| directory even if files or directories already exist. This allows you | |
| to update an existing directory tree, or to Unsquashfs to a partially | |
| filled directory. Without the "-force" option, Unsquashfs will | |
| refuse to overwrite any existing files, or to create any directories if they | |
| already exist. This is done to protect data in case of mistakes, and | |
| so the "-force" option should be used with caution. | |
| The "-stat" option displays filesystem superblock information. This is | |
| useful to discover the filesystem version, byte ordering, whether it has a NFS | |
| export table, and what options were used to compress the filesystem, etc. | |
| Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x, 3.x and | |
| 4.0 filesystems. | |
| 5. FILESYSTEM LAYOUT | |
| -------------------- | |
| A squashfs filesystem consists of a maximum of nine parts, packed together on a | |
| byte alignment: | |
| --------------- | |
| | superblock | | |
| |---------------| | |
| | compression | | |
| | options | | |
| |---------------| | |
| | datablocks | | |
| | & fragments | | |
| |---------------| | |
| | inode table | | |
| |---------------| | |
| | directory | | |
| | table | | |
| |---------------| | |
| | fragment | | |
| | table | | |
| |---------------| | |
| | export | | |
| | table | | |
| |---------------| | |
| | uid/gid | | |
| | lookup table | | |
| |---------------| | |
| | xattr | | |
| | table | | |
| --------------- | |
| Compressed data blocks are written to the filesystem as files are read from | |
| the source directory, and checked for duplicates. Once all file data has been | |
| written the completed super-block, compression options, inode, directory, | |
| fragment, export, uid/gid lookup and xattr tables are written. | |
| 5.1 Compression options | |
| ----------------------- | |
| Compressors can optionally support compression specific options (e.g. | |
| dictionary size). If non-default compression options have been used, then | |
| these are stored here. | |
| 5.2 Inodes | |
| ---------- | |
| Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each | |
| compressed block is prefixed by a two byte length, the top bit is set if the | |
| block is uncompressed. A block will be uncompressed if the -noI option is set, | |
| or if the compressed block was larger than the uncompressed block. | |
| Inodes are packed into the metadata blocks, and are not aligned to block | |
| boundaries, therefore inodes overlap compressed blocks. Inodes are identified | |
| by a 48-bit number which encodes the location of the compressed metadata block | |
| containing the inode, and the byte offset into that block where the inode is | |
| placed (<block, offset>). | |
| To maximise compression there are different inodes for each file type | |
| (regular file, directory, device, etc.), the inode contents and length | |
| varying with the type. | |
| To further maximise compression, two types of regular file inode and | |
| directory inode are defined: inodes optimised for frequently occurring | |
| regular files and directories, and extended types where extra | |
| information has to be stored. | |
| 5.3 Directories | |
| --------------- | |
| Like inodes, directories are packed into compressed metadata blocks, stored | |
| in a directory table. Directories are accessed using the start address of | |
| the metablock containing the directory and the offset into the | |
| decompressed block (<block, offset>). | |
| Directories are organised in a slightly complex way, and are not simply | |
| a list of file names. The organisation takes advantage of the | |
| fact that (in most cases) the inodes of the files will be in the same | |
| compressed metadata block, and therefore, can share the start block. | |
| Directories are therefore organised in a two level list, a directory | |
| header containing the shared start block value, and a sequence of directory | |
| entries, each of which share the shared start block. A new directory header | |
| is written once/if the inode start block changes. The directory | |
| header/directory entry list is repeated as many times as necessary. | |
| Directories are sorted, and can contain a directory index to speed up | |
| file lookup. Directory indexes store one entry per metablock, each entry | |
| storing the index/filename mapping to the first directory header | |
| in each metadata block. Directories are sorted in alphabetical order, | |
| and at lookup the index is scanned linearly looking for the first filename | |
| alphabetically larger than the filename being looked up. At this point the | |
| location of the metadata block the filename is in has been found. | |
| The general idea of the index is ensure only one metadata block needs to be | |
| decompressed to do a lookup irrespective of the length of the directory. | |
| This scheme has the advantage that it doesn't require extra memory overhead | |
| and doesn't require much extra storage on disk. | |
| 5.4 File data | |
| ------------- | |
| Regular files consist of a sequence of contiguous compressed blocks, and/or a | |
| compressed fragment block (tail-end packed block). The compressed size | |
| of each datablock is stored in a block list contained within the | |
| file inode. | |
| To speed up access to datablocks when reading 'large' files (256 Mbytes or | |
| larger), the code implements an index cache that caches the mapping from | |
| block index to datablock location on disk. | |
| The index cache allows Squashfs to handle large files (up to 1.75 TiB) while | |
| retaining a simple and space-efficient block list on disk. The cache | |
| is split into slots, caching up to eight 224 GiB files (128 KiB blocks). | |
| Larger files use multiple slots, with 1.75 TiB files using all 8 slots. | |
| The index cache is designed to be memory efficient, and by default uses | |
| 16 KiB. | |
| 5.5 Fragment lookup table | |
| ------------------------- | |
| Regular files can contain a fragment index which is mapped to a fragment | |
| location on disk and compressed size using a fragment lookup table. This | |
| fragment lookup table is itself stored compressed into metadata blocks. | |
| A second index table is used to locate these. This second index table for | |
| speed of access (and because it is small) is read at mount time and cached | |
| in memory. | |
| 5.6 Uid/gid lookup table | |
| ------------------------ | |
| For space efficiency regular files store uid and gid indexes, which are | |
| converted to 32-bit uids/gids using an id look up table. This table is | |
| stored compressed into metadata blocks. A second index table is used to | |
| locate these. This second index table for speed of access (and because it | |
| is small) is read at mount time and cached in memory. | |
| 5.7 Export table | |
| ---------------- | |
| To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems | |
| can optionally (disabled with the -no-exports Mksquashfs option) contain | |
| an inode number to inode disk location lookup table. This is required to | |
| enable Squashfs to map inode numbers passed in filehandles to the inode | |
| location on disk, which is necessary when the export code reinstantiates | |
| expired/flushed inodes. | |
| This table is stored compressed into metadata blocks. A second index table is | |
| used to locate these. This second index table for speed of access (and because | |
| it is small) is read at mount time and cached in memory. | |
| 5.8 Xattr table | |
| --------------- | |
| The xattr table contains extended attributes for each inode. The xattrs | |
| for each inode are stored in a list, each list entry containing a type, | |
| name and value field. The type field encodes the xattr prefix | |
| ("user.", "trusted." etc) and it also encodes how the name/value fields | |
| should be interpreted. Currently the type indicates whether the value | |
| is stored inline (in which case the value field contains the xattr value), | |
| or if it is stored out of line (in which case the value field stores a | |
| reference to where the actual value is stored). This allows large values | |
| to be stored out of line improving scanning and lookup performance and it | |
| also allows values to be de-duplicated, the value being stored once, and | |
| all other occurences holding an out of line reference to that value. | |
| The xattr lists are packed into compressed 8K metadata blocks. | |
| To reduce overhead in inodes, rather than storing the on-disk | |
| location of the xattr list inside each inode, a 32-bit xattr id | |
| is stored. This xattr id is mapped into the location of the xattr | |
| list using a second xattr id lookup table. | |
| 6. AUTHOR INFO | |
| -------------- | |
| Squashfs was written by Phillip Lougher, email phillip@lougher.demon.co.uk, | |
| in Chepstow, Wales, UK. If you like the program, or have any problems, | |
| then please email me, as it's nice to get feedback! |