Hello everybody!
In today’s video, we will discuss sparse files.
Keep on watching to find out: what files are called sparse files, what are their pros and
cons, what file systems support them, how to create a sparse file, and how to transform
an ordinary file into a
sparse file.
Many applications - and especially databases - keep their information in sparse files.
This is a kind of file that tries to use the file system in a more efficient way without
occupying disk space when the file is almost empty.
In fact, with this file type your storage device is only used when necessary.
The information which is not written to the disk is presented in the form of zeros and
is stored in the file system metadata.
This way, such files will take less space on the disk than their size which is displayed.
This file type is supported by most file systems, and you can see their list on the screen right
now.
…Sparse files are supported by the following operating systems: BTRFS, NILFS, ZFS, NTFS[2],
ext2, ext3, ext4, XFS, JFS, ReiserFS, Reiser4, UFS, Rock Ridge, UDF, ReFS, APFS, F2FS...
Although sparse files are supported by these file systems, it is impossible to access their
functionality directly through the interface of the operating system.
Files of this type can only be managed with the command prompt.
A bit later, we’ll explore it in more detail.
The difference between compressed files and sparse files
In addition to sparse file support, operating systems also have a feature known as file
compression.
Both tools give you an advantage of saving disk space, but they achieve this goal in
different ways.
The main downside of using file compression is that affects system performance when reading
or writing such files, Because the system needs extra resources to compress or decompress
the data.
For some programs, it is inadmissible, and that is the case when it’s better to use
sparse files.
Advantages and disadvantages of sparse files The biggest advantage of sparse files is that
users can create vary large files which take very little disk space.
The space for storing data is allocated automatically as more and more information is written.
Large sparse files are created relatively quickly because the file system doesn’t
have to pre-allocate disk space for writing zeros.
However, the advantages of sparse files are limited to the applications that support them.
If a certain program cannot recognize or use such files, it will save a sparse file in
its original - uncompressed - form, and there’s no advantage to talk about.
Users have to be careful when dealing with such files: a sparse file of a few megabytes
may suddenly turn into a several-gigabyte monster when an application which doesn’t
support this file type copies it into a directory.
One more disadvantage is that you can’t copy or create a sparse file if its nominal
size exceeds the amount of available free space (or quota restrictions applied to user
accounts within the operating system).
For example, if the original size of a sparse file (with all of its zero bytes) is 500 MB,
and the quota limit for the user account where this sparse file is created is only 400 MB,
such situation generates an error, even if the actual disk space occupied by this sparse
file is only 50 MB.
As to the hard disks where sparse files are stores, they are also subject to fragmentation,
because the file system will write data to sparse files as and when necessary.
Over time, it may result in reduced performance.
Besides, some disk management utilities may have issues with displaying free disk space.
When the file system containing files of this specific type is low on space, it may lead
to unpredictable situations.
For example, it may generate disk overfill issues when data is copied over the existing
part of the file which is marked as sparse.
How to create a sparse file in Windows To create a sparse file in Windows, we’ll
be using command prompt.
Type “cmd” in the search field and run it as Administrator.
In Windows, sparse files are managed with the help of fsutil tool (a file system utility).
When the command “create” is performed, by default it creates a file which is not
a sparse one.
Go to the folder where a sparse file needs to be created and
type the following command: fsutil file createnew sparse-file 1000000000
Where sparse-file is the file name, and the number at the end is the file size in bytes.
To assign the sparse attribute to the file, use this command:
fsutil sparse setflag sparse-file To remove the “sparse” flag, use another
command: fsutil sparse setflag sparse-file 0
To assign the “sparse” attribute again, type one more command:
fsutil sparse queryflag sparse-file The attribute by itself doesn’t let you
start saving disk space.
To make it happen, you need to set the empty space which should be freed inside the file.
Set the empty space with this command: fsutil sparse setrange sparse-file 0 1000000000
At the end of the command, the offset and length are given in bytes.
In my case, this is the range from zero to 1 GB.
To set a fully sparse file, specify the whole capacity.
You can expand the file if necessary by typing a larger value.
To make sure the file has been assigned this attribute, run the command layout
fsutil file layout sparse-file The sparse attribute can be assigned to any
file by just running this command with the corresponding file’s name and setting the
desired file size.
In the properties of the file we have created earlier, you can see that with the size of
1 GB this file occupies 0 bytes on disk.
This set of commands works with all Windows file systems that support sparse files (NTFS,
ReFS and so on).
How to create a sparse file in Linux In Linux, the process of creating a sparse
file is a bit easier, because there are several commands to create files of this type.
This set of commands works with all Linux file systems that support sparse files.
You can either use the dd command or the truncate command.
The first command looks like this: dd if=/dev/zero of=file-sparse bs=1 count=0
seek=2G Where file-sparse is the file name, and the
number at the end is the size that can be set in bytes, megabytes and so on.
The second command is simpler: truncate -s2G file-sparse
Where the s value stands for the file size, and it is followed by the file name.
Contrary to Windows, when you create a file with either command it will be a sparse file
by default.
To turn an ordinary file into a sparse one, there is a special command:
cp --sparse=always ./025.jpg ./026.jpg Where 025.jpg is the name of the first file,
an ordinary one, and 026.jpg is the name of the second file,
a sparse one.
How to expand a file If you need to expand an existing file, use
the first command below: replace the name and specify the size you need.
dd if=/dev/zero of=025.jpg bs=1 count=0 seek=2G This command will increase its size to 2 GB.
Use the following command to check the size: du -h --apparent-size 025.jpg
Sparse files in APFS In fact, this set of commands works with the
Apple file system – ApFS, as Linux and MacOS are based on the Unix architecture and they
both provide access to Unix commands and the Bash command shell.
Launch the Terminal and run any command for creating a sparse file that I have just used
in Linux.
In MacOS Catalina, only the first command is supported, and the size should be given
in bytes, otherwise the command will generate an error.
sudo dd if=/dev/zero of=sparse_APFS bs=1 count=0 seek=1000000000
When certain conditions are observed, ApFS file system creates sparse files by default,
so you can expand any file with the command used to increase the size of sparse files
in Linux: dd if=/dev/zero of=187.jpg bs=1 count=0 seek=500000000
Let’s set the size of 500 MB, and remember that in MacOS this size should be specified
in bytes.
When you run the command and open file properties, you can see that its size has increased to
500 MB.
Recovering sparse files If you removed some files, or formatted a
disk with important information, and those files turned out to be sparse files, keep
calm and carry on - with Hetman Partition Recovery or Hetman RAID Recovery.
Our products can work with sparse files in all operating systems and file systems mentioned
in this video.
Download, install and run the program to scan your disk.
As you can see, this program can find both existing and deleted sparse files easily,
and display their contents.
When you open the file in HEX editor, you can make sure that is it filled with zeros.
The program managed to find previously removed sparse files on the disk using NTFS, REFS,
BtrFS, Ext3, Ext4 and ApFS file systems.
All you have to do is to select the data you need
and click Recovery.
When the recovery process is over, you’ll find all of the files in the folder you have
chosen.
Check the file properties to find out that the files have been recovered in full.
Conclusion The decision to use sparse files in any of
the operating systems that support them has its pros and cons, so think about possible
downsides before choosing this option.
Knowing about issues you may face while using sparse files can help you to avoid potential
problems in the future.
That is all for now!
Hopefully, this video was useful.
Remember to click the Like button and subscribe to our channel.
Push the bell button to receive notifications and never miss new videos.
Leave comments to ask questions.
Thank you for watching.
Good luck.