rsync is a file synchronization and file transfer program for Unix-like systems that minimizes
network data transfer by using a form of delta encoding called the rsync algorithm.
rsync can compress the data transferred further using zlib compression, and SSH or stunnel
can be used to encrypt the transfer.
rsync is typically used to synchronize files and directories between two different systems,
one local and one remote.
For example, if the command rsync local-file user@remote-host:remote-file is run, rsync
will use SSH to connect as user to remote-host.
Once connected, it will invoke another copy of rsync on the remote host, and then the
two programs will talk to each other over the connection, working together to determine
what parts of the file are already on the remote host and don't need to be transferred
over the connection.
rsync can also operate in daemon mode, where it listens by default on TCP port 873, serving
files in the native rsync protocol.
Released under the GNU General Public License version 3 and widely used.
History Andrew Tridgell and Paul Mackerras wrote the
original rsync.
Tridgell discusses the design, implementation and performance of rsync in chapters 3 through
5 of his Ph.D. thesis in 1999.
rsync was first announced on 19 June 1996 and the first release of major version 3 was
issued on 1 March 2008.
Uses rsync originated as a replacement for rcp
and scp.
As such, it has a similar syntax to its parent programs.
Like its predecessors, it requires the specification of a source and of a destination; either of
them may be remote, but not both.
Because of the flexibility, speed and scriptability of rsync, it has become a standard Linux utility,
included in all popular Linux distributions.
It has been ported to Windows and Mac OS.
Generic syntax:
...where SRC is the file or directory to copy from, and DEST represents the file or directory
to copy to.
rsync can synchronize Unix clients to a central Unix server using rsync/ssh and standard Unix
accounts.
It can be used in desktop environments, for example to efficiently synchronize files with
a backup copy on an external hard drive.
A scheduling utility such as cron can carry out tasks such as automated encrypted rsync-based
mirroring between multiple hosts and a central server.
Examples A command line to mirror FreeBSD might look
like: % rsync -avz --delete ftp4.de.FreeBSD.org::FreeBSD/
FreeBSD/ The Apache HTTP Server supports only rsync
for updating mirrors.
rsync -avz --delete --safe-links rsync.apache.org::apache-dist to/mirror
The preferred way to mirror the PuTTY website to the current directory is to use rsync.
rsync -auH rsync:rsync.chiark.greenend.org.ukusersputty-website-mirror/ .
A way to mimic the capabilities of Time Machine - see also tym.
#date=`date "+%Y-%m-%dT%H:%M:%S"` date=`date "+%FT%T"`
rsync -aP --link-dest=$HOMEcurrent to/important_files $HOMEback-$date
ln -nfs $HOMEback-$date $HOMEcurrent
Algorithm Determining which files to send
By default rsync determines which files differ between the sending and receiving systems
by checking the modification time and size of each file.
This method incurs hardly any I/O and CPU usage, but will miss files whose content,
unusually, has changed without modification to size or timestamp.
rsync can perform a more comprehensive check via the --checksum flag, forcing a full checksum
comparison on every file present on both systems.
Barring rare checksum collisions this avoids the risk of missing changed files, but incurs
reading of every file present on both systems and is thus much slower.
Determining which parts of a file have changed The rsync utility uses an algorithm invented
by Australian computer programmer Andrew Tridgell for efficiently transmitting a structure across
a communications link when the receiving computer already has a similar, but not identical,
version of the same structure.
The recipient splits its copy of the file into fixed-size non-overlapping chunks and
computes two checksums for each chunk: the MD5 hash, and a weaker 'rolling checksum'.
It sends these checksums to the sender.
The sender computes the rolling checksum for every chunk of size S in its own version of
the file, even overlapping chunks.
This can be calculated efficiently because of a special property of the rolling checksum:
if the rolling checksum of bytes n through n+S-1 is R, the rolling checksum of bytes
n+1 through n+S can be computed from R, byte n, and byte n+S without having to examine
the intervening bytes.
Thus, if one had already calculated the rolling checksum of bytes 1...25, one could calculate
the rolling checksum of bytes 2...26 solely from the previous checksum, byte 1, and byte
26.
The rolling checksum used in rsync is based on Mark Adler's adler-32 checksum, which is
used in zlib, and is itself based on Fletcher's checksum.
The sender then compares its rolling checksums with the set sent by the recipient to determine
if any matches exist.
If they do, it verifies the match by computing the hash for the matching block and by comparing
it with the hash for that block sent by the recipient.
The sender then sends the recipient those parts of its file that did not match the recipient's
blocks, along with information on where to merge these blocks into the recipient's version.
This makes the copies identical.
However, there is a small probability that differences between chunks in the sender and
recipient are not detected, and thus remains uncorrected.
This requires a simultaneous hash collision in MD5 and the rolling checksum.
It is possible to generate MD5 collisions, and the rolling checksum is not cryptographically
strong, but the chance for this to occur by accident is nevertheless extremely remote.
With 128 bits from MD5 plus 32 bits from the rolling checksum, and assuming maximum entropy
in these bits, the probability of a hash collision with this combined checksum is 2−(128+32)
= 2−160.
The actual probability is a few times higher, since good checksums approach maximum output
entropy but very rarely achieve it.
If the sender's and recipient's versions of the file have many sections in common, the
utility needs to transfer relatively little data to synchronize the files.
Note that if usual data compression algorithms are used, files that are similar when uncompressed
may be very different when compressed, and thus the entire file will need to be transferred
– local changes in uncompressed files yield global changes in compressed files.
This is particularly an issue with mirroring of archive files, such as disk images and
compressed tarballs, where often individual files change.
Some compression programs, such as gzip, provide a special "rsyncable" mode which allows these
files to be efficiently rsynced, by ensuring that local changes in the uncompressed file
yield only local changes in the compressed file.
While the rsync algorithm forms the heart of the rsync application that essentially
optimizes transfers between two computers over TCP/IP, the rsync application supports
other key features that aid significantly in data transfers or backup.
They include compression and decompression of data block by block using zlib at sending
and receiving ends, and support for protocols such as ssh that enables encrypted transmission
of compressed and efficient differential data using rsync algorithm.
Instead of ssh, stunnel can also be used to create an encrypted tunnel to secure the data
transmitted.
Variations A utility called rdiff uses the rsync algorithm
to generate delta files with the difference from file A to file B.
The delta file can then be applied to file A, turning it into file B.
Unlike diff, the process of creating a delta file has two steps: first a signature file
is created from file A, and then this signature and file B are used to create the delta file.
Also unlike diff, rdiff works well with binary files.
Using the library underlying rdiff, librsync, a utility called rdiff-backup has been created,
capable of maintaining a backup mirror of a file or directory either locally or remotely
over the network, on another server.
rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible
to recreate any backup point.
The librsync library used by rdiff is an independent implementation of the rsync algorithm.
It does not use the rsync network protocol and does not share any code with the rsync
application.
The librsync library is used by Dropbox, rdiff-backup, duplicity, and other utilities.
Duplicity is a variation on rdiff-backup that allows for backups without cooperation from
the storage server, as with simple storage services like Amazon S3.
It works by generating the hashes for each block in advance, encrypting them, and storing
them on the server, then retrieving them when doing an incremental backup.
The rest of the data is also stored encrypted for security purposes.
rsyncrypto is a utility to encrypt files in an rsync-friendly fashion.
The rsyncrypto algorithm ensures that two almost identical files, when encrypted with
rsyncrypto and the same key, will produce almost identical encrypted files.
This allows for the low-overhead data transfer achieved by rsync while providing encryption
for secure transfer and storage of sensitive data in a remote location.
An alternative to manually scripting rsync is the GPL licensed GUI program BackupPC,
which performs automatic scheduled backups to rsync servers.
As of Mac OS X 10.5 and later, there is a special -E or—extended-attributes switch
which allows retaining much of the HFS file metadata when syncing between two machines
supporting this feature.
This is achieved by transmitting the Resource Fork along with the Data Fork.
Solutions using rsync See also
Remote Differential Compression References
External links Official website
rsync algorithm