Rsync
External
- Official rsync documentation: https://rsync.samba.org/documentation.html
Internal
Overview
rsync is a differential, recursive file transfer utility. rsync copies files to or from a remote host, or locally on the current host. For remote transfers, rsync must be installed on both the source and destination machines. rsync does not support copying files between two remote hosts.
rsync may connect to the remote system either using a remote-shell programs, such as rsh or ssh, or directly to the rsync daemon running on the remote system, via TCP. The remote-shell transport is used whenever there is a ":" after the host specification. If not configured otherwise with RSYNC_RSH environment variable or with -e command line option, rsync uses ssh by default. The rsync daemon transport is used when there is a "::" separator after host specification, or when "rsync://" is explicitly specified. If neither the source or destination path specify a remote host, the copy occurs locally.
rsync [options] src [src ... src] dest
If src is a directory that does not end with "/", it will be recreated, including its content, in the destination directory. If src is a directory that does end with "/", then its content, including subdirectories, will be recreated in the destination directory. If the destination directory does not exist, it will be created. More details in Source Argument Trailing Slash Semantics.
If a single source argument is specified without a destination, the files are listed in an output format similar to "ls -l".
Examples: see Use Cases below.
Installation
Mac
brew install rsync
Concepts
Source Argument Trailing Slash Semantics
By default, if the source argument is a directory without a trailing slash, rsync will create that directory in the target directory specified as argument. If the source directory has a trailing slash, only the contents of the source directory will be copied, and no directory matching the top level source directory will be created in the destination directory.
rsync Daemon
The rsync daemon transport is used when there is a "::" separator after host specification, or when "rsync://" is explicitly specified. If neither the source or destination path specify a remote host, the copy occurs locally.
rsync User
The rsync damon accepts users, which can log in into modules.
Module
A module is a symbolic name for a directory on the local host. Modules are configured in /etc/rsyncd.conf.
rsyncd.conf
rsync over ssh
Rsync supports connecting to a host using a remote shell or ssh and then spawning a single-use "daemon" server that expects to read its config file in the home dir of the remote user. The remote-shell transport is used whenever there is a ":" after the host specification. If not configured otherwise with RSYNC_RSH environment variable or with -e command line option, rsync uses ssh by default.
All the usual ssh arguments (-p, -i, etc.) can be passed on the rsync command line as follows:
rsync ... -e "ssh <ssh-options>" ...
ssh command line options:
- "-l" specifies the ssh user.
- "-p" specifies the ssh port.
- "-i" specifies the ssh identity file.
Example:
rsync -avzhe "ssh -l <ssh-user> -p <ssh-port> -i <ssh-identity-file> ..." ... 192.168.1.2:/volume1/tmp ./rsync-target
Different ssh and rsync Users
Different users can be used for ssh and rsync, as follows TODO, this did not actually work, more experimentation required.:
rsync -avzhe "ssh -l <ssh-user>" <rsync-user>@<src-host>:/volume1/tmp ./rsync-target rsync -avzhe "ssh -l <ssh-user>" /volume1/tmp <rsync-user>@<dest-host>:/rsync-target
Synology rsync
Note that Synology NAS servers do not react well to the "ssh" command, and the rsync path has to be explicitly specified as shown here:
Use Cases
Pull Files from a Remote Directory
Recursively pull all files from the directory /remote-dir on the machine "remote-host" into the /data/local-dir directory on the local machine. The files are transferred in "archive" mode, which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.
rsync -avz remote-host:/remote-dir /data/local-dir
Push Files to a Remote Directory
The reverse operation, recursively transfer the content (all files and directories) of the local 'local-dir' into the remote 'remote-dir'.
rsync -avz /data/local-dir/ remote-host:/remote-dir
What if I Locally Delete a File and Want to Push
There are situations when we want to keep the local and remote areas in sync, including the files that have been deleted, locally or remotely. This can be done using:
--delete-during
Setup a Remotely Accessible rsync Server
Preserving Spaces in rsync Path
If the path on the remote machine contains spaces, it can be specified as follows:
rsync --protect-args -avzhe ssh "ovidiu@192.168.1.1:/Users/ovidiu/VMware Fusion VMs" .
Excluding Files
Excluding File Patterns
--exclude <pattern>
Example:
--exclude .svn
Multiple patterns can be specified using multiple "--exclude" instances:
--exclude .svn -exclude blah -exclude blah2
Excluding Individual Files
Use the file name as you would use a pattern.
Option Reference
-a
Archive mode, same as -rlptgoD (no -H). Archive mode ensures that symbolic links, devices, attributes, timestamps, permissions, user and group ownerships, etc. are preserved in the transfer. Archive mode copies files recursively, so explicit -r is not necessary.
-v
Verbose.
-z
Compress data during transfer.
--delete
Delete local files that don't exist on sender system.
-h
Human-readable, output numbers in a human-readable format.
--rsync-path
Specifies the rsync to run on the remote machine.
rsync ... --rsync-path=/usr/syno/bin/rsync ... [ssh remote-user@remote-host:/...]
Also see:
--rsync-path can also be used to force the rsync command at the other end to run as root NOT TESTED:
sudo rsync ... --rsync-path="sudo rsync" ...
-s|--protect-args
This option sends all filenames and most options to the remote rsync without allowing the remote shell to interpret them. This means that spaces are not split in names, and any non-wildcard special characters are not translated (such as ~, $, ;, &, etc.). Wildcards are expanded on the remote host by rsync instead of the shell doing it. Also see Preserving Spaces in rsync Path above.
Operations
Configure a rsync Daemon to Start at Boot
systemctl enable rsyncd systemctl is-enabled rsyncd systemctl start rsyncd systemctl status rsyncd