cc_staff
1,486
edits
Transferring data: Difference between revisions
no edit summary
(Note about using "p" or "a" options with rsync.) |
No edit summary |
||
| Line 51: | Line 51: | ||
[https://en.wikipedia.org/wiki/Rsync Rsync] is a popular tool for ensuring that two separate datasets are the same but can be quite slow if there are a lot of files or there is a lot of latency between the two sites, i.e. they are geographically apart or on different networks. Running rsync will check the modification time and size of each file, and will only transfer the file if one or the other does not match. If you expect modification times not to match on the two systems you can use the "-c" option, which will compute checksums at the source and destination, and transfer only if the checksums do not match. | [https://en.wikipedia.org/wiki/Rsync Rsync] is a popular tool for ensuring that two separate datasets are the same but can be quite slow if there are a lot of files or there is a lot of latency between the two sites, i.e. they are geographically apart or on different networks. Running rsync will check the modification time and size of each file, and will only transfer the file if one or the other does not match. If you expect modification times not to match on the two systems you can use the "-c" option, which will compute checksums at the source and destination, and transfer only if the checksums do not match. | ||
When using rsync to transfer to /project if you encounter the message <code>Disk quota exceeded</code> ensure you have not included the <code>p</code> option, either directly or indirectly with the <code>a</code> option as this can cause the group ID of the file to be incorrectly set | When using rsync to transfer to /project if you encounter the message <code>Disk quota exceeded</code> ensure you have not included the <code>p</code> option, either directly or indirectly with the <code>a</code> option as this can cause the group ID of the file to be incorrectly set. The group ID is used to apply a file's size towards a project quota. | ||
===Using checksums to check if files match=== <!--T:13--> | ===Using checksums to check if files match=== <!--T:13--> | ||