cc_staff
1,857
edits
Transferring data: Difference between revisions
m
no edit summary
mNo edit summary |
mNo edit summary |
||
| Line 75: | Line 75: | ||
When transferring files into the <code>/project</code> file systems, do not use <code>-p</code> and <code>-g</code> flags since the quotas in <code>/project</code> are enforced based on group ownership, and thus preserving the group ownership will lead to the [[Frequently_Asked_Questions#Disk_quota_exceeded_error_on_.2Fproject_filesystems | Disk quota exceeded]] error message. Since <code>-a</code> includes <code>-p</code> and <code>-g</code> by default, the <code>--no-g --no-p</code> options should be added, like so | When transferring files into the <code>/project</code> file systems, do not use <code>-p</code> and <code>-g</code> flags since the quotas in <code>/project</code> are enforced based on group ownership, and thus preserving the group ownership will lead to the [[Frequently_Asked_Questions#Disk_quota_exceeded_error_on_.2Fproject_filesystems | Disk quota exceeded]] error message. Since <code>-a</code> includes <code>-p</code> and <code>-g</code> by default, the <code>--no-g --no-p</code> options should be added, like so | ||
{{Command|rsync -avzh --no-g --no-p LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | {{Command|rsync -avzh --no-g --no-p LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | ||
where LOCALNAME can be a directory or file preceded by its path location and somedir will be created if it doesn't exist. The <code>-z</code> option compresses files | where LOCALNAME can be a directory or file preceded by its path location and somedir will be created if it doesn't exist. The <code>-z</code> option compresses files (not in the default file suffixes skip-compress list) and requires additional cpu resources while the <code>-h</code> option makes transferred file sizes human readable. If you are transferring very large files add the <code>--partial</code> option so interrupted transfers maybe restarted: | ||
{{Command|rsync -avzh --no-g --no-p --partial --progress LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | {{Command|rsync -avzh --no-g --no-p --partial --progress LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | ||
The <code>--progress</code> option will display the percent progress of each file as its transferred. If you are transferring very many smaller files, then it maybe more desirable to display a single progress bar that represents the transfer progress of all files: | The <code>--progress</code> option will display the percent progress of each file as its transferred. If you are transferring very many smaller files, then it maybe more desirable to display a single progress bar that represents the transfer progress of all files: | ||
{{Command|rsync -azh --no-g --no-p --info{{=}}progress2 LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | {{Command|rsync -azh --no-g --no-p --info{{=}}progress2 LOCALNAME someuser@graham.computecanada.ca:projects/def-professor/someuser/somedir/}} | ||
The above rsync examples all involve transfers from a local system into a project directory on a remote system. Rsync transfers from a remote system into a project directory on a local system work in much the same way, for example: | The above rsync examples all involve transfers from a local system into a project directory on a remote system. Rsync transfers from a remote system into a project directory on a local system work in much the same way, for example: | ||
{{Command|rsync - | {{Command|rsync -avzh --no-g --no-p someuser@graham.computecanada.ca:REMOTENAME ~/projects/def-professor/someuser/somedir/}} | ||
where REMOTENAME can be a directory or file preceded by its path location and somedir will be created if it doesn't already exist. In its simplest incarnation rsync can be used within a single system to transfer a directory or file (from home or scratch) into project by dropping the cluster name | where REMOTENAME can be a directory or file preceded by its path location and somedir will be created if it doesn't already exist. In its simplest incarnation rsync can also be used locally within a single system to transfer a directory or file (from home or scratch) into project by dropping the cluster name: | ||
{{Command|rsync -avh --no-g --no-p LOCALNAME ~/projects/def-professor/someuser/somedir/}} | {{Command|rsync -avh --no-g --no-p LOCALNAME ~/projects/def-professor/someuser/somedir/}} | ||
where somedir will be created if it doesn't already exist | where somedir will be created if it doesn't already exist before copying LOCALNAME into it. | ||
===Using checksums to check if files match=== <!--T:13--> | ===Using checksums to check if files match=== <!--T:13--> | ||