Transferring data

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

From your personal computer

You will need software that supports secure transfer of files between your computer and the Compute Canada machines. The command line programs scp and sftp can be used from within terminal programs on Linux or Mac OS X computers. On Microsoft Windows platforms, MobaXterm offers both file transfer and a terminal function, while WinSCP is another free program that supports file transfer. PuTTY comes with pscp and psftp which are essentially the same as the Linux and Mac command line programs.

If it takes more than about a minute to move your files to or from Compute Canada servers, you should install Globus Personal Connect and try it. Globus transfers can be set up and will go on in the background without you. Most (but not all) Compute Canada legacy systems can be reached with Globus.

Between Compute Canada resources

Globus is the preferred tool for transferring data between Compute Canada systems, and if it can be used, it should.

However, other common tools can also be found for transferring data both inside and outside of Compute Canada, including

• Secure copy scp (examples here)
• SFTP
• rsync

From the World Wide Web

The standard tool for downloading data from websites is wget.

Synchronize or verify files after transfer

Two synchronize or "sync" two files (or two directories) stored in two different locations means to ensure that the two copies are the same. Here are several different ways to do this.

Globus Transfer

We find Globus Transfer usually gives the greatest performance and reliability.

Normally when a Globus Transfer is initiated it will overwrite the files on the destination with the files from the source, which means all of the files on the source will be transferred. If some of the files may already exist on the destination, and need not be transferred if they match, you should go to the bottom of the transfer window as shown in the screenshot and choose to "sync" instead.

You may choose how Globus decides which files to transfer:

Their checksums are different	This is the slowest option but most accurate. This will catch changes or errors that result in the same size of file, but with different contents.
File doesn't exist on destination	This will only transfer files that have been created since the last sync. Useful if you are incrementally creating files.
File size is different	A quick test. If the file size has changed then its contents must have changed, and it will be re-transferred.
Modification time is newer	This will check the file's recorded modification time and only transfer the file if it is newer on the source than the destination. If you want to depend on this it is important to check the "preserve source file modification times" option when initiating a Globus Transfer.

For more information about Globus please see Globus.

Rsync

Rsync is a popular tool for ensuring that two separate datasets are the same but can be quite slow if your dataset has a lot of files or there is a lot of latency between the two sites, i.e. they are geographically apart or on different networks. Running rsync will check the modification time and size of each file before transmitting it. If for some reason your modification times do not match on the two systems you can also run using the "-c" option which will create a checksum of the source and destination file before transferring. Generating checksums for files can slow down.

Using checksums to check if files match

If Globus is unavailable between the two systems being synchronized and Rsync is taking too long, then you can use a checksum utility on both systems to determine if the files match. In this example we use sha1sum.

[name@server]$ find /home/username/ -type f -print0 | xargs -0 sha1sum | tee checksum-result.log

This command will create a new file called checksum-result.log in the current directory that will contain all of the checksums for the files in /home/username/. It will also print out all of the checksums to the screen as it goes. If you have a lot of files or very large files you may want to run this command in the background, in a screen or tmux session; anything that allows it to continue if your SSH connection times out.

After you run it on both systems you can use the diff utility to find files that don't match.

[name@server ~]$ diff checksum-result-silo.log checksum-dtn.log
69c69
 < 017f14f6a1a194a5f791836d93d14beead0a5115  /home/username/file-0025048576-0000008
 ---
 > 8836913c2cc2272c017d0455f70cf0d698daadb3  /home/username/file-0025048576-0000008

It is possible that the find command will crawl through the directories in a different order resulting in a lot of false differences so you may need to run sort on both files before running diff such as:

[name@server ~]$ sort -k2 checksum-result-silo.log -o checksum-result-silo.log
[name@server ~]$ sort -k2 checksum-dtn.log -o checksum-dtn.log

Globus is a file transfer service