Transferring data: Difference between revisions

<translate>

<!--T:4-->

* Secure copy [https://en.wikipedia.org/wiki/Secure_copy scp] (examples [http://www.hypexr.org/linux_scp_help.php here])

* [[Transferring_data#SFTP | SFTP]]

* [https://en.wikipedia.org/wiki/Rsync rsync]

[[Globus]] is the preferred tool for transferring data between Compute Canada systems, and if it can be used, it should.

You will need software that supports secure transfer of files between your computer and the Compute Canada machines. The commands <code>scp</code> and <code>sftp</code> can be used in a command-line environment on '''Linux''' or '''Mac''' OS X computers. On '''Microsoft Windows''' platforms, [https://docs.computecanada.ca/wiki/Connecting_with_MobaXTerm/en MobaXterm] offers both a graphical file transfer function and a [[Linux introduction|command-line]] interface via [[SSH]], while [http://winscp.net/eng/index.php WinSCP] is another free program that supports file transfer. [https://docs.computecanada.ca/wiki/Connecting_with_PuTTY/en PuTTY] comes with <code>pscp</code> and <code>psftp</code> which are essentially the same as the Linux and Mac command line programs.

If it takes more than one minute to move your files to or from Compute Canada servers, we recommend you install and try [[Globus#Personal_Computers|Globus Personal Connect]]. [[Globus]] transfers can be set up and will go on in the background without you. Most Compute Canada legacy systems can be reached with Globus.

The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget].

To synchronize or "sync" files (or directories) stored in two different locations means to ensure that the two copies are the same. Here are several different ways to do this.

We find Globus usually gives the best performance and reliability.

For more information about Globus please see [[Globus]].

[https://en.wikipedia.org/wiki/Rsync Rsync] is a popular tool for ensuring that two separate datasets are the same but can be quite slow if there are a lot of files or there is a lot of latency between the two sites, i.e. they are geographically apart or on different networks. Running rsync will check the modification time and size of each file, and will only transfer the file if one or the other does not match. If you expect modification times not to match on the two systems you can use the "-c" option, which will compute checksums at the source and destination, and transfer only if the checksums do not match.

If Globus is unavailable between the two systems being synchronized and Rsync is taking too long, then you can use a  [https://en.wikipedia.org/wiki/Checksum checksum] utility on both systems to determine if the files match. In this example we use <code>sha1sum</code>.

}}

[https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol SFTP] (Secure File Transfer Protocol) uses the SSH protocol to transfer files between machines which encrypts data being transferred.