General directives for migration: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Draft}}
<languages/>


== This page is a draft --- work in progress ==
<translate>
<!--T:1-->
This page explains issues related to transferring your data between our facilities and our regional partners.


== Notes for users to prepare for Data Migration ==
<!--T:2-->
If you are in any doubt about details of the following advice, contact our [[technical support]] for help.


This page is dedicated to users of Compute Canada clusters concerned by the process of data migration. It is also useful for other users since it explains how to proceed when it comes to transfer your data from one place to another between Compute Canada facilities and its regional partners ([http://www.ace-net.ca/ ACENET], [http://www.calculquebec.ca/en/ Calcul Quebec], [http://computeontario.ca/ Compute Ontario] and [https://www.westgrid.ca/ WestGrid]). You will find the best practices and useful links related to [[https://docs.computecanada.ca/wiki/Archiving_and_Compressing_Data_for_Migration|Data Archiving]] and [[https://docs.computecanada.ca/wiki/Globus|Data Transfer]] or data migration. In this page, you will find some tips, instructions and links on how to prepare your data and archives to facilitate the migration process. Cleaning your directories and archiving your data is part of migration process. Here, we give you some information and tips on what to do before, during and after the migration process.
== What to do before migrating ? == <!--T:3-->
Make sure you know whether you are responsible for your own data migration, or whether our staff will be migrating your data. If you are in any doubt, contact our [[technical support]].


== What to do before the migration starts? ==
<!--T:4-->
If you haven't used [[Globus]] before, read about it now and verify that it works on the system you are migrating from. Test any other tools you will use (like [http://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/ tar], [https://www.gnu.org/software/gzip/manual/gzip.html gzip], [https://www.cyberciti.biz/faq/how-to-create-a-zip-file-in-unix/ zip]) on test data to ensure you know how they work before using them on important data.


It is always a good practice to go through your directories on your working directories to check what data you have and how their structure is in your directories. In most of the clusters, each user on CC has two working directories: '''/home/your_user_name''' and '''/global/scratch/your_user_name''' (in some clusters the path your directories may be different). Each user needs to go through his or her directories to check their structure and start organizing and cleaning them as needed, especially before proceeding with data migration. You should reduce the amount of the files you need to migrate by using [[Archiving_and_Compressing_Data_for_Migration|archiving utilities]] and removing any unnecessary data.  
<!--T:5-->
Do not wait until the last minute to start your migration. Depending on how much data you have and how much load there is on the machines and network, you may be surprised at how long it will take to finish a large transfer. Expect hundreds of gigabytes to take hours to transfer, but give yourself days in case there is a problem. Expect terabytes to take days.


Taking some time to organize and clean your directories now is not a waste of time because you will considerably speed up your migration process and give you the opportunity to better manage your data on the new system(s) to retrieve your files more quickly in the near future when you will be looking for them. These are some of recommended practices:
=== Clean up === <!--T:6-->
* Look at your data and see how your directories are structured and stored in your directories.
It is a good practice to look at your files regularly and see what can be deleted, but unfortunately many of us do not have this habit. A major data migration is a good reminder to clean up your files and directories. Moving less data will take less time, and storage space even on new systems is in great demand and should not be wasted.
* Clean your data and directories by removing any unnecessary files you do not need.
* If you compile programs and keep source code, delete any intermediate files. One or more of <code>make clean</code>, <code>make realclean</code>, or <code>rm *.o</code> might be appropriate, depending on your [[Make|makefile]].
* If you build programs in your home directory, start by removing all object files and keep only source files and configuration files to be able to rebuild your applications in the new clusters.
* If you find any large files named like <code>core.12345</code> and you don't know that they are, they are probably [https://en.wikipedia.org/wiki/Core_dump core dumps] and can be deleted.
* The main goal from this preparation process that consists on cleaning and archiving your data is to find a good and best way to monitor your data and for better handling of the migration process. It is easier for the secure copy protocol or file transfer programs to migrate one archive file of a reasonable size than migrating thousands of small files. These files could be archived and compressed to reduce their size. To avoid any interruption or slowing down of the migration process, it is recommended to transfer archives rather than the whole directory with all files individually.
* Identify large data that can be compressed separately (this will save space and speed up the migration process).
* Identify the directories with large number of small files and use archiving and compressing utilities. When it comes to transfer files or data from one system to another, it is much faster for example to transfer an archive file [archive.tar.gz or archive.tar.bz2] that has for example 1000 small files than transferring the 1000 files individually. As an example to see why it is important to archive and compress your data, some files (especially those in text format) can be reduced by more than 50 % of their initial size. The compression of some files is very low but if you have hundreds or thousands of those files, you will find that you can reduce your space by 5 to 10 % or more. The numbers here are just indications to have an idea because the compression rate depend on the type of your data and in which format they are write. For images and binaries for example, the compression rate is very low, however for text data, it can be more than 60 %. 
* If you are migrating data from more than one place to a unique final destination, you may think about a better way to do not have the same names. If you do so, the first data you moved will be replaced by the last ones. To avoid these situations, it may be necessary to prepare directories according to the initial place where you moved your data. This will help you retrieve and recognize easily the origin of your data.
* Choose carefully the names of the archives and be sure to do not give the same name to two different archives especially if you put them on the same directory.
* Try to have a more cleaned and structured directories.
* Check if your data are duplicated. It is not necessary to transfer the same file twice.
* list item Read carefully the instruction on how to prepare data archives by visiting [[https://docs.computecanada.ca/wiki/Archiving_and_Compressing_Data_for_Migration|this web page]].
* Try some test files to see how all the archiving utilities work and how to use the transfer tools.
* Read carefully the instruction on how to use [[https://docs.computecanada.ca/wiki/Globus|Globus]] for file transfer. Other tools can be used to transfer but it is recommended to migrate your Data between Compute Canada facilities using [[https://docs.computecanada.ca/wiki/Globus|Globus]]. You can start by transferring few files and find out how it works.


== What to do during the migration process? ==
=== Archive and compress === <!--T:7-->
Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use [[Archiving and compressing files|tar]] to combine (archive) them.


* Be patient. Migrating data from one site to another can be long and time consuming. Depending on the amount of data you have and how many users are going to migrate their data, this process can be scheduled over few days.
<!--T:8-->
* if you have a huge amount of data to transfer, do not stay till the last minute to start the migration. Depending on how much data you have, it can take a while to finish the transfer. This is another reason to prepare archives before migrating your data.
Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file <b>only</b> for the purpose of transferring it, and then decompressing it at the end of the transfer will not necessarily save time. It depends on how much the file can be compressed, how long it takes to compress it, and the transfer bandwidth. The calculation is described under <i>Data Compression and transfer discussion</i> in [https://bluewaters.ncsa.illinois.edu/data-transfer-doc this document] from the US National Center for Supercomputing Applications.
* Once your directories are cleaned and your data compressed, you can start the migration process to the new facilities using the Globus file transfer protocol.
* Do not try to migrate the whole data at once. Depending on the number of users and the amount of the data to migrate, the file system can slow down and stops. If it happens, the migration process will take longer.
* Make a schedule to migrate your data part by part. With this strategy, you can control what data you migrate. If for any reason, the system stops or your connection interrupted you will be able to try again later to migrate the same data instead of starting again if you tried to migrate your whole data or directories at once.
* Make sure to check out that the process did not stop after you started the migration. For this reason, it is highly recommended to migrate your data part by part.
* Be sure that you did not miss anything of your data if the migration process requires more time. It may be necessary to create a new directory with a name like Data_Migrated for example. The idea is to move any data you already migrated to this directory when you are sure that the data is in the destination cluster. The next time, you want to continue with the migration process, you will just look at the data outside this directory and you will not ask yourself or have to check again in the destination directory if your data have already been transferred or not. You may also keep records of the data you moved to see what is left to migrate and make a schedule to do it within the period dedicated for the migration (sooner is better than waiting to the last minute).
* Make sure that the data you are about to transfer are not corrupted. This can be achieved by different means (compare the file size on your local directory and the destination directory, try to check your data if they are not altered or corrupted during the migration process. To do so, you can look at your data using some utilities (to learn more [[https://docs.computecanada.ca/wiki/Archiving_and_Compressing_Data_for_Migration|visit this page]]).  
== What to do after migration? ==


* Connect to the remote machine and check out that your data is there and compare the size of your archives between the initial and final destination. If there is a problem during the migration, the system may copy part of the archive and not the whole file. You may see the archive name but its size will be different from the original one. In this case, the data may be corrupted and you will not be able to retrieve all your data. You have to transfer again this particular file or archive. For more details about how to check if your data is not corrupted during the migration process can be found on the following link: [[https://docs.computecanada.ca/wiki/Archiving_and_Compressing_Data_for_Migration| Archiving and Compressing Data for Migration]].
<!--T:9-->
If you decide compression is worthwhile, you can again use [[Archiving and compressing files|tar]] for this, or [https://www.gnu.org/software/gzip/manual/gzip.html gzip].


* Organizer your data on your new working directories on the new clusters.
=== Avoid duplication === <!--T:10-->
Try not to move the same data twice. If you are migrating from more than one existing system to one new system and you have data duplicated on the sources, choose one and only move the duplicate data from that one.  


== Where and how to get HELP? ==
<!--T:11-->
Beware of files with duplicate names, but which do not contain duplicate information. Ensure that you will not accidentally overwrite one file with another of the same name.


* To know how to use different archiving and compression utilities, use the Linux command like man <command> or <command> --help. Most common commands and useful examples can be found on the [[Archiving_and_Compressing_Data_for_Migration|Archiving and Compressing]] page.  
== What to do during the migration process == <!--T:12-->
If it is supported at your source site, use [[Globus|Globus]] to set up your file transfer. It is the most user-friendly and efficient tool we know for this task. Globus is designed to recover from network interruptions automatically. We recommend you enable the setting to <i>preserve source file modification times</i> in the <i>Transfer & Timer Options</i>.


* Ask around you for more help and clarification.
<!--T:13-->
* Contact Migration support team by email: (no email provided yet)
If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using [https://en.wikipedia.org/wiki/Secure_copy scp], [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol sftp], or [https://en.wikipedia.org/wiki/Rsync rsync], then:
* Make a schedule to migrate your data in blocks of a few hundreds of GBs at a time. If the transfer stops for some reason, you will be able to try again starting from the incomplete file, but you will not have to re-transfer files that are already complete. An organized list of files will help here.
* Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our [[technical support]].
 
<!--T:14-->
Be patient. Even with Globus, transferring large volumes of data can be time consuming. Specific transfer speeds will vary, but expect hundreds of gigabytes to take hours and terabytes to take days.
 
== What to do after migration == <!--T:15-->
If you did not use Globus, or if you did but did not check <i>verify file integrity</i>, make sure that the data you have transferred are not corrupted. A crude way to do this is to compare file sizes at the source with file sizes at the destination. For greater assurance, you can use [http://man7.org/linux/man-pages/man1/cksum.1.html cksum] or [http://man7.org/linux/man-pages/man1/md5sum.1.html md5sum] at each end, and see if the results match. Any files with mismatching sizes or checksums should be transferred again.
 
== Where and how to get help == <!--T:16-->
* To know how to use different archiving and compression utilities, use a Linux command like <code>man <command></code> or <code><command> --help</code>.  
* Contact our [[technical support]]
 
</translate>

Latest revision as of 16:40, 27 November 2023

Other languages:

This page explains issues related to transferring your data between our facilities and our regional partners.

If you are in any doubt about details of the following advice, contact our technical support for help.

What to do before migrating ?[edit]

Make sure you know whether you are responsible for your own data migration, or whether our staff will be migrating your data. If you are in any doubt, contact our technical support.

If you haven't used Globus before, read about it now and verify that it works on the system you are migrating from. Test any other tools you will use (like tar, gzip, zip) on test data to ensure you know how they work before using them on important data.

Do not wait until the last minute to start your migration. Depending on how much data you have and how much load there is on the machines and network, you may be surprised at how long it will take to finish a large transfer. Expect hundreds of gigabytes to take hours to transfer, but give yourself days in case there is a problem. Expect terabytes to take days.

Clean up[edit]

It is a good practice to look at your files regularly and see what can be deleted, but unfortunately many of us do not have this habit. A major data migration is a good reminder to clean up your files and directories. Moving less data will take less time, and storage space even on new systems is in great demand and should not be wasted.

  • If you compile programs and keep source code, delete any intermediate files. One or more of make clean, make realclean, or rm *.o might be appropriate, depending on your makefile.
  • If you find any large files named like core.12345 and you don't know that they are, they are probably core dumps and can be deleted.

Archive and compress[edit]

Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use tar to combine (archive) them.

Large files can benefit from compression in some cases, especially text files which can usually be compressed a great deal. Compressing a file only for the purpose of transferring it, and then decompressing it at the end of the transfer will not necessarily save time. It depends on how much the file can be compressed, how long it takes to compress it, and the transfer bandwidth. The calculation is described under Data Compression and transfer discussion in this document from the US National Center for Supercomputing Applications.

If you decide compression is worthwhile, you can again use tar for this, or gzip.

Avoid duplication[edit]

Try not to move the same data twice. If you are migrating from more than one existing system to one new system and you have data duplicated on the sources, choose one and only move the duplicate data from that one.

Beware of files with duplicate names, but which do not contain duplicate information. Ensure that you will not accidentally overwrite one file with another of the same name.

What to do during the migration process[edit]

If it is supported at your source site, use Globus to set up your file transfer. It is the most user-friendly and efficient tool we know for this task. Globus is designed to recover from network interruptions automatically. We recommend you enable the setting to preserve source file modification times in the Transfer & Timer Options.

If Globus is not supported at your source site, then compressing data and avoiding duplication is even more important. If you are using scp, sftp, or rsync, then:

  • Make a schedule to migrate your data in blocks of a few hundreds of GBs at a time. If the transfer stops for some reason, you will be able to try again starting from the incomplete file, but you will not have to re-transfer files that are already complete. An organized list of files will help here.
  • Check regularly to see that the transfer process has not stopped. File size is a good indicator of progress. If no files have changed size for several minutes, then something may have gone wrong. If restarting the transfer does not work, contact our technical support.

Be patient. Even with Globus, transferring large volumes of data can be time consuming. Specific transfer speeds will vary, but expect hundreds of gigabytes to take hours and terabytes to take days.

What to do after migration[edit]

If you did not use Globus, or if you did but did not check verify file integrity, make sure that the data you have transferred are not corrupted. A crude way to do this is to compare file sizes at the source with file sizes at the destination. For greater assurance, you can use cksum or md5sum at each end, and see if the results match. Any files with mismatching sizes or checksums should be transferred again.

Where and how to get help[edit]

  • To know how to use different archiving and compression utilities, use a Linux command like man <command> or <command> --help.
  • Contact our technical support

Globus is a file transfer service

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=General_directives_for_migration&oldid=147537"