General directives for migration

This article is a draft

This is not a complete article: This is a draft, a work in progress that is intended to be published into an article, which may or may not be ready for inclusion in the main wiki. It should not necessarily be considered factual or authoritative.

This page is dedicated to users of Compute Canada clusters concerned by the process of data migration. It is also useful for other users since it explains how to proceed when it comes to transfer your data from one place to another between Compute Canada facilities and its regional partners (ACENET, Calcul Quebec, Compute Ontario and WestGrid). You will find the best practices and useful links related to [Archiving] and [Transfer] or data migration. In this page, you will find some tips, instructions and links on how to prepare your data and archives to facilitate the migration process. Cleaning your directories and archiving your data is part of migration process. Here, we give you some information and tips on what to do before, during and after the migration process.

What to do before the migration starts?

Test any tools you will use (like tar, gzip, zip, or Globus) on small test data so you know how they work before starting a large transfer. If you are in any doubt about details of the following advice, contact support for help.

Clean up

It is a good practice to look at your files regularly and see what can be deleted, but unfortunately many of us do not have the habit. A major data migration is a good reminder to clean up your files and directories. Moving less data will take less time, and storage space even on new systems is in great demand and should not be wasted.

If you compile programs and keep source code, delete any intermediate files. One or more of make clean, make realclean, or rm *.o might be appropriate, depending on your makefile.
If you find any large files named like core.12345 and you don't know that they are, they are probably core dumps and can be deleted.

Compress and archive

Most file transfer programs move one file of a reasonable size more efficiently than thousands of small files of equal total size. If you have directories or directory trees containing many small files, use tar or zip to combine (archive) and compress them.

Large files can also benefit from compression in many cases, especially text files or numeric data stored as human-readable text. You can use again use tar for this, or gzip, or zip.

Avoid duplication

Try not to move the same data twice. If you are migrating from more than one existing system to one new system and you have data duplicated on the two sources, choose one and only move the duplicate data from that one.

Beware of files with duplicate names, but which do not contain duplicate information. Ensure that you will not accidentally over-write one file with another of the same name.

What to do during the migration process?

Be patient. Migrating data from one site to another can be long and time consuming. Depending on the amount of data you have and how many users are going to migrate their data, this process can be scheduled over few days.
if you have a huge amount of data to transfer, do not stay till the last minute to start the migration. Depending on how much data you have, it can take a while to finish the transfer. This is another reason to prepare archives before migrating your data.
Once your directories are cleaned and your data compressed, you can start the migration process to the new facilities using the Globus file transfer protocol.
Do not try to migrate the whole data at once. Depending on the number of users and the amount of the data to migrate, the file system can slow down and stops. If it happens, the migration process will take longer.
Make a schedule to migrate your data part by part. With this strategy, you can control what data you migrate. If for any reason, the system stops or your connection interrupted you will be able to try again later to migrate the same data instead of starting again if you tried to migrate your whole data or directories at once.
Make sure to check out that the process did not stop after you started the migration. For this reason, it is highly recommended to migrate your data part by part.
Be sure that you did not miss anything of your data if the migration process requires more time. It may be necessary to create a new directory with a name like Data_Migrated for example. The idea is to move any data you already migrated to this directory when you are sure that the data is in the destination cluster. The next time, you want to continue with the migration process, you will just look at the data outside this directory and you will not ask yourself or have to check again in the destination directory if your data have already been transferred or not. You may also keep records of the data you moved to see what is left to migrate and make a schedule to do it within the period dedicated for the migration (sooner is better than waiting to the last minute).
Make sure that the data you are about to transfer are not corrupted. This can be achieved by different means (compare the file size on your local directory and the destination directory, try to check your data if they are not altered or corrupted during the migration process. To do so, you can look at your data using some utilities (to learn more [this page]).

What to do after migration?

Connect to the remote machine and check out that your data is there and compare the size of your archives between the initial and final destination. If there is a problem during the migration, the system may copy part of the archive and not the whole file. You may see the archive name but its size will be different from the original one. In this case, the data may be corrupted and you will not be able to retrieve all your data. You have to transfer again this particular file or archive. For more details about how to check if your data is not corrupted during the migration process can be found on the following link: [Archiving and Compressing Data for Migration].

Organizer your data on your new working directories on the new clusters.

Where and how to get HELP?

To know how to use different archiving and compression utilities, use the Linux command like man <command> or <command> --help. Most common commands and useful examples can be found on the Archiving and Compressing page.

Ask around you for more help and clarification.
Contact Migration support team by email: (no email provided yet)