Tar: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
(former content copied to docs-dev)
Tag: New redirect
 
(72 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Draft}}
#REDIRECT [[A tutorial on 'tar']]
 
== This page is a draft ... Work in progress ==
 
== How to prepare archives of your data before migration process? ==
Before starting the Data Migration from one place to another or between Compute Canada clusters or its regional partners, the users should start by preparing and archiving their Data in order to facilitate the migration process and improve the efficiency of the whole process.  It is easier for the secure copy protocol to migrate one archive file of a reasonable size than migrating thousands of small files. To avoid any interruption or slowing down of the file system that can affect the migration process, it is recommended to transfer archives rather than the whole directory with all files and directories individually especially the directories with large number of files. In this page, you can find examples and more practical commands to prepare your archives and save time and increase the efficiency of the file transfer.
 
=== Why should you archive and compress your files? ===
 
Archiving your files reduces the total number of files you need to migrate and gives you more control of your data when it comes to move them from one place to another. In addition, the compression will reduce the space you need for your data and will speed up the migration process.
 
=== What is an archive or a tar-file? ===
 
An archive puts many files or directories together into a single file on your disk. You can restore individual files from the archive or the whole archive on any Unix-like system (Unix, Linux, MacOS, Cygwin). To create archives, some utilities like <code>tar</code> are used.
 
=== How do you archive files with the tar command? ===
 
The primary archiving utility is the <code>tar</code> command that bundles a bunch of files or directories together and generates a single file or archive file with ['''.tar'''] as the extension. This command is an archiving utility designed to store and extract files from an archive file known as a tar-file: ['''your_archive_name.tar'''].
Depending on the options and arguments invoked (see below), this command can create an archive, add files to an existing archive, list the content of an archive, extract one or more files, or extract the whole archive, etc ... The use of a directory name always implies that all the sub-directories below should be included in the final archive except if the command is invoked with <code>--exclude</code> option that will not take into account the specified files when the archive is created.
 
=== How to compress/uncompress tar files? ===
 
Once the archive is created ['''your_archive.tar'''], it is possible to use the compression commands to reduce the size of the tar file. This can be done by using one of the compression algorithms that are installed on each Linux or MacOs environment:
* '''<code>gzip</code>''': this utility or command is used to compress any file or any archive file ['''your_file''' or '''your_archive_name.tar'''] and produce the output file: ['''your_file.gz''' or '''your_archive_name.tar.gz'']. The command <code>gunzip</code> can be used later to uncompress and retrieve your original file or archive file: ['''your_file''' or '''your_archive_name.tar'''].
* '''<code>bzip2</code>''': is another utility that can be used to compress any file or any archive file: ['''your_file''' or '''your_archive_name.tar'''] and produce the output file: ['''your_file.bz2''' or '''your_archive_name.tar.bz2''']. The command <code>bunzip2</code> can be invoked later to uncompress and retrieve the original file or archive: ['''your_file''' or '''your_archive_name.tar'''].
 
'''Note:''' Condensed commands can combine at the same time <code>tar</code> and <code>gzip</code> (or <code>bzip2</code>); or <code>tar</code> and <code>gunzip</code> (or <coce>bunzip2</code>). For users who are not very familiar with the combined commands, it is possible to create your compressed archives and retrieve your original files in two steps:
 
# '''Compressing:''' <code>tar</code> ['''some-adequate-options'''] followed by <code>gzip</code> or <code>bzip2</code>.
# '''Uncompressing:''' <code>gunzip</code> (or <code>bunzip2</code>) followed by <code>tar</code> ['''some-adequate-options'''].
 
These archiving utilities are invoked with some options and arguments. For more details on how to use these utilities, you can type on your terminal: <code>man <command></code>.
 
The general syntax for <code>tar</code>, <code>gzip</code>, <code>gunzip</code>, <code>bzip2</code> and <code>bunzip2</code> is as follow:
* <code>tar ['''option(s)'''] ['''your_file.tar''' or '''your_archive_name.tar'''] ['''filename(s), directory or directories''']</code>
* <code>gzip ['''your_file or your_archive_name.tar''']</code>
* <code>gunzip ['''your_file.gz''' or '''your_archive_name.tar.gz''']</code>
* <code>bzip ['''your_file''' or '''your_archive_name.tar''']</code>
* <code>bunzip2 ['''your_file.bz2''' or '''your_archive_name.tar.bz2''']</code> 
 
Let us mention that:
* <code>gunzip</code> is only used to uncompress files with '''gz''' extension.
* <code>bunzip2</code> is only used to uncompress files with '''bz2''' extension.
 
These are the most common options for tar command:
* <code> -c </code>: {option is used to create a new archive.}
* <code> -v </code>: {verbosely list files which are processed.}
* <code> -f </code>: {following is the archive file name.}
* <code> -t </code>: {list the content of an archive file.}
* <code> -r </code>: {to add files an existing archive.}
* <code> -A </code>: {to append an archive at the end on another.}
* <code> -x </code>: {extract files from archive.}
* <code> -z </code>: {filter the archive through <code>gzip</code>.}
* <code> -C </code>: {directory file: performs a chdir [change directory] operation on directory and performs the <code>c</code> (create) or <code>r</code> (replace) operation on file.}
 
== Common and useful commands to use to prepare your archives: ==
 
To illustrate the different commands and how to use archive utilities, we use a given directory that looks like a home directory or any other directory that contains files, sub-directories ... etc. Let us suppose that you have already cleaned and removed the data you do not need and your data is ready for migration. Before that, there is one more step which is to compress your data. In the following, you will find the most common use of archiving and compressing utilities with adequate options. As an example, we use one directory called here '''Migration''' (or whatever is the name of your directory) and see how we can apply the different archiving and compressing utilities.
On your terminal, change the directory to '''Migration''' (or the directory you want to work with) then: <br>
* Use pwd {present work directory} to see the current working path.
* Use ls {list} command to see the files and the sub-directories in the current working path.
* Use du -sh {disk usage} to see the size of the files, directories and sub-directories. This information will help you to see how to prepare your archives and which files to put together or to compress separately.
As shown in this example:
 
<source lang="console">
[user_name@localhost]$  pwd
/global/scratch/user_name/Migration
</source>
 
<source lang="console">
[user_name@localhost]$  ls
bin/  documents/  jobs/  new.log.dat  programs/  report/  results/  tests/  work/
</source>
 
<source lang="console">
[user_name@localhost]$  du -sh *
3,0K bin
876K documents
136K jobs
12K  new.log.dat
68K  programs
1,8M report
120K results
48K  tests
46K  work
</source>
 
This example shows that we are currently working on the directory called '''Migration''' and it contains the following files and directories ('''bin''', '''documents''', '''jobs''', '''new.log.dat''', '''programs''', '''report''', '''results''', '''tests''', '''work'''). The size of each file or directory is given by the use of the command <code>du -sh</code>. We will explain later why it is important to use this command to determine the size of your files or directories before starting compression.
 
In this example, we have used few directories and small files. In your case, you may have more directories, more files and large files. But the idea is the same and it consists on creating your archives using the <code>tar</code>, <code>gzip</code>, <code>bzip2</code> from your terminal. You can recover them later by <code>tar</code> {with specific options}, <code>gunzip</code> and <code>bunzip2</code>. We will explain how these utilities work by giving the most common and used commands and options.
 
'''Notes:'''
* Before starting compression, make sure you are not running out of space or quota because the tar command uses the free space to create the archive. At the end, it is like you have added data with the same size as the file or the directory you are trying to tar. When using tar, the original file stays without any change unless you make changes later or remove it.
* For gzip and bzip2, they also use some free space to create the final archive but in this case the new file you will get is '''your_file.gz''' if you use gzip or '''your_file.bz2''' if you use bzip2; if it is a tar file; you will get the new file '''your_archive.tar.gz''' or '''your_archive.tar.bz2'''
* The tar command can be applied to multiple files or directories in order to put them together into a final one file archive.
* The gzip and bzip2 are applied to a single file or a single archive file but not a directory.

Latest revision as of 19:24, 18 July 2019

Redirect to:

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Tar&oldid=74049"