Alliance Doc

Using nearline storage: Difference between revisions

Page Discussion

Using nearline storage (view source)

Revision as of 19:26, 29 September 2020

123 bytes removed ,  4 years ago
major revision following Sep 28 S2S seminar
(Marked this version for translation)
(major revision following Sep 28 S2S seminar)
Line 1: Line 1:
<languages />
<languages />
<translate>
<translate>
<!--T:30-->
<!--T:30-->
The nearline storage space is tape-based and designed to store '''large''' files which are '''infrequently accessed'''.
Nearline is a tape-based file system intended for *inactive data*.  Data sets which you do not expect to access for months are good candidates to be stored in nearline.
 
= Best practices, and restrictions =
 
==== Size of files ====
 
Retrieving small files from tape is inefficient, while extremely large files pose other problems.  Please observe these guidelines about the size of files to store in nearline:
 
<!--T:9-->
*Files smaller than ~200MB should be combined into archive files (''tarballs'') using [[A tutorial on 'tar'|tar]] or a [[Archiving and compressing files|similar tool]].
*Files larger than 300GB should be split in chunks of 100GB using the [[A_tutorial_on_'tar'#split|split]] command or a similar tool.
 
==== Using tar or dar ====
 
Use [[A tutorial on 'tar'|tar]] or [[dar]] to create an archive file directly on nearline.  There is no advantage to creating the archive on a different filesystem and then copying it to nearline once complete.
 
If you have hundreds of gigabytes of data, the <code>tar</code> options <code>-M (--muti-volume)</code> and <code>-L (--tape-length)</code> can be used to produce archive files of suitable size.
 
If you are using <code>dar</code>, you can similarly use the <code>-s (--slice)</code> option.
 
==== No access from compute nodes ====
 
Because data retrieval from nearline may take an uncertain amount of time (see "How it works" below), we do not permit reading from nearline in a job context.  Nearline is not mounted on compute nodes.


<!--T:31-->
==== Use a data-transfer node if available ====
The nearline filesystem is only accessible from the login and data-transfer nodes of the cluster (not compute nodes).  If you have many and/or small files, you should use a tool like [[Tar]] or [[Dar]] to aggregate them into large archive files.


<!--T:32-->
<!--T:32-->
Because such operations can be resource-intensive, they may not be appropriate on login nodes so use data-transfer nodes,
Creating a tar or dar file for a large volume of data can be resource-intensive.  Please do this on a data-transfer node (DTN) instead of a login node if login to a DTN is supported at the cluster you are using.
or create the archive files on another filesystem before moving to nearline.
 
= Why nearline? =


==Nearline is a filesystem virtualized onto tape== <!--T:1-->
Tape as a storage medium has these advantages over disk and solid-state ("SSD") media.
Nearline storage is a hybrid disk/tape filesystem with a layout like [[Project layout|Project]], which however uses its hybrid nature to take advantage of both the the large amount of inexpensive storage available by tape and the rapid access to data that disk offers. You can move your less frequently needed data to tape, where they will no longer count against your project space quota. If you later need these files, you can recall them from tape back to disk with a delay ranging from a few minutes up to an hour or two.
# Cost per unit of data stored is lower.
# The volume of data stored can be easily expanded by buying more tapes.
# Energy consumption per unit of data stored is effectively zero.
 
Consequently we can offer much greater volumes of storage on nearline than we can on project.  Also, keeping inactive data ''off'' of project reduces the load and improves its performance.
 
= How it works =
 
<!--T:22-->
# When a file is first copied to (or created on) nearline, the file exists only on disk, not tape.
# After a period (on the order of a day), and if the file meets certain criteria, the system will copy the file to tape. At this stage, the file will be on both disk and tape.
# After a further period the disk copy may be deleted, and the file will only be on tape.
# When such a file is recalled, it is copied from tape back to disk, returning it to the second state.


<!--T:2-->
<!--T:2-->
This is useful because the capacity of our tape libraries is both large and expandable.  When a file has been moved to tape or ''virtualized'', it will still appear in the directory listing.  If the virtual file is read, the reading process will block for some time, probably a few minutes, while the file's contents are recalled from tape to disk.
When a file has been moved entirely to tape (that is, when it is ''virtualized'') it will still appear in the directory listing.  If the virtual file is read, it will take some time for the tape to be retrieved from the library and copied back to disk. The process which is trying to read the file will block while this is happening.  This may take from less than a minute to over an hour, depending on the size of the file and the demand on the tape system.


<!--T:24-->
<!--T:24-->
You can determine whether or not a given file has been moved to tape or is still on disk using the <code>lfs hsm_state</code> command:
You can determine whether or not a given file has been moved to tape or is still on disk using the `lfs hsm_state` command.  The "hsm" stands for "hierarchical storage manager".
 
<source lang="bash">
#  Here, <FILE> has not been copied to tape.
$ lfs hsm_state <FILE>
<FILE>: (0x00000000)


<!--T:25-->
<!--T:25-->
<source lang="bash">
# Here, <FILE> is still on the disk
# Here, <FILE> is still on the disk
$ lfs hsm_state <FILE>
$ lfs hsm_state <FILE>
Line 33: Line 72:


<!--T:27-->
<!--T:27-->
The "hsm" used in this command stands for "hierarchical storage manager".  When you attempt to read a file which is on tape, the system will automatically copy it back from tape to disk; this may take a few minutes or even a few hours if the tape system is very busy. If you wish to explicitly force a file to be recalled from tape, you can use the <code>lfs hsm_restore <FILE></code> command.
You can explicitly force a file to be recalled from tape without actually reading it with the command <code>lfs hsm_restore <FILE></code>.


<!--T:29-->
<!--T:29-->
Note that as of July 2020, the output of the command <code>diskusage_report</code> does not provide reliable values about the consumption of nearline space for the clusters [[Graham]] and [[Béluga/en|Béluga]].
Note that as of October 2020, the output of the command <code>diskusage_report</code>, also known as <code>quota</code>, does not report on nearline space consumption.


== Using nearline == <!--T:3-->
== Site-specific information == <!--T:6-->
Because of the delay in reading from tape, nearline is not intended to be used by jobs where allocated time would be wasted.  It is only accessible as a directory on certain nodes of the clusters, but never on compute nodes.
 
<!--T:9-->
Nearline is intended for use with relatively large files and should not be used for a large number of small files.  In fact, files smaller than a certain threshold size may not be moved to tape at all.
*Files smaller than ~200MB should be combined into archive files (''tarballs'') using [[A tutorial on 'tar'|tar]] or a [[Archiving and compressing files|similar tool]].
*Files larger than 300GB should be split in chunks of 100GB using the [[A_tutorial_on_'tar'#split|split]] command or a similar tool.
 
<!--T:22-->
The basic model for using nearline is that you put files there, and later you may access them, like a normal filesystem, except that reading the files may involve a significant pause. You may also remove files from nearline.  It's important to realize that nearline files can be in several different states:
* Immediately upon creation, the file is on disk, not tape.
* After a period (on the order of a day), the system will copy the file to tape. At this stage, the file will be on both disk and tape; it will behave just like a disk file, unless you modify it.
* After a further period, the disk copy will be dropped, and the file will only be on tape (our policy is two tape copies: one local and one remote).  At this point, the file will be slow to read, since content must be recalled from tape.
* When such a file is recalled, it returns to the second state.
 
== Access == <!--T:6-->


<!--T:10-->
<!--T:10-->
Line 62: Line 86:
<!--T:11-->
<!--T:11-->
To use nearline, just put files into your <tt>~/nearline/PROJECT</tt> directory. After a period of time (24 hours as of February 2019), they will be copied onto tape. If the file remains unchanged for another period (24 hours as of February 2019), the copy on disk will be removed, making the file virtualized on tape.  
To use nearline, just put files into your <tt>~/nearline/PROJECT</tt> directory. After a period of time (24 hours as of February 2019), they will be copied onto tape. If the file remains unchanged for another period (24 hours as of February 2019), the copy on disk will be removed, making the file virtualized on tape.  


<!--T:8-->
<!--T:8-->
Rdickson
Bureaucrats, cc_docs_admin, cc_staff
2,879

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/90255"