Using nearline storage: Difference between revisions

Nearline storage is  a disk-tape hybrid filesystem with a layout like [[Project layout|Project]], except that it may virtualize files by moving them to tape-based storage on criteria like age and size, and then back again upon read or recall operations. This is a way to manage less used files. On tape, the files do not consume your disk quota, but they can still be accessed, albeit more slowly.

This is useful because the capacity of our tape libraries is both large and expandable.  When a file has been moved to tape (or ''virtualized''), it still appears in the directory listing.  If the virtual file is read, the reading process will block for some time, probably a few minutes, while the file contents is read from tape to disk.   

Because of the delay in reading from tape, Nearline is not intended to be used by jobs where allocated time would be wasted.  It is only accessible as a directory on certain nodes of the clusters, and in particular, not on the compute nodes.  

To use Nearline, just put files into your <tt>~/nearline/PROJECT</tt> directory. After a period of time (currently 24 hours), they'll be copied onto tape.  If the file remains unchanged for another period (also 24h), the copy on disk will be removed, making the file virtualized on tape.  

2. For small data management of files in HPSS, you can use the VFS (''Virtual File System'') node, which is accessed with the command <tt>salloc --time=1:00:00 -pvfsshort</tt>.