Bureaucrats, cc_docs_admin, cc_staff
2,879
edits
Using nearline storage: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 8: | Line 8: | ||
This is useful because the capacity of our tape libraries is both large and expandable. When a file has been moved to tape (or ''virtualized''), it still appears in the directory listing. If the virtual file is read, the reading process will block for some time, probably a few minutes, while the file contents is copied from tape to disk. | This is useful because the capacity of our tape libraries is both large and expandable. When a file has been moved to tape (or ''virtualized''), it still appears in the directory listing. If the virtual file is read, the reading process will block for some time, probably a few minutes, while the file contents is copied from tape to disk. | ||
== | == Using nearline == <!--T:3--> | ||
Because of the delay in reading from tape, nearline is not intended to be used by jobs where allocated time would be wasted. It is only accessible as a directory on certain nodes of the clusters, but never on compute nodes. | Because of the delay in reading from tape, nearline is not intended to be used by jobs where allocated time would be wasted. It is only accessible as a directory on certain nodes of the clusters, but never on compute nodes. | ||
| Line 16: | Line 16: | ||
*Files larger than 300GB should be split in chunks of 100GB using the [[A_tutorial_on_'tar'#split|split]] command or a similar tool. | *Files larger than 300GB should be split in chunks of 100GB using the [[A_tutorial_on_'tar'#split|split]] command or a similar tool. | ||
<!--T:22--> | |||
The basic model for using nearline is that you put files there, and later you may access them, like a normal filesystem, except that reading the files may involve a significant pause. You may also remove files from nearline. It's important to realize that nearline files can be in several different states: | The basic model for using nearline is that you put files there, and later you may access them, like a normal filesystem, except that reading the files may involve a significant pause. You may also remove files from nearline. It's important to realize that nearline files can be in several different states: | ||
* Immediately upon creation, the file is on disk, not tape. | * Immediately upon creation, the file is on disk, not tape. | ||