rsnt_translations
56,420
edits
Using nearline storage: Difference between revisions
no edit summary
(Marked this version for translation) |
No edit summary |
||
| Line 3: | Line 3: | ||
==Nearline is a filesystem virtualized onto tape== <!--T:1--> | ==Nearline is a filesystem virtualized onto tape== <!--T:1--> | ||
Nearline storage is a disk-tape hybrid filesystem with a layout like [[Project layout|Project]], except that it can virtualize files by moving them to tape-based storage on criteria | Nearline storage is a disk-tape hybrid filesystem with a layout like [[Project layout|Project]], except that it can virtualize files by moving them to tape-based storage on criteria such as age and size, and then back again upon read or recall operations. This is a way to manage less-used files. On tape, the files do not consume your disk quota, but they can still be accessed, albeit slower than with the home, scratch and project filesystems. | ||
<!--T:2--> | <!--T:2--> | ||
| Line 13: | Line 13: | ||
<!--T:9--> | <!--T:9--> | ||
Nearline is intended for use with relatively large files and should not be used for a large number of small files. In fact, files smaller than a certain threshold size may not be moved to tape at all. | Nearline is intended for use with relatively large files and should not be used for a large number of small files. In fact, files smaller than a certain threshold size may not be moved to tape at all. | ||
*Files smaller than ~200MB should be combined into archive files (''tarballs'') using [[Archiving and compressing files|tar]] or a similar tool. | *Files smaller than ~200MB should be combined into archive files (''tarballs'') using [[Archiving and compressing files|<tt>tar</tt>]] or a similar tool. | ||
*Files larger than 300GB should be split in chunks of 100GB using the [[Archiving_and_compressing_files#Other_Useful_Utilities|split]] command or a similar tool. | *Files larger than 300GB should be split in chunks of 100GB using the [[Archiving_and_compressing_files#Other_Useful_Utilities|split]] command or a similar tool. | ||
== Using | == Using nearline == <!--T:22--> | ||
The basic model for using nearline is that you put files there, and later you may access them, like a normal filesystem, except that reading the files may involve a significant pause. | The basic model for using nearline is that you put files there, and later you may access them, like a normal filesystem, except that reading the files may involve a significant pause. You may also remove files from nearline. It's important to realize that nearline files can be in several different states: | ||
* immediately upon creation, the file is on disk, not tape. | * immediately upon creation, the file is on disk, not tape. This is mainly to avoid having to guess about when you're done putting the file into place. | ||
* after a period (on the order of a day), the system will copy the file to tape. | * after a period (on the order of a day), the system will copy the file to tape. At this stage, the file will be on both disk and tape; it will behave just like a disk file, unless you modify it. | ||
* after a further period, the disk copy will be dropped, and the file will only be on tape (our policy is two tape copies: one local and one remote). | * after a further period, the disk copy will be dropped, and the file will only be on tape (our policy is two tape copies: one local and one remote). At this point, the file will be slow to read, since content must be recalled from tape. | ||
* when such a file is recalled, it returns to the second state. | * when such a file is recalled, it returns to the second state. | ||
<!--T:23--> | <!--T:23--> | ||
Thus, you may wish to inquire about the status of a | Thus, you may wish to inquire about the status of a nearline file: | ||
lfs hsm_state ~/nearline/def-mine/user/somefile | lfs hsm_state ~/nearline/def-mine/user/somefile | ||
will report on the HSM status of the file. If you wish to ensure that the file is brought in from tape, you can also | will report on the HSM status of the file. If you wish to ensure that the file is brought in from tape, you can also use | ||
lfs hsm_restore | lfs hsm_restore | ||