Translations:Storage and file management/9/en: Difference between revisions
Jump to navigation
Jump to search
(Importing a new version from external source) |
(Importing a new version from external source) |
||
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Best practices == | == Best practices == | ||
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections. | |||
* Only use text format for files that are smaller than a few megabytes. | * Only use text format for files that are smaller than a few megabytes. | ||
* As far as possible, use local storage for temporary files. | * As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>. | ||
* If your program must search within a file, it is fastest to do it by first reading it completely before searching | * If your program must search within a file, it is fastest to do it by first reading it completely before searching. | ||
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]]. | |||
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible | * For more on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files. | ||
* If your needs are not well served by the available storage options please contact | * Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. | ||
* If your needs are not well served by the available storage options please contact [[technical support]]. | |||
Latest revision as of 15:41, 17 September 2022
Best practices
- Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
- Only use text format for files that are smaller than a few megabytes.
- As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the job scheduler for this, named
$SLURM_TMPDIR. - If your program must search within a file, it is fastest to do it by first reading it completely before searching.
- If you no longer use certain files but they must be retained, archive and compress them, and if possible move them to an alternative location like nearline.
- For more on managing many files, see Handling large collections of files, especially if you are limited by a quota on the number of files.
- Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as MPI-IO.
- If your needs are not well served by the available storage options please contact technical support.