Translations:Storage and file management/9/en: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
(Importing a new version from external source)
 
(Importing a new version from external source)
Line 1: Line 1:
== Best practices ==
== Best practices ==
* Only use text format for files that are smaller than a few megabytes.
* Only use text format for files that are smaller than a few megabytes.
* As far as possible, use local storage for temporary files.
* As far as possible, use local storage for temporary files. It is best to use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>.
* If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
* If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible copy them elsewhere.
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible copy them elsewhere.
* If your needs are not well served by the available storage options please contact us by sending an e-mail to [mailto:support@computecanada.ca Compute Canada support].
* If your needs are not well served by the available storage options please contact us by sending an e-mail to [mailto:support@computecanada.ca Compute Canada support].

Revision as of 19:48, 25 August 2017

Information about message (contribute)
This message has no documentation. If you know where or how this message is used, you can help other translators by adding documentation to this message.
Message definition (Storage and file management)
== Best practices ==
* Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
* Only use text format for files that are smaller than a few megabytes.
* As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the [[Running jobs|job scheduler]] for this, named <code>$SLURM_TMPDIR</code>.
* If your program must search within a file, it is fastest to do it by first reading it completely before searching.
* If you no longer use certain files but they must be retained, [[Archiving and compressing files|archive and compress]] them, and if possible move them to an alternative location like [[Using nearline storage|nearline]].
* For more on managing many files, see [[Handling large collections of files]], especially if you are limited by a quota on the number of files. 
* Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as [https://en.wikipedia.org/wiki/Message_Passing_Interface#I/O MPI-IO]. 
* If your needs are not well served by the available storage options please contact [[technical support]].

Best practices

  • Only use text format for files that are smaller than a few megabytes.
  • As far as possible, use local storage for temporary files. It is best to use the temporary directory created by the job scheduler for this, named $SLURM_TMPDIR.
  • If your program must search within a file, it is fastest to do it by first reading it completely before searching, or to use a RAM disk.
  • Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
  • If you no longer use certain files but they must be retained, archive and compress them, and if possible copy them elsewhere.
  • If your needs are not well served by the available storage options please contact us by sending an e-mail to Compute Canada support.
Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Translations:Storage_and_file_management/9/en&oldid=37312"