Translations:Storage and file management/9/en

From CC Doc
Jump to navigation Jump to search

Best practices

  • Regularly clean up your data in the scratch and project spaces, because those filesystems are used for huge data collections.
  • Only use text format for files that are smaller than a few megabytes.
  • As far as possible, use scratch and local storage for temporary files. For local storage you can use the temporary directory created by the job scheduler for this, named $SLURM_TMPDIR.
  • If your program must search within a file, it is fastest to do it by first reading it completely before searching.
  • If you no longer use certain files but they must be retained, archive and compress them, and if possible move them to an alternative location like nearline.
  • For more notes on managing many files, see Handling large collections of files, especially if you are limited by a quota on the number of files.
  • Having any sort of parallel write access to a file stored on a shared filesystem like home, scratch and project is likely to create problems unless you are using a specialized tool such as MPI-IO.
  • If your needs are not well served by the available storage options please contact technical support.