Scratch purging policy: Difference between revisions

Jump to navigation Jump to search

Scratch purging policy (view source)

Revision as of 18:46, 1 March 2018

1 byte added ,  6 years ago
no edit summary
No edit summary
Line 5: Line 5:
The scratch filesystem on Compute Canada clusters is intended as temporary, fast storage for data being used during job execution.  Data needed for long term storage and reference should be kept in either [[Project layout|<tt>/project</tt>]] or other archival storage areas. In order to ensure adequate space on scratch older files are periodically deleted according to the policy outlined in this page. The threshold for purging is 60 days, which is a little more than twice the maximum duration of a job on the cluster.
The scratch filesystem on Compute Canada clusters is intended as temporary, fast storage for data being used during job execution.  Data needed for long term storage and reference should be kept in either [[Project layout|<tt>/project</tt>]] or other archival storage areas. In order to ensure adequate space on scratch older files are periodically deleted according to the policy outlined in this page. The threshold for purging is 60 days, which is a little more than twice the maximum duration of a job on the cluster.


The determination of a file's age is done by checking the most recent value between the access time (<tt>atime</tt>) and change time (<tt>ctime</tt>) of the file. You can find the <tt>ctime</tt> of a file using
=Expiration procedure=
{{Command|ls -lc <filename>}}
while the <tt>atime</tt> can be obtained with the command
{{Command|ls -lu <filename>}}
We do not use the modify time (<tt>mtime</tt>) of the file for age determination because it can be modified by the user or other programs to display incorrect information.


Ordinarily simple use of the <tt>atime</tt> property would be sufficient, as the it is updated by the system in sync with the <tt>ctime</tt>, however userspace programs are also able to alter the atime, potentially to times in the past, which could result in early expiration of a file - the fallback of the ctime property prevents this undesirable behaviour.
The scratch filesystem is checked at the end of the month for files which will be candidates for expiry on the 15th of the following month. On the first day of the month, a notification e-mail is sent to all users who have at least one file which is a candidate for purging and containing the location of a file which lists all the candidates for purging. You will thus have two weeks to make arrangements to move data to your project space or some other location if you wish to save the data in question.  


=Expiration Procedure=
On the 12th of the month, a final notification e-mail will be sent with an updated assessment of candidate files for expiration on the 15th, giving you 72 hours to make arrangements for moving these files. At the end of day on the 15th any remaining files on the scratch filesystem for which both the <tt>ctime</tt> and the <tt>atime</tt> are older than 60 days will be deleted.


The scratch filesystem is checked at the end of the month for files which will be candidates for expiry on the 15th of the following month. On the first day of the month, a notification e-mail is sent to all users who have at least one file which is a candidate for purging and containing the location of a file which lists all the candidates for purging. You will thus have two weeks to make arrangements to move data to your project space or some other location if you wish to save the data in question.  
Note that simplying copying or using the <tt>rsync</tt> command to displace your files will update the <tt>atime</tt> for the original data on scratch, making them ineligible for deletion. Once you have put the data in another location please delete the original files and directories in scratch, rather than depending on the automatic purging.


On the 12th of the month, a final notification e-mail will be sent with an updated assessment of candidate files for expiration on the 15th, giving you 72 hours to make arrangements for moving these files. At the end of day on the 15th any remaining files on the scratch filesystem with the property that the minimum of their <tt>ctime</tt> and <tt>atime</tt> is greater than 60 days will be deleted.
= How do I check the age of a file? =
We define a file's age as the most recent of:
* the access time (<tt>atime</tt>) and
* the change time (<tt>ctime</tt>).
You can find the <tt>ctime</tt> of a file using
{{Command|ls -lc <filename>}}
while the <tt>atime</tt> can be obtained with the command
{{Command|ls -lu <filename>}}
We do not use the modify time (<tt>mtime</tt>) of the file because it can be modified by the user or other programs to display incorrect information.  


Note that simplying copying or using the <tt>rsync</tt> command to displace your files will update the <tt>atime</tt> for the original data on scratch, so that once you have put the data in another location be sure to delete the original files and directories in scratch ahead of the automatic purging.
Ordinarily simple use of the <tt>atime</tt> property would be sufficient, as it is updated by the system in sync with the <tt>ctime</tt>. However userspace programs are able to alter <tt>atime</tt>, potentially to times in the past, which could result in early expiration of a file. The use of <tt>ctime</tt> as a fallback guards against this undesirable behaviour.


=Abuse=
=Abuse=
This method of tracking file age does allow for potential abuse the system by periodically running a recursive <tt>touch</tt> command on your files, preventing them from being flagged for expiration. Compute Canada staff have methods for detecting this and similar tactics to circumvent the purging policy. Users who employ such techniques will be contacted and asked to modify their behaviour, in particular to move the "retouched" data from scratch to a more appropriate location.
This method of tracking file age does allow for potential abuse the system by periodically running a recursive <tt>touch</tt> command on your files, preventing them from being flagged for expiration. Compute Canada staff have methods for detecting this and similar tactics to circumvent the purging policy. Users who employ such techniques will be contacted and asked to modify their behaviour, in particular to move the "retouched" data from scratch to a more appropriate location.
Rdickson
Bureaucrats, cc_docs_admin, cc_staff
2,879

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/47594"

Navigation menu