cc_staff
82
edits
Scratch purging policy: Difference between revisions
Jump to navigation
Jump to search
m
heading levels
No edit summary |
m (heading levels) |
||
| Line 8: | Line 8: | ||
The scratch filesystem on our clusters is intended as temporary, fast storage for data being used during job execution. Data needed for long-term storage and reference should be kept in either [[Project layout|<code>/project</code>]] or other archival storage areas. In order to ensure adequate space on scratch, files older than 60 days are periodically deleted according to the policy outlined in this page. Note that the purging of a file is based on its age, not its location within scratch; simply moving a file from one directory in scratch to another directory in scratch will not in general prevent it from being purged. | The scratch filesystem on our clusters is intended as temporary, fast storage for data being used during job execution. Data needed for long-term storage and reference should be kept in either [[Project layout|<code>/project</code>]] or other archival storage areas. In order to ensure adequate space on scratch, files older than 60 days are periodically deleted according to the policy outlined in this page. Note that the purging of a file is based on its age, not its location within scratch; simply moving a file from one directory in scratch to another directory in scratch will not in general prevent it from being purged. | ||
=Expiration procedure= <!--T:2--> | ==Expiration procedure== <!--T:2--> | ||
<!--T:3--> | <!--T:3--> | ||
| Line 19: | Line 19: | ||
Note that simply copying or using the <code>rsync</code> command to displace your files will update the <code>atime</code> for the original data on scratch, making them ineligible for deletion. Once you have put the data in another location please delete the original files and directories in scratch instead of depending on the automatic purging. | Note that simply copying or using the <code>rsync</code> command to displace your files will update the <code>atime</code> for the original data on scratch, making them ineligible for deletion. Once you have put the data in another location please delete the original files and directories in scratch instead of depending on the automatic purging. | ||
=How/where to check which files are slated for purging= <!--T:13--> | ==How/where to check which files are slated for purging== <!--T:13--> | ||
<!--T:14--> | <!--T:14--> | ||
| Line 36: | Line 36: | ||
If you access/read/move/delete some or the candidates after the 12th, then you have to check yourself to confirm your files won't be purged on the 15th (see below) | If you access/read/move/delete some or the candidates after the 12th, then you have to check yourself to confirm your files won't be purged on the 15th (see below) | ||
= How do I check the age of a file? = <!--T:6--> | ==How do I check the age of a file?== <!--T:6--> | ||
We define a file's age as the most recent of: | We define a file's age as the most recent of: | ||
* the access time (<code>atime</code>) and | * the access time (<code>atime</code>) and | ||
| Line 49: | Line 49: | ||
Ordinarily, simple use of the <code>atime</code> property would be sufficient, as it is updated by the system in sync with the <code>ctime</code>. However, userspace programs are able to alter <code>atime</code>, potentially to times in the past, which could result in early expiration of a file. The use of <code>ctime</code> as a fallback guards against this undesirable behaviour. | Ordinarily, simple use of the <code>atime</code> property would be sufficient, as it is updated by the system in sync with the <code>ctime</code>. However, userspace programs are able to alter <code>atime</code>, potentially to times in the past, which could result in early expiration of a file. The use of <code>ctime</code> as a fallback guards against this undesirable behaviour. | ||
=Abuse= <!--T:8--> | ==Abuse== <!--T:8--> | ||
This method of tracking file age does allow for potential abuse by periodically running a recursive <code>touch</code> command on your files to prevent them from being flagged for expiration. Our staff have methods for detecting this and similar tactics to circumvent the purging policy. Users who employ such techniques will be contacted and asked to modify their behaviour, in particular to move the "retouched" data from scratch to a more appropriate location. | This method of tracking file age does allow for potential abuse by periodically running a recursive <code>touch</code> command on your files to prevent them from being flagged for expiration. Our staff have methods for detecting this and similar tactics to circumvent the purging policy. Users who employ such techniques will be contacted and asked to modify their behaviour, in particular to move the "retouched" data from scratch to a more appropriate location. | ||
=How to safely copy a directory with symlinks= <!--T:10--> | ==How to safely copy a directory with symlinks== <!--T:10--> | ||
<!--T:11--> | <!--T:11--> | ||