rsnt_translations
56,430
edits
Scratch purging policy: Difference between revisions
Jump to navigation
Jump to search
no edit summary
(remove reference to 30-day jobs) |
No edit summary |
||
| Line 6: | Line 6: | ||
<!--T:1--> | <!--T:1--> | ||
The scratch filesystem on | The scratch filesystem on our clusters is intended as temporary, fast storage for data being used during job execution. Data needed for long term storage and reference should be kept in either [[Project layout|<code>/project</code>]] or other archival storage areas. In order to ensure adequate space on scratch, files older than 60 days are periodically deleted according to the policy outlined in this page. Note that the purging of a file is based on its age, not its location within scratch; simply moving a file from one directory in scratch to another directory in scratch will not in general prevent it from being purged. | ||
=Expiration procedure= <!--T:2--> | =Expiration procedure= <!--T:2--> | ||
| Line 14: | Line 14: | ||
<!--T:4--> | <!--T:4--> | ||
On the 12th of the month, a final notification e-mail will be sent with an updated assessment of candidate files for expiration on the 15th, giving you 72 hours to make arrangements for moving these files. At the end of day on the 15th, any remaining files on the scratch filesystem for which both the < | On the 12th of the month, a final notification e-mail will be sent with an updated assessment of candidate files for expiration on the 15th, giving you 72 hours to make arrangements for moving these files. At the end of day on the 15th, any remaining files on the scratch filesystem for which both the <code>ctime</code> and the <code>atime</code> are older than 60 days will be deleted. Please remember that the e-mail reminders and login notice are a courtesy offered to our users, whose ultimate responsibility it is to keep files older than 60 days from being located in the scratch space. | ||
<!--T:5--> | <!--T:5--> | ||
Note that simply copying or using the < | Note that simply copying or using the <code>rsync</code> command to displace your files will update the <code>atime</code> for the original data on scratch, making them ineligible for deletion. Once you have put the data in another location please delete the original files and directories in scratch instead of depending on the automatic purging. | ||
=How/where to check which files are slated for purging= <!--T:13--> | =How/where to check which files are slated for purging= <!--T:13--> | ||
| Line 38: | Line 38: | ||
= How do I check the age of a file? = <!--T:6--> | = How do I check the age of a file? = <!--T:6--> | ||
We define a file's age as the most recent of: | We define a file's age as the most recent of: | ||
* the access time (< | * the access time (<code>atime</code>) and | ||
* the change time (< | * the change time (<code>ctime</code>). | ||
You can find the < | You can find the <code>ctime</code> of a file using | ||
{{Command|ls -lc <filename>}} | {{Command|ls -lc <filename>}} | ||
while the < | while the <code>atime</code> can be obtained with the command | ||
{{Command|ls -lu <filename>}} | {{Command|ls -lu <filename>}} | ||
We do not use the modify time (< | We do not use the modify time (<code>mtime</code>) of the file because it can be modified by the user or by other programs to display incorrect information. | ||
<!--T:7--> | <!--T:7--> | ||
Ordinarily, simple use of the < | Ordinarily, simple use of the <code>atime</code> property would be sufficient, as it is updated by the system in sync with the <code>ctime</code>. However, userspace programs are able to alter <code>atime</code>, potentially to times in the past, which could result in early expiration of a file. The use of <code>ctime</code> as a fallback guards against this undesirable behaviour. | ||
=Abuse= <!--T:8--> | =Abuse= <!--T:8--> | ||
This method of tracking file age does allow for potential abuse by periodically running a recursive < | This method of tracking file age does allow for potential abuse by periodically running a recursive <code>touch</code> command on your files to prevent them from being flagged for expiration. Our staff have methods for detecting this and similar tactics to circumvent the purging policy. Users who employ such techniques will be contacted and asked to modify their behaviour, in particular to move the "retouched" data from scratch to a more appropriate location. | ||
=How to safely copy a directory with symlinks= <!--T:10--> | =How to safely copy a directory with symlinks= <!--T:10--> | ||
<!--T:11--> | <!--T:11--> | ||
In most cases, < | In most cases, <code>cp</code> or <code>rsync</code> will be sufficient to copy data from scratch to project. But if you have symbolic links in scratch, copying them will cause problems since they will still point to scratch. To avoid this, you can use <code>tar</code> to make an archive of your files on scratch, and extract this archive in your project. You can do this in one go: | ||
<!--T:12--> | <!--T:12--> | ||