cc_staff
318
edits
No edit summary |
No edit summary |
||
Line 45: | Line 45: | ||
<!--T:6--> | <!--T:6--> | ||
The example above illustrates a frequent problem: <code>/project</code> for user <code>ymartin</code> contains too much data in files belonging to group <code>ymartin</code>. The data should instead be in files belonging to <code>def-zrichard</code>. | The example above illustrates a frequent problem: <code>/project</code> for user <code>ymartin</code> contains too much data in files belonging to group <code>ymartin</code>. The data should instead be in files belonging to <code>def-zrichard</code>. To see the project groups you may use, run the following command: | ||
stat -c %G $HOME/projects/*/ | |||
<!--T:8--> | <!--T:8--> | ||
Line 59: | Line 60: | ||
* files were moved from <code>/home</code> to <code>/project</code> with the <code>mv</code>command; to avoid this, see [[#Moving files between scratch and home filesystems | advice above]]; | * files were moved from <code>/home</code> to <code>/project</code> with the <code>mv</code>command; to avoid this, see [[#Moving files between scratch and home filesystems | advice above]]; | ||
* files were transferred from another cluster using [[Transferring_data#Rsync|rsync]] or [[Transferring_data#SCP|scp]] with an option to preserve the original group ownership. If you have a recurring problem with ownership, check the options you are using with your file transfer program; | * files were transferred from another cluster using [[Transferring_data#Rsync|rsync]] or [[Transferring_data#SCP|scp]] with an option to preserve the original group ownership. If you have a recurring problem with ownership, check the options you are using with your file transfer program; | ||
* you have no <tt>setgid</tt> bit set on your Project folders. | * you have no <tt>setgid</tt> bit set on your Project folders. | ||
=== How to fix the problem === | |||
If you already have data in your <code>/project</code> directory with the wrong group ownership, you can use the <code>find</code> command in conjunction with <code>readlink</code> to find and display those files: | |||
lfs find ~/projects/*/ -group $USER | |||
If the output of <code>diskusage_report</code> indicates that a different group is over quota, use that group name instead of <code>$USER</code>. | |||
Next, change group ownership from $USER to the project group, for example: | |||
chown -R $USER:def-professor ~/projects/def-professor/$USER/ | |||
Then, set the <code>setgid</code> bit on all directories to ensure that newly files created will inherit the directory's group membership, for example: | |||
lfs find ~/projects/def-professor/$USER -type d -print0 | xargs -0 chmod g+s | |||
Finally, verify that project space directories have correct permissions set | |||
chmod 2770 ~/projects/def-professor/ | |||
chmod 2700 ~/projects/def-professor/$USER | |||
=== Another explanation === | |||
<!--T:23--> | <!--T:23--> | ||
Each file in Linux belongs to a person and a group at the same time. | Each file in Linux belongs to a person and a group at the same time. By default, a file you create belongs to you, user '''username''', and your group, named the same '''username'''. That is it is owned by '''username:username'''. Your group is created at the same time your account was created and you are the only user in that group. | ||
By default, a file you create belongs to you, user '''username''', and your group, named the | |||
same '''username'''. That is it is owned by '''username:username'''. | |||
Your group is created at the same time your account was created and you are the only user | |||
in that group. | |||
<!--T:24--> | <!--T:24--> | ||
Line 95: | Line 97: | ||
<!--T:27--> | <!--T:27--> | ||
The other two lines are set for groups '''username''' and '''def-professor''' in Project space. | The other two lines are set for groups '''username''' and '''def-professor''' in Project space. It is not important what users own the files in that space, but the group the files belong to determines the quota limit. | ||
It is not important what users own the files in that space, but the group the files belong | |||
to determines the quota limit. | |||
<!--T:28--> | <!--T:28--> | ||
You see, that files that are owned by '''username''' group (your default group) have very small | You see, that files that are owned by '''username''' group (your default group) have very small limit in the project space, only 2MB, and you already have 34 GB of data that is owned by your group (your files). This is why you cannot write more data there. Because you are trying to place data there owned by a group that has very little allocation there. | ||
limit in the project space, only 2MB, and you already have 34 GB of data that is owned by | |||
your group (your files). This is why you cannot write more data there. Because you are | |||
trying to place data there owned by a group that has very little allocation there. | |||
<!--T:29--> | <!--T:29--> | ||
The allocation for the group '''def-professor''', your professor's group, on the other hand does | The allocation for the group '''def-professor''', your professor's group, on the other hand does not use almost any space and has 1 TB limit. The files that can be put there should have '''username:def-professor''' ownership. | ||
not use almost any space and has 1 TB limit. The files that can be put there should have | |||
'''username:def-professor''' ownership. | |||
<!--T:30--> | <!--T:30--> | ||
Now, depending on how you copy you files, what software you use, that software either will | Now, depending on how you copy you files, what software you use, that software either will respect the ownership of the directory and apply the correct group, or it may insist on retaining the ownership of the source data. In the latter case you will have a problem like you have now. | ||
respect the ownership of the directory and apply the correct group, or it may insist on | |||
retaining the ownership of the source data. In the latter case you will have a problem like | |||
you have now. | |||
<!--T:31--> | <!--T:31--> | ||
Most probably your original data belongs to '''username:username''', properly, upon moving it, | Most probably your original data belongs to '''username:username''', properly, upon moving it, it should belong to '''username:def-professor''', but you software probably insists on keeping the original ownership and this causes the problem. | ||
it should belong to '''username:def-professor''', but you software probably insists on keeping | |||
the original ownership and this causes the problem | |||
== ''sbatch: error: Batch job submission failed: Socket timed out on send/recv operation'' == <!--T:10--> | == ''sbatch: error: Batch job submission failed: Socket timed out on send/recv operation'' == <!--T:10--> |