Frequently Asked Questions: Difference between revisions

Jump to navigation Jump to search
no edit summary
No edit summary
No edit summary
Line 45: Line 45:


<!--T:6-->
<!--T:6-->
The example above illustrates a frequent problem: <code>/project</code> for user <code>ymartin</code> contains too much data in files belonging to group <code>ymartin</code>. The data should instead be in files belonging to <code>def-zrichard</code>.
The example above illustrates a frequent problem: <code>/project</code> for user <code>ymartin</code> contains too much data in files belonging to group <code>ymartin</code>. The data should instead be in files belonging to <code>def-zrichard</code>. To see the project groups you may use, run the following command:
stat -c %G $HOME/projects/*/


<!--T:8-->
<!--T:8-->
Line 59: Line 60:
* files were moved from <code>/home</code> to <code>/project</code> with the <code>mv</code>command; to avoid this, see [[#Moving files between scratch and home filesystems | advice above]];
* files were moved from <code>/home</code> to <code>/project</code> with the <code>mv</code>command; to avoid this, see [[#Moving files between scratch and home filesystems | advice above]];
* files were transferred from another cluster using [[Transferring_data#Rsync|rsync]] or [[Transferring_data#SCP|scp]] with an option to preserve the original group ownership. If you have a recurring problem with ownership, check the options you are using with your file transfer program;
* files were transferred from another cluster using [[Transferring_data#Rsync|rsync]] or [[Transferring_data#SCP|scp]] with an option to preserve the original group ownership. If you have a recurring problem with ownership, check the options you are using with your file transfer program;
* you have no <tt>setgid</tt> bit set on your Project folders. You can reset correct permissions like so (note the trailing slash)
* you have no <tt>setgid</tt> bit set on your Project folders.
  chmod 2770 ~/projects/def-professor/
 
=== How to fix the problem ===
If you already have data in your <code>/project</code> directory with the wrong group ownership, you can use the <code>find</code> command in conjunction with <code>readlink</code> to find and display those files:
  lfs find ~/projects/*/ -group $USER
If the output of <code>diskusage_report</code> indicates that a different group is over quota, use that group name instead of <code>$USER</code>.


<!--T:3-->
Next, change group ownership from $USER to the project group, for example:
To see the project groups you may use, run the following command:
chown -R $USER:def-professor ~/projects/def-professor/$USER/
{{Command|stat -c %G $HOME/projects/*/}}


<!--T:4-->
Then, set the <code>setgid</code> bit on all directories to ensure that newly files created will inherit the directory's group membership, for example:
If you are the owner of the files, you can run the <code>chgrp</code> command to change their group ownership to the appropriate project group. To ask us to change the group owner for several users, contact [[Technical Support|technical support]].
lfs find ~/projects/def-professor/$USER -type d -print0 | xargs -0 chmod g+s
You can also use the command <tt>chmod g+s <directory name></tt> to ensure that files created in that directory will inherit the directory's group membership.


=== Another explanation === <!--T:22-->
Finally, verify that project space directories have correct permissions set
chmod 2770 ~/projects/def-professor/
chmod 2700 ~/projects/def-professor/$USER


=== Another explanation ===
<!--T:23-->
<!--T:23-->
Each file in Linux belongs to a person and a group at the same time.
Each file in Linux belongs to a person and a group at the same time. By default, a file you create belongs to you, user '''username''', and your group, named the same '''username'''. That is it is owned by '''username:username'''. Your group is created at the same time your account was created and you are the only user in that group.  
By default, a file you create belongs to you, user '''username''', and your group, named the
same '''username'''. That is it is owned by '''username:username'''.  
Your group is created at the same time your account was created and you are the only user
in that group.  


<!--T:24-->
<!--T:24-->
Line 95: Line 97:


<!--T:27-->
<!--T:27-->
The other two lines are set for groups '''username''' and '''def-professor''' in Project space.  
The other two lines are set for groups '''username''' and '''def-professor''' in Project space. It is not important what users own the files in that space, but the group the files belong to determines the quota limit.  
It is not important what users own the files in that space, but the group the files belong
to determines the quota limit.  


<!--T:28-->
<!--T:28-->
You see, that files that are owned by '''username''' group (your default group) have very small
You see, that files that are owned by '''username''' group (your default group) have very small limit in the project space, only 2MB, and you already have 34 GB of data that is owned by your group (your files). This is why you cannot write more data there. Because you are trying to place data there owned by a group that has very little allocation there.
limit in the project space, only 2MB, and you already have 34 GB of data that is owned by
your group (your files). This is why you cannot write more data there. Because you are
trying to place data there owned by a group that has very little allocation there.


<!--T:29-->
<!--T:29-->
The allocation for the group '''def-professor''', your professor's group, on the other hand does
The allocation for the group '''def-professor''', your professor's group, on the other hand does not use almost any space and has 1 TB limit. The files that can be put there should have '''username:def-professor''' ownership.  
not use almost any space and has 1 TB limit. The files that can be put there should have
'''username:def-professor''' ownership.  


<!--T:30-->
<!--T:30-->
Now, depending on how you copy you files, what software you use, that software either will
Now, depending on how you copy you files, what software you use, that software either will respect the ownership of the directory and apply the correct group, or it may insist on retaining the ownership of the source data. In the latter case you will have a problem like you have now.
respect the ownership of the directory and apply the correct group, or it may insist on
retaining the ownership of the source data. In the latter case you will have a problem like
you have now.


<!--T:31-->
<!--T:31-->
Most probably your original data belongs to '''username:username''', properly, upon moving it,
Most probably your original data belongs to '''username:username''', properly, upon moving it, it should belong to '''username:def-professor''', but you software probably insists on keeping the original ownership and this causes the problem.
it should belong to '''username:def-professor''', but you software probably insists on keeping
the original ownership and this causes the problem.
 
<!--T:32-->
If you already have data in your project directory with wrong ownership, you can correct this with
commands:
<pre>
$ cd project/$USER
$ chown -R username:def-professor data_dir
</pre>
This will correct the ownership of the files inside '''data_dir''' directory in your project space.
 
=== Finding files with the wrong group ownership === <!--T:18-->
You may find it difficult to identify files that are contributing to an over-quota condition in <code>/project</code>. The <code>find</code> command can be used in conjunction with <code>readlink</code> to solve this:
{{Command|lfs find $(readlink $HOME/projects/*) -group $USER}}
This will identify files belonging to the user's unique group, e.g. <code>ymartin</code> in the example shown earlier. If the output of <code>quota</code> indicates that a different group is over quota, use that group name instead of <code>$USER</code>.
 
<!--T:7-->
See [[Project layout]] for further explanations.


== ''sbatch: error: Batch job submission failed: Socket timed out on send/recv operation'' == <!--T:10-->
== ''sbatch: error: Batch job submission failed: Socket timed out on send/recv operation'' == <!--T:10-->
cc_staff
318

edits

Navigation menu