Diskusage Explorer
Content of Folders
Warning: At the moment this feature is only available on Béluga and Narval.
You can get a breakdown by folder of how the diskspace is being consumed in your home, scratch and project spaces. That information is currently updated once a day and is stored in an SQLite format for fast access.
Here is a walkthrough of how to explore your disk consumption, using the example of the project space def-professor as the particular directory to investigate.
Ncurse User Interface
Choose a project space you have access to and which you wish to analyze, which for the purpose of this discussion we will assume to be def-professor.
[name@server ~]$ diskusage_explorer /project/def-professor
This command loads a browser that shows the resources consumed by all files under any directory tree
Type c to toggle between consumed disk space and the number of files, q or <esc> to quit and h for help.
If you are only interested in a sub-directory of this project space and do not want to navigate the whole tree in the ncurse user interface,
[name@server ~]$ diskusage_explorer /project/def-professor/subdirectory/
A complete manual page is available with the man duc command.
Graphical User Interface
Note that when the login node is especially busy or if you have an especially large amount of files in you project space, the graphical interface mode can be slow and choppy. For a better experience, you can read the section below to run diskusage_explorer on your own machine.
Note that Compute Canada recommends the use of the standard text-based ncurse mode on our cluster login nodes but diskusage_explorer does also include a nice graphical user interface (GUI).
First make sure that you are connected to the cluster in such a way that SSH is capable of correctly displaying GUI applications. You can then use a graphical interface by means of the command,
[name@server ~]$ duc gui -d /project/.duc_databases/def-professor.sqlite /project/def-professor
You can navigate the folders with the mouse and still type c to toggle between the size of the files and the number of files.
Browse faster on your own machine
First install the diskusage_explorer software on your local machine and then, still on your local machine, download the SQLite file from your cluster and run duc.
rsync -v --progress username@beluga.calculcanada.ca:/project/.duc_databases/def-professor.sqlite . duc gui -d ./def-professor.sqlite /project/def-professor
This immediately leads to a smoother and more satisfying browsing experience.
Space and inode usage per user on cedar
On cedar, it is possible for any member of a group to run diskusage_report with the following options --per_user and --all_users to have a break-down usage per user. The first option displays only heavy users. In other terms, members of the group who have more files and/or occupy more space. When both options are used, the command gives the break-down usage for all members of the group. This is a handy command that helps to identify the users within a group who have more files and/or a large amount of data and ask them to better manage their data by reducing their inode usage for example.
In the following example, a user user01 runs the command and got the following output:
[user01@cedar1 ~]$ diskusage_report --per_user --all_users
Description Space # of files
/home (user user01) 109k/50G 12/500k
/scratch (user user01) 4000/20T 1/1000k
/project (group user01) 0/2048k 0/1025
/project (group def-professor) 9434G/10T 4062k/5000k
Breakdown for project def-professor (Last update: 2023-05-02 01:03:10)
User File count Size Location
-------------------------------------------------------------------------
user01 141567 4.00 GiB On disk
user02 11926 3.74 GiB On disk
user03 725383 6121.03 GiB On disk
user04 4010 377.86 GiB On disk
user05 1296466 262.75 GiB On disk
user06 1010996 60.51 GiB On disk
user07 848066 1721.33 GiB On disk
user08 26516 947.23 GiB On disk
Total 4064930 9510.43 GiB On disk
Breakdown for nearline def-professor (Last update: 2023-05-02 01:01:30)
User File count Size Location
-------------------------------------------------------------------------
sbelharb 5 1197.90 GiB On disk and tape
Total 5 1197.90 GiB On disk and tape
This group has 8 users and the output shows clearly that at least 4 of them have a large number of files for a small amount of data:
User File count Size Location
-------------------------------------------------------------------------
user01 141567 4.00 GiB On disk
user02 11926 3.74 GiB On disk
user05 1296466 262.75 GiB On disk
user06 1010996 60.51 GiB On disk