Visualization: Difference between revisions

From Alliance Doc
Jump to navigation Jump to search
No edit summary
m (heading levels)
 
(120 intermediate revisions by 13 users not shown)
Line 1: Line 1:
<languages />
<languages />
 
[[Category:Software]]
<translate>
<translate>
= External documentation for popular visualization packages = <!--T:1-->
== Popular visualization packages == <!--T:1-->


=== ParaView === <!--T:2-->
=== ParaView === <!--T:2-->
[http://www.paraview.org ParaView] is a general-purpose 3D scientific visualization tool. It is open source and compiles on all popular platforms (Linux, Windows, Mac), understands a large number of input file formats, provides multiple rendering modes, supports Python scripting, and can scale up to tens of thousands of processors for rendering of very large datasets.
[http://www.paraview.org ParaView] is a general-purpose 3D scientific visualization tool. It is open-source and compiles on all popular platforms (Linux, Windows, Mac), understands a large number of input file formats, provides multiple rendering modes, supports Python scripting, and can scale up to tens of thousands of processors for rendering of very large datasets.
 
<!--T:148-->
* [[ParaView|Using ParaView on Alliance systems]]
* [http://www.paraview.org/documentation ParaView official documentation]
* [http://www.paraview.org/documentation ParaView official documentation]
* [http://www.paraview.org/gallery ParaView gallery]
* [http://www.paraview.org/gallery ParaView gallery]
Line 13: Line 16:
=== VisIt === <!--T:3-->
=== VisIt === <!--T:3-->
Similar to ParaView, [https://wci.llnl.gov/simulation/computer-codes/visit/ VisIt] is an open-source, general-purpose 3D scientific data analysis and visualization tool that scales from interactive analysis on laptops to very large HPC projects on tens of thousands of processors.
Similar to ParaView, [https://wci.llnl.gov/simulation/computer-codes/visit/ VisIt] is an open-source, general-purpose 3D scientific data analysis and visualization tool that scales from interactive analysis on laptops to very large HPC projects on tens of thousands of processors.
* [https://wci.llnl.gov/simulation/computer-codes/visit/manuals VisIt manuals]
 
* [https://wci.llnl.gov/simulation/computer-codes/visit/gallery VisIt gallery]
<!--T:149-->
* [[VisIt|Using VisIt on Alliance systems]]
* [https://visit-dav.github.io/visit-website VisIt website]
* [https://visit-dav.github.io/visit-website/examples VisIt gallery]
* [http://www.visitusers.org VisIt user community wiki]
* [http://www.visitusers.org VisIt user community wiki]
* [http://www.visitusers.org/index.php?title=VisIt_Tutorial VisIt tutorials] along with [http://www.visitusers.org/index.php?title=Tutorial_Data sample datasets]
* [http://www.visitusers.org/index.php?title=VisIt_Tutorial VisIt tutorials] along with [http://www.visitusers.org/index.php?title=Tutorial_Data sample datasets]
Line 20: Line 26:
=== VMD === <!--T:4-->
=== VMD === <!--T:4-->
[http://www.ks.uiuc.edu/Research/vmd VMD] is an open-source molecular visualization program for displaying, animating, and analyzing large biomolecular systems in 3D. It supports scripting in Tcl and Python and runs on a variety of platforms (MacOS X, Linux, Windows). It reads many molecular data formats using an extensible plugin system and supports a number of different molecular representations.
[http://www.ks.uiuc.edu/Research/vmd VMD] is an open-source molecular visualization program for displaying, animating, and analyzing large biomolecular systems in 3D. It supports scripting in Tcl and Python and runs on a variety of platforms (MacOS X, Linux, Windows). It reads many molecular data formats using an extensible plugin system and supports a number of different molecular representations.
<!--T:150-->
* [[VMD|Using VMD on Alliance systems]]
* [http://www.ks.uiuc.edu/Research/vmd/current/ug VMD User's Guide]
* [http://www.ks.uiuc.edu/Research/vmd/current/ug VMD User's Guide]


=== VTK === <!--T:5-->
=== VTK === <!--T:5-->
The Visualization Toolkit (VTK) is an open-source package for 3D computer graphics, image processing, and visualization. The toolkit includes a C++ class library as well as several interfaces for interpreted languages such as Tcl/Tk, Java, and Python. VTK was the basis for many excellent visualization packages including ParaView and VisIt.
The Visualization Toolkit (VTK) is an open-source package for 3D computer graphics, image processing, and visualization. The toolkit includes a C++ class library as well as several interfaces for interpreted languages such as Tcl/Tk, Java, and Python. VTK was the basis for many excellent visualization packages including ParaView and VisIt.
<!--T:151-->
* [[VTK|Using VTK on Alliance systems]]
* [https://itk.org/Wiki/VTK/Tutorials VTK tutorials]
* [https://itk.org/Wiki/VTK/Tutorials VTK tutorials]


= Visualization on new Compute Canada systems = <!--T:6-->
=== YT === <!--T:152-->
 
YT is a Python library for analyzing and visualizing volumetric, multi-resolution data. Initially developed for astrophysical simulation data, it can handle any uniform and multiple-resolution data on Cartesian, curvilinear, unstructured meshes and on particles.
== Client-server visualization on Cedar and Graham == <!--T:43-->
 
<!--T:44-->
Currently, we are working to bring GPU rendering to [[Cedar]] (SFU) and [[Graham]]. In the meantime, you can do client-server ParaView rendering on cluster CPUs. To get started, install ParaView 5.3.x (where "x" could be anything) on your laptop as you need the same version as on the cluster. Log in to Cedar or Graham and start a serial non-GPU interactive job:


  <!--T:45-->
<!--T:153-->
salloc --time=1:00:0 --ntasks=1 --account=xwp-462-aa
* [[yt|Using YT on Alliance systems]]


<!--T:46-->
== Visualization on Alliance systems == <!--T:6-->
The job should automatically start on one of CPU interactive nodes. Once you get an interactive prompt (that is part of your job), load the offscreen ParaView module and start the server:


  <!--T:47-->
<!--T:154-->
module load paraview-offscreen/5.3.0
There are many options for remote visualization on our systems. In general, whenever possible, for interactive rendering we recommend '''client-server visualization''' on interactive or high-priority nodes, and for non-interactive visualization we recommend '''off-screen batch jobs''' on regular compute nodes.
  pvserver --mesa-swr-avx2


<!--T:48-->
<!--T:155-->
The flag "--mesa-swr-avx2" is important for much faster software rendering with OpenSWR library. Wait for the server to be ready to accept client connection:
Other, ''less efficient'' options are X11-forwarding and VNC. For some packages these are the only available remote GUI options.


<!--T:49-->
=== Client-server interactive visualization === <!--T:156-->
Connection URL: cs://cdr774.int.cedar.computecanada.ca:11111
Accepting connection(s): cdr774.int.cedar.computecanada.ca:11111


<!--T:50-->
<!--T:157-->
Take a note of the node (in this case cdr774) and the port (usually 11111) and then in another terminal on your laptop (on Mac/Linux; in Windows use a terminal emulator) link the port 11111 on your laptop and the same port on the compute node (make sure to use the correct compute node):
In the client-server mode, supported by both ParaView and VisIt, all data will be processed remotely on the cluster, using either CPU or GPU rendering, while you interact with your visualization through a familiar GUI client on your local computer. You can find the details of setting up client-server visualization in [[ParaView]] and [[VisIt]] pages.


  <!--T:51-->
=== Remote windows with X11-forwarding === <!--T:158-->
ssh cedar.computecanada.ca -L 11111:cdr774:11111


<!--T:52-->
<!--T:159-->
Start ParaView on your laptop, go to File -> Connect (or click on the green Connect button in the toolbar) and then click Add Server. You'll need to point ParaView to your local port 11111, so you can do something like name = cedar, server type = Client/Server, host = localhost, port = 11111, then click Configure, then select Manual and click Save. Once the remote is added to the configuration, simply select the server from the list and click Connect. The first terminal window that was saying "Accepting connection ..." will now say "Client connected" and possibly "SWR detected AVX2" (or it may confirm AVX2 support when you start rendering).
In general, X11-forwarding should be avoided for any heavy graphics, as it requires many round trips and is much slower than VNC (below). However, in some cases you can connect via ssh with X11. Below we show how you would do this on our clusters. We assume you have an X-server installed on your local computer.


<!--T:53-->
<!--T:160-->
Then open a file in ParaView (it'll point you to the remote filesystem) and visualize it as usual. An important setting in ParaView's preferences is Render View -> Remote/Parallel Rendering Options -> Remote Render Threshold. If you set it to default (20MB) or similar, small rendering will be done on your laptop's GPU, the rotation with a mouse will be fast, but anything modestly intensive (under 20MB) will be shipped to your laptop and -- depending on your connection -- visualization might be slow. If you set it to 0MB, all rendering will be remote including rotation, so you'll be really using the cluster's CPU for everything, good for large data processing but not so good for interactivity. You'll need to play with this setting to see what works best for you.
<tabs>
<tab name="Cedar, Graham and Béluga">


<!--T:54-->
<!--T:161-->
If you want to do parallel rendering on multiple CPUs, start a parallel job (don't forget to specify the correct maximum walltime limit):
Connect to the cluster with the <code>-X/-Y</code> flag for X11-forwarding. You can start your graphical application on the login node (small visualizations)


  <!--T:55-->
  <!--T:162-->
salloc --time=0:30:0 --ntasks=8 --account=xwp-462-aa
module load vmd
  vmd


<!--T:56-->
<!--T:163-->
and then start ParaView server with "srun":
or you can request interactive resources on a compute node (large visualizations)


   <!--T:57-->
   <!--T:164-->
module load paraview-offscreen/5.3.0
salloc --time=1:00:0 --ntasks=1 --mem=3500 --account=def-someprof --x11
  srun pvserver --mesa


<!--T:58-->
<!--T:165-->
The flag "--mesa-swr-avx2" does not seem to have any effect when in parallel so we replaced it with the more generic "--mesa" to (hopefully) enable automatic detection of the best software rendering option.
: and, once the job is running, start your graphical application inside the job


<!--T:59-->
  <!--T:166-->
To check that you are doing parallel rendering, you can pass your visualization through the Process Id Scalars filter and then colour it by "process id".
module load vmd
  vmd


== Client-server visualization in a cloud VM == <!--T:20-->
<!--T:167-->
</tab>
<tab name="Niagara">


=== Prerequisites === <!--T:21-->
<!--T:168-->
Since runtime is limited on the login nodes, you might want to request a testing job in order to have more time for exploring and visualizing your data. On the plus side, you will have access to 40 cores on each of the nodes requested. For performing an interactive visualization session in this way please follow these steps:


<!--T:22-->
<!--T:169-->
You can launch a new cloud virtual machine (VM) as described in the [[Cloud Quick Start|Cloud Quick Start Guide]]. Once you log into the VM, you'll need to install some additional packages to be able to compile ParaView or VisIt. For example, on a CentOS VM you can type:
<ol>
<li> ssh into niagara.scinet.utoronto.ca with the <code>-X/-Y</code> flag for X11-forwarding
<li> Request an interactive job, ie.</li>
  debugjob
This will connect you to a node, let's say for the argument "niaXYZW".
<li> Run your visualization program, eg. VMD </li>


</translate>
  <!--T:170-->
  sudo yum install xauth wget gcc gcc-c++ ncurses-devel python-devel libxcb-devel
module load vmd
  sudo yum install patch imake libxml2-python mesa-libGL mesa-libGL-devel
  vmd
  sudo yum install mesa-libGLU mesa-libGLU-devel bzip2 bzip2-libs libXt-devel zlib-devel flex byacc
  sudo ln -s /usr/include/GL/glx.h /usr/local/include/GL/glx.h
<translate>


<!--T:23-->
<!--T:171-->
If you have your own private-public SSH key pair (as opposed to the cloud key), you may want to copy the public key to the VM to simplify logins, by issuing the following command on your laptop:
<li> Exit the debug session.
</ol>


</translate>
<!--T:172-->
  cat ~/.ssh/id_rsa.pub | ssh -i ~/.ssh/cloudwestkey.pem centos@vm.ip.address 'cat >>.ssh/authorized_keys'
</tab>
<translate>
</tabs>


=== ParaView client-server === <!--T:40-->  
=== Remote off-screen windows via Xvfb === <!--T:176-->  


==== Compiling ParaView with OSMesa ==== <!--T:24-->
<!--T:177-->
Some applications insist on displaying graphical output, but you don't actually need to see them since the results are saved in a file.
To work with offscreen rendering, the job can run as a regular batch job, using either the CPU or the GPU for 3D rendering. To enable this you can run
the application you are calling with the X virtual frame buffer (Xvfb) in a job script as follows:


<!--T:25-->
  <!--T:178-->
Since the VM does not have access to a GPU (most Cloud West VMs don't), we need to compile ParaView with OSMesa support so that it can do offscreen (software) rendering. The default configuration of OSMesa will enable OpenSWR (Intel's software rasterization library to run OpenGL). What you'll end up with is a ParaView server that uses OSMesa for offscreen CPU-based rendering without X but with both llvmpipe (older and slower) and SWR (newer and faster) drivers built. We recommend using SWR.
xvfb-run <name-of-application>


<!--T:26-->
<!--T:179-->
Back on the VM, compile cmake::
if using the CPU for rendering or
</translate>
wget https://cmake.org/files/v3.7/cmake-3.7.0.tar.gz
unpack and cd there
./bootstrap
make
sudo make install
<translate>
<!--T:27-->
Next, compile llvm:
</translate>
cd
wget http://releases.llvm.org/3.9.1/llvm-3.9.1.src.tar.xz
unpack and cd there
mkdir -p build && cd build
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_BUILD_LLVM_DYLIB=ON \
  -DLLVM_ENABLE_RTTI=ON \
  -DLLVM_INSTALL_UTILS=ON \
  -DLLVM_TARGETS_TO_BUILD:STRING=X86 \
  ..
make
sudo make install
<translate>
<!--T:28-->
Next, compile Mesa with OSMesa:
</translate>
cd
wget ftp://ftp.freedesktop.org/pub/mesa/mesa-17.0.0.tar.gz
unpack and cd there
./configure \
  --enable-opengl --disable-gles1 --disable-gles2 \
  --disable-va --disable-xvmc --disable-vdpau \
  --enable-shared-glapi \
  --disable-texture-float \
  --enable-gallium-llvm --enable-llvm-shared-libs \
  --with-gallium-drivers=swrast,swr \
  --disable-dri \
  --disable-egl --disable-gbm \
  --disable-glx \
  --disable-osmesa --enable-gallium-osmesa
make
sudo make install
<translate>
<!--T:29-->
Next, compile ParaView server:
</translate>
cd
wget http://www.paraview.org/files/v5.2/ParaView-v5.2.0.tar.gz
unpack and cd there
mkdir -p build && cd build
cmake \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=/home/centos/paraview \
      -DPARAVIEW_USE_MPI=OFF \
      -DPARAVIEW_ENABLE_PYTHON=ON \
      -DPARAVIEW_BUILD_QT_GUI=OFF \
      -DVTK_OPENGL_HAS_OSMESA=ON \
      -DVTK_USE_OFFSCREEN=ON \
      -DVTK_USE_X=OFF \
      ..
make
make install
<translate>


==== Running ParaView in client-server mode ==== <!--T:30-->
  <!--T:180-->
xvfb-run vglrun -d egl <name-of-application>


<!--T:31-->
<!--T:181-->
Now you are ready to start ParaView server on the VM with SWR rendering:
if using the GPU for rendering, in which case you need to reserve one GPU with Slurm, see [[Using_GPUs_with_Slurm|Using GPUs with Slurm]].
</translate>
Note that, depending on the workload the GPU may not necessarily be faster than the CPU, so it's important to benchmark before
./paraview/bin/pvserver --mesa-swr-avx2
committing to using the more expensive GPU.
<translate>
<!--T:32-->
Back on your laptop, organize an SSH tunnel from the local port 11111 to the VM's port 11111:
</translate>
ssh centos@vm.ip.address -L 11111:localhost:11111
<translate>
<!--T:33-->
Finally, start the ParaView client on your laptop and connect to localhost:11111. If successful, you should be able to open files on the remote VM. During rendering in the console you should see the message "SWR detected AVX2".


=== VisIt client-server === <!--T:41-->
=== Start a remote desktop via VNC === <!--T:92-->


==== Compiling VisIt with OSMesa ==== <!--T:42-->
<!--T:93-->
Frequently, it may be useful to start up graphical user interfaces for various software packages like Matlab. Doing so over X11-forwarding can result in a very slow connection to the server. Instead, we recommend using VNC to start and connect to a remote desktop. For more information, please see [[VNC|the article on VNC]].


<!--T:35-->
= Visualization training = <!--T:7-->
VisIt with offscreen rendering support can be built with a single script:
</translate>
wget http://portal.nersc.gov/project/visit/releases/2.12.1/build_visit2_12_1
chmod u+x build_visit2_12_1
./build_visit2_12_1 --prefix /home/centos/visit --mesa --system-python \
    --hdf4 --hdf5 --netcdf --silo --szip --xdmf --zlib
<translate>
<!--T:36-->
This may take a couple of hours. Once finished, you can test the installation with:
</translate>
~/visit/bin/visit -cli -nowin
<translate>
<!--T:37-->
This should start a VisIt Python shell.


==== Running VisIt in client-server mode ==== <!--T:38-->
<!--T:138-->
Please [mailto:support@tech.alliancecan.ca let us know] if you would like to see a visualization workshop at your institution.


<!--T:39-->
=== Full- or half-day workshops === <!--T:9-->
Start VisIt on your laptop and in Options -> Host profiles... edit the connection nickname (let's call it Cloud West), the VM host name, path to VisIt installation (/home/centos/visit) and your username on the VM, and enable tunneling through ssh. Don't forget to save settings with Options -> Save Settings. Then opening a file (File -> Open file... -> Host = Cloud West) you should see the VM's filesystem. Load a file and try to visualize it. Data processing and rendering should be done on the VM, while the result and the GUI controls will be displayed on your laptop.
* [https://docs.alliancecan.ca/mediawiki/images/5/5d/Visit201606.pdf VisIt workshop slides] from HPCS'2016 in Edmonton by <i>Marcelo Ponce</i> and <i>Alex Razoumov</i>
* [https://docs.alliancecan.ca/mediawiki/images/6/6c/Paraview201707.pdf ParaView workshop slides] from July 2017 by <i>Alex Razoumov</i>
* [https://support.scinet.utoronto.ca/~mponce/courses/ss2016/ss2016_visualization-I.pdf Gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib] slides by <i>Marcelo Ponce</i> (SciNet/UofT) from Ontario HPC Summer School 2016
* [https://support.scinet.utoronto.ca/~mponce/courses/ss2016/ss2016_visualization-II.pdf  Brief overview of ParaView & VisIt] slides by <i>Marcelo Ponce</i> (SciNet/UofT) from Ontario HPC Summer School 2016


= Upcoming visualization events = <!--T:7-->
=== Webinars and other short presentations === <!--T:10-->
* visualization webinar in May 2017, topic TBA
* full-day [https://www.westgrid.ca/events/data_visualization_workshop_university_calgary VisIt visualization workshop at UofCalgary] (May-04)


= Compute Canada visualization presentation materials = <!--T:8-->
<!--T:173-->
[https://training.westdri.ca/tools/visualization Western Canada visualization training materials page] has video recordings and slides from many visualization webinars including:


=== Full- or half-day workshops === <!--T:9-->
<!--T:174-->
* [https://docs.computecanada.ca/mediawiki/images/5/5d/Visit201606.pdf VisIt workshop slides] from HPCS'2016 in Edmonton by <i>Marcelo Ponce</i> and <i>Alex Razoumov</i>
* YT series: “Using YT for analysis and visualization of volumetric data” (Part 1) and "Working with data objects in YT” (Part 2)
* [https://docs.computecanada.ca/mediawiki/images/d/d0/Paraview201602.pdf ParaView workshop slides] from February 2016 by <i>Alex Razoumov</i>
* “Scientific visualization with Plotly”
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-I.pdf Gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib] slides by <i>Marcelo Ponce</i> (SciNet/UofT) from Ontario HPC Summer School 2016
* “Novel Visualization Techniques from the 2017 Visualize This Challenge”
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-II.pdf  Brief overview of ParaView & VisIt] slides by <i>Marcelo Ponce</i> (SciNet/UofT) from Ontario HPC Summer School 2016
* “Data Visualization on Compute Canada’s Supercomputers” contains recipes and demos of running client-server ParaView and batch ParaView scripts on both CPU and GPU partitions of Cedar and Graham
* “Using ParaViewWeb for 3D Visualization and Data Analysis in a Web Browser”
* “Scripting and other advanced topics in VisIt visualization”
* “CPU-based rendering with OSPRay”
* “3D graphs with NetworkX, VTK, and ParaView”
* “Graph visualization with Gephi”


=== Webinars and other short presentations === <!--T:10-->
<!--T:175-->
Other visualization presentations:


<!--T:16-->
<!--T:16-->
*[https://docs.computecanada.ca/mediawiki/images/0/0c/Slides.pdf Using ParaViewWeb for 3D visualization and data analysis in a browser] from March 2017 by <i>Alex Razoumov</i>
* [https://oldwiki.scinet.utoronto.ca/wiki/images/5/51/Remoteviz.pdf Remote Graphics on SciNet's GPC system (Client-Server and VNC)] slides by <i>Ramses van Zon</i> (SciNet/UofT) from October 2015 SciNet User Group Meeting
* [https://docs.computecanada.ca/mediawiki/images/0/0c/Slides.pdf Visualization support in WestGrid / Compute Canada] from January 2017 by <i>Alex Razoumov</i>
* [https://docs.computecanada.ca/mediawiki/images/e/e5/VisitScripting.pdf VisIt scripting] from November 2016 by <i>Alex Razoumov</i>
* [https://docs.computecanada.ca/mediawiki/images/5/5f/Batch201503.pdf Batch visualization webinar] slides from March 2015 by <i>Alex Razoumov</i>
* [https://docs.computecanada.ca/mediawiki/images/5/59/OspraySlides.pdf CPU-based rendering with OSPRay] from September 2016 by <i>Alex Razoumov</i>
* [https://docs.computecanada.ca/mediawiki/images/f/fc/Gephi201603.pdf Gephi webinar notes] from March 2016 by <i>Alex Razoumov</i>
* [https://docs.computecanada.ca/mediawiki/images/6/60/Graphs201605.pdf 3D graphs with NetworkX, VTK, and ParaView] slides from May 2016 by <i>Alex Razoumov</i>
* [https://wiki.scinet.utoronto.ca/wiki/images/5/51/Remoteviz.pdf Remote Graphics on SciNet's GPC system (Client-Server and VNC)] slides by <i>Ramses van Zon</i> (SciNet/UofT) from October 2015 SciNet User Group Meeting
* [https://support.scinet.utoronto.ca/education/go.php/242/file_storage/index.php/download/1/files%5B%5D/6399/ VisIt Basics], slides by <i>Marcelo Ponce</i> (SciNet/UofT) from February 2016 SciNet User Group Meeting
* [https://support.scinet.utoronto.ca/education/go.php/242/file_storage/index.php/download/1/files%5B%5D/6399/ VisIt Basics], slides by <i>Marcelo Ponce</i> (SciNet/UofT) from February 2016 SciNet User Group Meeting
* [https://wiki.scinet.utoronto.ca/wiki/images/e/ea/8_ComplexNetworks.pdf Intro to Complex Networks Visualization, with Python], slides by <i>Marcelo Ponce</i> (SciNet/UofT)
* [https://oldwiki.scinet.utoronto.ca/wiki/images/e/ea/8_ComplexNetworks.pdf Intro to Complex Networks Visualization, with Python], slides by <i>Marcelo Ponce</i> (SciNet/UofT)
* [https://wiki.scinet.utoronto.ca/wiki/images/9/9c/Tkinter.pdf Introduction to GUI Programming with Tkinter], from Sept.2014 by <i>Erik Spence</i> (SciNet/UofT)
* [https://oldwiki.scinet.utoronto.ca/wiki/images/9/9c/Tkinter.pdf Introduction to GUI Programming with Tkinter], from Sept.2014 by <i>Erik Spence</i> (SciNet/UofT)


= Tips and tricks = <!--T:11-->
== Tips and tricks == <!--T:11-->


<!--T:19-->
<!--T:19-->
This section will describe visualization workflows not included into the workshop/webinar slides above. It is meant to be user-editable, so please feel free to add your cool visualization scripts and workflows here so that everyone can benefit from them.
This section will describe visualization workflows not included into the workshop/webinar slides above. It is meant to be user-editable, so please feel free to add your cool visualization scripts and workflows here so that everyone can benefit from them.


= Regional visualization pages = <!--T:12-->
== Regional visualization pages == <!--T:12-->
== [http://www.westgrid.ca WestGrid] ==
* [https://www.westgrid.ca/support/visualization/vis_quickstart visualization quickstart guide]
* [https://www.westgrid.ca/support/visualization/remote_visualization remote visualization]
* [https://www.westgrid.ca/support/visualization/batch_rendering batch rendering]
 
== [http://www.scinet.utoronto.ca SciNet, HPC at the University of Toronto] == <!--T:13-->
* [https://wiki.scinet.utoronto.ca/wiki/index.php/Software_and_Libraries#anchor_viz visualization software]
* [https://wiki.scinet.utoronto.ca/wiki/index.php/VNC VNC]
* [https://wiki.scinet.utoronto.ca/wiki/index.php/Visualization_Nodes visualization nodes]
* [https://wiki.scinet.utoronto.ca/wiki/index.php/Knowledge_Base:_Tutorials_and_Manuals#Visualization further resources and viz-tech talks]
* [https://wiki.scinet.utoronto.ca/wiki/index.php/Using_Paraview using ParaView]
 
== [https://www.sharcnet.ca SHARCNET] == <!--T:17-->
* [https://www.sharcnet.ca/help/index.php/Visualization_in_SHARCNET Overview]
* [https://www.sharcnet.ca/help/index.php/Remote_Graphical_Connections Running pre-/post-processing graphical applications]
* [https://www.sharcnet.ca/my/software Supported software (see visualization section at bottom)]
 
= Visualization gallery = <!--T:14-->


<!--T:18-->
=== [http://www.scinet.utoronto.ca SciNet HPC at the University of Toronto] === <!--T:13-->
You can find a gallery of visualizations based on models run on Compute Canada systems in the [https://www.computecanada.ca/research-portal/national-services/visualization visualization gallery]. There you can click on individual thumbnails to get more details on each visualization.
* [https://docs.scinet.utoronto.ca/index.php/Visualization Visualization in Niagara]
* [https://oldwiki.scinet.utoronto.ca/wiki/index.php/Software_and_Libraries#anchor_viz visualization software]
* [https://oldwiki.scinet.utoronto.ca/wiki/index.php/VNC VNC]
* [https://oldwiki.scinet.utoronto.ca/wiki/index.php/Visualization_Nodes visualization nodes]
* [https://oldwiki.scinet.utoronto.ca/wiki/index.php/Knowledge_Base:_Tutorials_and_Manuals#Visualization further resources and viz-tech talks]
* [https://oldwiki.scinet.utoronto.ca/wiki/index.php/Using_Paraview using ParaView]


= How to get visualization help = <!--T:15-->
== How to get visualization help == <!--T:15-->
You can contact us via [mailto:vis-support@computecanada.ca email].
Please contact [[Technical support]].
</translate>
</translate>

Latest revision as of 14:19, 5 June 2024

Other languages:

Popular visualization packages

ParaView

ParaView is a general-purpose 3D scientific visualization tool. It is open-source and compiles on all popular platforms (Linux, Windows, Mac), understands a large number of input file formats, provides multiple rendering modes, supports Python scripting, and can scale up to tens of thousands of processors for rendering of very large datasets.

VisIt

Similar to ParaView, VisIt is an open-source, general-purpose 3D scientific data analysis and visualization tool that scales from interactive analysis on laptops to very large HPC projects on tens of thousands of processors.

VMD

VMD is an open-source molecular visualization program for displaying, animating, and analyzing large biomolecular systems in 3D. It supports scripting in Tcl and Python and runs on a variety of platforms (MacOS X, Linux, Windows). It reads many molecular data formats using an extensible plugin system and supports a number of different molecular representations.

VTK

The Visualization Toolkit (VTK) is an open-source package for 3D computer graphics, image processing, and visualization. The toolkit includes a C++ class library as well as several interfaces for interpreted languages such as Tcl/Tk, Java, and Python. VTK was the basis for many excellent visualization packages including ParaView and VisIt.

YT

YT is a Python library for analyzing and visualizing volumetric, multi-resolution data. Initially developed for astrophysical simulation data, it can handle any uniform and multiple-resolution data on Cartesian, curvilinear, unstructured meshes and on particles.

Visualization on Alliance systems

There are many options for remote visualization on our systems. In general, whenever possible, for interactive rendering we recommend client-server visualization on interactive or high-priority nodes, and for non-interactive visualization we recommend off-screen batch jobs on regular compute nodes.

Other, less efficient options are X11-forwarding and VNC. For some packages these are the only available remote GUI options.

Client-server interactive visualization

In the client-server mode, supported by both ParaView and VisIt, all data will be processed remotely on the cluster, using either CPU or GPU rendering, while you interact with your visualization through a familiar GUI client on your local computer. You can find the details of setting up client-server visualization in ParaView and VisIt pages.

Remote windows with X11-forwarding

In general, X11-forwarding should be avoided for any heavy graphics, as it requires many round trips and is much slower than VNC (below). However, in some cases you can connect via ssh with X11. Below we show how you would do this on our clusters. We assume you have an X-server installed on your local computer.

Connect to the cluster with the -X/-Y flag for X11-forwarding. You can start your graphical application on the login node (small visualizations) 

  module load vmd
  vmd

or you can request interactive resources on a compute node (large visualizations) 

 salloc --time=1:00:0 --ntasks=1 --mem=3500 --account=def-someprof --x11
and, once the job is running, start your graphical application inside the job
 module load vmd
 vmd

Since runtime is limited on the login nodes, you might want to request a testing job in order to have more time for exploring and visualizing your data. On the plus side, you will have access to 40 cores on each of the nodes requested. For performing an interactive visualization session in this way please follow these steps:

  1. ssh into niagara.scinet.utoronto.ca with the -X/-Y flag for X11-forwarding
  2. Request an interactive job, ie.
    3. debugjob This will connect you to a node, let's say for the argument "niaXYZW".
  3. Run your visualization program, eg. VMD
    4. module load vmd vmd
  4. Exit the debug session.

Remote off-screen windows via Xvfb

Some applications insist on displaying graphical output, but you don't actually need to see them since the results are saved in a file. To work with offscreen rendering, the job can run as a regular batch job, using either the CPU or the GPU for 3D rendering. To enable this you can run the application you are calling with the X virtual frame buffer (Xvfb) in a job script as follows: 

 xvfb-run <name-of-application>

if using the CPU for rendering or 

 xvfb-run vglrun -d egl <name-of-application>

if using the GPU for rendering, in which case you need to reserve one GPU with Slurm, see Using GPUs with Slurm. Note that, depending on the workload the GPU may not necessarily be faster than the CPU, so it's important to benchmark before committing to using the more expensive GPU.

Start a remote desktop via VNC

Frequently, it may be useful to start up graphical user interfaces for various software packages like Matlab. Doing so over X11-forwarding can result in a very slow connection to the server. Instead, we recommend using VNC to start and connect to a remote desktop. For more information, please see the article on VNC.

Visualization training

Please let us know if you would like to see a visualization workshop at your institution.

Full- or half-day workshops

Webinars and other short presentations

Western Canada visualization training materials page has video recordings and slides from many visualization webinars including:

  • YT series: “Using YT for analysis and visualization of volumetric data” (Part 1) and "Working with data objects in YT” (Part 2)
  • “Scientific visualization with Plotly”
  • “Novel Visualization Techniques from the 2017 Visualize This Challenge”
  • “Data Visualization on Compute Canada’s Supercomputers” contains recipes and demos of running client-server ParaView and batch ParaView scripts on both CPU and GPU partitions of Cedar and Graham
  • “Using ParaViewWeb for 3D Visualization and Data Analysis in a Web Browser”
  • “Scripting and other advanced topics in VisIt visualization”
  • “CPU-based rendering with OSPRay”
  • “3D graphs with NetworkX, VTK, and ParaView”
  • “Graph visualization with Gephi”

Other visualization presentations:

Tips and tricks

This section will describe visualization workflows not included into the workshop/webinar slides above. It is meant to be user-editable, so please feel free to add your cool visualization scripts and workflows here so that everyone can benefit from them.

Regional visualization pages

SciNet HPC at the University of Toronto

How to get visualization help

Please contact Technical support.

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Visualization&oldid=154921"