Alliance Doc

Visualization

Revision as of 16:02, 8 January 2018 by Razoumov (talk | contribs)
Other languages:

External documentation for popular visualization packages

ParaView

ParaView is a general-purpose 3D scientific visualization tool. It is open-source and compiles on all popular platforms (Linux, Windows, Mac), understands a large number of input file formats, provides multiple rendering modes, supports Python scripting, and can scale up to tens of thousands of processors for rendering of very large datasets.

VisIt

Similar to ParaView, VisIt is an open-source, general-purpose 3D scientific data analysis and visualization tool that scales from interactive analysis on laptops to very large HPC projects on tens of thousands of processors.

VMD

VMD is an open-source molecular visualization program for displaying, animating, and analyzing large biomolecular systems in 3D. It supports scripting in Tcl and Python and runs on a variety of platforms (MacOS X, Linux, Windows). It reads many molecular data formats using an extensible plugin system and supports a number of different molecular representations.

VTK

The Visualization Toolkit (VTK) is an open-source package for 3D computer graphics, image processing, and visualization. The toolkit includes a C++ class library as well as several interfaces for interpreted languages such as Tcl/Tk, Java, and Python. VTK was the basis for many excellent visualization packages including ParaView and VisIt.

Visualization on the new Compute Canada systems

GPU-based ParaView client-server visualization on Cedar and Graham

Cedar and Graham have a number of interactive GPU nodes that can be used for remote ParaView client-server visualization.

1. First, install on your laptop the same ParaView version as the one available on the cluster you will be using; log into Cedar or Graham and start a serial GPU interactive job. 

 salloc --time=1:00:0 --ntasks=1 --gres=gpu:1 --account=xwp-462-aa
The job should automatically start on one of the GPU interactive nodes.

2. At the prompt that is now running inside your job, load the ParaView GPU+EGL module, change your display variable so that ParaView does not attempt to use the X11 rendering context, and start the ParaView server. 

 module load paraview-offscreen-gpu/5.4.0
 unset DISPLAY
 pvserver
Wait for the server to be ready to accept client connection.
 Waiting for client...
 Connection URL: cs://cdr347.int.cedar.computecanada.ca:11111
 Accepting connection(s): cdr347.int.cedar.computecanada.ca:11111

3. Make a note of the node (in this case cdr347) and the port (usually 11111) and in another terminal on your laptop (on Mac/Linux; in Windows use a terminal emulator), link the port 11111 on your laptop and the same port on the compute node (make sure to use the correct compute node). 

 ssh <username>@cedar.computecanada.ca -L 11111:cdr347:11111

4. Start ParaView on your laptop, go to File -> Connect (or click on the green Connect button on the toolbar) and click Add Server. You will need to point ParaView to your local port 11111, so you can do something like name = cedar, server type = Client/Server, host = localhost, port = 11111, then click Configure, select Manual and click Save.

Once the remote is added to the configuration, simply select the server from the list and click Connect. The first terminal window that read Accepting connection ... will now read Client connected.

5. Open a file in ParaView (it will point you to the remote filesystem) and visualize it as usual.

NOTE: An important setting in ParaView's preferences is Render View -> Remote/Parallel Rendering Options -> Remote Render Threshold. If you set it to default (20MB) or similar, small rendering will be done on your laptop's GPU, the rotation with a mouse will be fast, but anything modestly intensive (under 20MB) will be shipped to your laptop and—depending on your connection—visualization might be slow. If you set it to 0MB, all rendering will be remote including rotation, so you will be really using the cluster's GPU for everything, which is good for large data processing but not so good for interactivity. Experiment with the threshold to find a suitable value.

CPU-based ParaView client-server visualization on Cedar and Graham

You can also do interactive client-server ParaView rendering on cluster CPUs. For some types of rendering, modern CPU-based libraries such as OSPRay and OpenSWR offer performance quite similar to GPU-based rendering. Also, since the ParaView server uses MPI for distributed-memory processing, for very large datasets one can do parallel rendering on a large number of CPU cores, either on a single node, or scattered across multiple nodes.

1. First, install on your laptop the same ParaView version as the one available on the cluster you will be using; log into Cedar or Graham and start a serial CPU interactive job. 

 salloc --time=1:00:0 --ntasks=1 --account=xwp-462-aa
The job should automatically start on one of the CPU interactive nodes.

2. At the prompt that is now running inside your job, load the offscreen ParaView module and start the server. 

 module load paraview-offscreen/5.3.0
 pvserver --mesa-swr-avx2
The --mesa-swr-avx2 flag is important for much faster software rendering with the OpenSWR library.
Wait for the server to be ready to accept client connection.
 Waiting for client...
 Connection URL: cs://cdr774.int.cedar.computecanada.ca:11111
 Accepting connection(s): cdr774.int.cedar.computecanada.ca:11111

3. Make a note of the node (in this case cdr774) and the port (usually 11111) and in another terminal on your laptop (on Mac/Linux; in Windows use a terminal emulator) link the port 11111 on your laptop and the same port on the compute node (make sure to use the correct compute node). 

 ssh <username>@cedar.computecanada.ca -L 11111:cdr774:11111

4. Start ParaView on your laptop, go to File -> Connect (or click on the green Connect button in the toolbar) and click Add Server. You will need to point ParaView to your local port 11111, so you can do something like name = cedar, server type = Client/Server, host = localhost, port = 11111, then click Configure, select Manual and click Save.

Once the remote is added to the configuration, simply select the server from the list and click Connect. The first terminal window that read Accepting connection ... will now read Client connected.

5. Open a file in ParaView (it will point you to the remote filesystem) and visualize it as usual.

NOTE: An important setting in ParaView's preferences is Render View -> Remote/Parallel Rendering Options -> Remote Render Threshold. If you set it to default (20MB) or similar, small rendering will be done on your laptop's CPU, the rotation with a mouse will be fast, but anything modestly intensive (under 20MB) will be shipped to your laptop and—depending on your connection—visualization might be slow. If you set it to 0MB, all rendering will be remote including rotation, so you will be really using the cluster's CPU for everything, which is good for large data processing but not so good for interactivity. Experiment with the threshold to find a suitable value.

If you want to do parallel rendering on multiple CPUs, start a parallel job; don't forget to specify the correct maximum walltime limit 

 salloc --time=0:30:0 --ntasks=8 --account=xwp-462-aa

and then start ParaView server with "srun": 

 module load paraview-offscreen/5.3.0
 srun pvserver --mesa

The flag "--mesa-swr-avx2" does not seem to have any effect when in parallel so we replaced it with the more generic "--mesa" to (hopefully) enable automatic detection of the best software rendering option.

To check that you are doing parallel rendering, you can pass your visualization through the Process Id Scalars filter and then colour it by "process id".

CPU-based VisIt client-server visualization on Cedar and Graham

On Cedar and Graham we have two versions of VisIt installed: visit/2.12.3 and visit/2.13.0. To use remote VisIt in client-server mode, on your laptop you need the matching major version, either 2.12.x or 2.13.x, respectively. Before starting VisIt, download the Host Profile XML file host_cedar.xml. On Linux/Mac copy it to ~/.visit/hosts/, and on Windows to "My Documents\VisIt 2.13.0\hosts\". Start VisIt on your laptop, and in its main menu in Options - Host Profiles you should see a host profile called cedar. If you want to do remote rendering on Graham instead of Cedar, set 

Host nickname = graham
Remote host name = graham.computecanada.ca

For both Cedar and Graham, set your CCDB username 

Username = yourOwwUserName

With the exception of your username, your settings should be similar to the ones shown below:

HostSetting.png

In the same setup window select the Launch Profiles tab. You should see two profiles (login and slurm):

LaunchProfiles.png

Login profile is for running VisIt's engine on a cluster's login node, which we do not recommend for heavy visualizations. Slurm profile is for running VisIt's engine inside an interactive job on a compute node. If you are planning to do the latter, select the slurm profile and then click on Parallel tab and below it on the Advanced tab and change Launcher arguments from --account=def-someuser to your default allocation, as shown below:

LauncherArguments.png

Save settings with Options - Save Settings and then restart VisIt on your laptop for settings to take effect. Start the file-open dialogue and switch the Host from localhost to cedar (or graham). Hopefully, the connection is established, the remote VisIt Component Launcher gets started on the cluster's login node, and you should be able to see the cluster's filesystem, navigate to your file and select it. You will be prompted to select between login (rendering on the login node) and slurm (rendering inside an interactive Slurm job on a compute node) profiles, and additionally for slurm profile you will need to specify the number of nodes and processors and the maximum time limit:

SelectProfile.png

Click Ok and wait for VisIt's engine to start. If you selected rendering on a compute node, it may take some time for your job to get started. Once your dataset appears in the Active source in the main VisIt window, the VisIt's engine is running, and you can proceed with creating and drawing your plot.

Client-server visualization in a cloud VM

Prerequisites

You can launch a new cloud virtual machine (VM) as described in the Cloud Quick Start Guide. Once you log into the VM, you will need to install some additional packages to be able to compile ParaView or VisIt. For example, on a CentOS VM you can type: 

 sudo yum install xauth wget gcc gcc-c++ ncurses-devel python-devel libxcb-devel
 sudo yum install patch imake libxml2-python mesa-libGL mesa-libGL-devel
 sudo yum install mesa-libGLU mesa-libGLU-devel bzip2 bzip2-libs libXt-devel zlib-devel flex byacc
 sudo ln -s /usr/include/GL/glx.h /usr/local/include/GL/glx.h

If you have your own private-public SSH key pair (as opposed to the cloud key), you may want to copy the public key to the VM to simplify logins, by issuing the following command on your laptop: 

 cat ~/.ssh/id_rsa.pub | ssh -i ~/.ssh/cloudwestkey.pem centos@vm.ip.address 'cat >>.ssh/authorized_keys'

ParaView client-server

Compiling ParaView with OSMesa

Since the VM does not have access to a GPU (most Cloud West VMs don't), we need to compile ParaView with OSMesa support so that it can do offscreen (software) rendering. The default configuration of OSMesa will enable OpenSWR (Intel's software rasterization library to run OpenGL). What you will end up with is a ParaView server that uses OSMesa for offscreen CPU-based rendering without X but with both llvmpipe (older and slower) and SWR (newer and faster) drivers built. We recommend using SWR.

Back on the VM, compile cmake:: 

wget https://cmake.org/files/v3.7/cmake-3.7.0.tar.gz
unpack and cd there
./bootstrap
make
sudo make install

Next, compile llvm: 

cd
wget http://releases.llvm.org/3.9.1/llvm-3.9.1.src.tar.xz
unpack and cd there
mkdir -p build && cd build
cmake \
 -DCMAKE_BUILD_TYPE=Release \
 -DLLVM_BUILD_LLVM_DYLIB=ON \
 -DLLVM_ENABLE_RTTI=ON \
 -DLLVM_INSTALL_UTILS=ON \
 -DLLVM_TARGETS_TO_BUILD:STRING=X86 \
 ..
make
sudo make install

Next, compile Mesa with OSMesa: 

cd
wget ftp://ftp.freedesktop.org/pub/mesa/mesa-17.0.0.tar.gz
unpack and cd there
./configure \
 --enable-opengl --disable-gles1 --disable-gles2 \
 --disable-va --disable-xvmc --disable-vdpau \
 --enable-shared-glapi \
 --disable-texture-float \
 --enable-gallium-llvm --enable-llvm-shared-libs \
 --with-gallium-drivers=swrast,swr \
 --disable-dri \
 --disable-egl --disable-gbm \
 --disable-glx \
 --disable-osmesa --enable-gallium-osmesa
make
sudo make install

Next, compile ParaView server: 

cd
wget http://www.paraview.org/files/v5.2/ParaView-v5.2.0.tar.gz
unpack and cd there
mkdir -p build && cd build
cmake \
     -DCMAKE_BUILD_TYPE=Release \
     -DCMAKE_INSTALL_PREFIX=/home/centos/paraview \
     -DPARAVIEW_USE_MPI=OFF \
     -DPARAVIEW_ENABLE_PYTHON=ON \
     -DPARAVIEW_BUILD_QT_GUI=OFF \
     -DVTK_OPENGL_HAS_OSMESA=ON \
     -DVTK_USE_OFFSCREEN=ON \
     -DVTK_USE_X=OFF \
     ..
make
make install

Running ParaView in client-server mode

Now you are ready to start ParaView server on the VM with SWR rendering: 

./paraview/bin/pvserver --mesa-swr-avx2

Back on your laptop, organize an SSH tunnel from the local port 11111 to the VM's port 11111: 

ssh centos@vm.ip.address -L 11111:localhost:11111

Finally, start the ParaView client on your laptop and connect to localhost:11111. If successful, you should be able to open files on the remote VM. During rendering in the console you should see the message "SWR detected AVX2".

VisIt client-server

Compiling VisIt with OSMesa

VisIt with offscreen rendering support can be built with a single script: 

wget http://portal.nersc.gov/project/visit/releases/2.12.1/build_visit2_12_1
chmod u+x build_visit2_12_1
./build_visit2_12_1 --prefix /home/centos/visit --mesa --system-python \
   --hdf4 --hdf5 --netcdf --silo --szip --xdmf --zlib

This may take a couple of hours. Once finished, you can test the installation with: 

~/visit/bin/visit -cli -nowin

This should start a VisIt Python shell.

Running VisIt in client-server mode

Start VisIt on your laptop and in Options -> Host profiles... edit the connection nickname (let's call it Cloud West), the VM host name, path to VisIt installation (/home/centos/visit) and your username on the VM, and enable tunneling through ssh. Don't forget to save settings with Options -> Save Settings. Then opening a file (File -> Open file... -> Host = Cloud West) you should see the VM's filesystem. Load a file and try to visualize it. Data processing and rendering should be done on the VM, while the result and the GUI controls will be displayed on your laptop.

Upcoming visualization events

  • "Advanced Techniques for Visualization of Very Large Datasets", UBC Okanagan, Kelowna, 2018-Jan-24
  • "Novel visualization techniques from the Visualize This competition" webinar, 2018-Jan-31
  • "Scientific visualization with ParaView", UNBC, Prince George, 2018-May-16

Compute Canada visualization presentation materials

Full- or half-day workshops

Webinars and other short presentations

Wherever possible, the following webinar links provide both a video recording and slides.

Tips and tricks

This section will describe visualization workflows not included into the workshop/webinar slides above. It is meant to be user-editable, so please feel free to add your cool visualization scripts and workflows here so that everyone can benefit from them.

Regional visualization pages

WestGrid

SciNet, HPC at the University of Toronto

SHARCNET

Visualization gallery

You can find a gallery of visualizations based on models run on Compute Canada systems in the visualization gallery. There you can click on individual thumbnails to get more details on each visualization.

How to get visualization help

Please contact Technical support.

Message Passing Interface

Compute Canada Data Base

Retrieved from "https://docs.alliancecan.ca/mediawiki/index.php?title=Visualization&oldid=45131"