1. HOME
  2. What's New
  3. Creating a Job Launcher using JupyterLab

What's New

Some contents are available in Japanese only.

Tech Blog

Creating a Job Launcher using JupyterLab

Now that you’ve read the Quick Starter Guide to using JupyterLab, let’s look at the benefits you can get by creating a job launcher using JupyterLab:

  • A user-friendly interface that allows even HPC beginners to run jobs.
  • Maximum flexibility for quickly modifying or updating job launchers, including when workflows need to be changed.

Simulation engineers in the manufacturing industry can be broadly classified into two types:

  • Type A: Engineers who use simulators as tools and focus on design and analysis work.
  • Type B: Engineers who use their advanced computing skills to build workflows for simulations and develop CAE/CFD technologies.

Type A engineers typically use a Windows environment and often have difficulties using the LINUX command-line for an HPC environment. Therefore, they generally follow templates or manuals created by Type B engineers that summarize workflows for analysis. However, the templates and manuals created by Type B engineers are, in general, not user-friendly enough for Type A engineers. In addition, even if we take the time to create a GUI for this purpose, it will quickly become obsolete when the workflow is revised.

If we assume an environment where JupyterLab is installed on the master node of the job scheduler, let’s firt perform the following two steps:

  • Create a job launcher on Jupyter Notebook using ipywidgets.
  • Use jupyterlab_templates, a JupyterLab extension, so that this job launcher can then be registered as a template, allowing the Admin to manage it centrally.

About ipywidgets:

ipywidgets, also called jupyter-widgets or simply widgets, is an interactive HTML widget for Jupyter Notebook and the IPython kernel. Let’s take a look at how ipywidgets works in Jupyter Notebook.

The first cell is a simple slider. By moving the slider, you can see that the integer value displayed changes. The second cell is a button that, when clicked, will execute the slurm sinfo -s command.

As you can see, it is possible to create a widget with a short python code. You can also edit the python code in the cells to make immediate changes to the widget.

About jupyterlab_templates:

jupyterlab_templates is one of the JupyterLab extensions. With this extension, you can register your Jupyter Notebook as a template. Specifically, by simply placing a Jupyter Notebook in a given directory, you can register this Notebook as a template that can be used by groups or individuals. You can also select the registered template from the Template icon on the Launcher screen. In other words, the templates are registered simply by saving them in a specified directory, so even if there are changes to the job launcher, deployment can easily be done, and admin users can manage the templates centrally. Let’s see how it works when the Jupyter Notebook we just created is registered in jupyterlab_templates.

  • Once registered, you can select registered templates from the Template icon, and use this template in different ways.
  • You can also register multiple versions for easy version control. In addition, by placing and registering templates in the Jupyterlab_templates shared directory, the job launcher templates can be shared within your company.
  • In the directory that the user has pre-selected, there is a file named untitled.ipynb that was created based on the template that was just selected, so it is best to rename this file.
  • Press the Run button to generate the job launcher.
  • As before, you will see a slider and a button to execute the sinfo -s command.
  • The python code in each cell was hidden before, so now we will show it.

A Sample Job Launcher

Let’s take a look at a sample job launcher for Ansys Fluent as an example.

  • Go to the directory where the Ansys Fluent “.jou” file is saved, select the Fluent template as before, and click the Run button to display the job launcher GUI.
  • Enter the parameters required to run Ansys Fluent, such as job name, number of cores, and input file name.
  • Click the Create run.sh button to generate the run script required to run Slurm. Make sure you have created a file with the Slurm settings starting with #SBATCH and the parameters needed to run Fluent.
  • Click the RUN button to execute the job.
  • Check the license consumption status.

Conclusion

By creating a job launcher on Jupyter Notebook using ipywidgets and registering this Notebook in jupyterlab_templates, we can do the following;

  • Provide a user-friendly job launcher that allows even HPC beginners to run jobs.
  • Provide the flexibility to modify and update it immediately by modifying the ipywideget code on the notebook and registering it again in jupyterlab_templates even if the workflow has changed.

In other words, you can easily create a GUI that runs in the browser, and it means you can easily create job launchers right where you are.

<About the Author>
Daisuke Nagao is leading the development of AXXE-L as CTO of XTREME-D.
He first engaged with HPC as a user, and he’s since expanded his expertise to encompass cloud, Deep Learning, and AI.
Daisuke plays jazz piano and is always following emerging technologies, even while playing the piano.

This article was originally published on August 3, 2021 and it’s re-published on August XX, 2021 with some modifications.