Article by Shingo OKUNO, Director and CTO
In the last post, we introduced XTREME-DNA’s two core functional capabilities. One of them was the HPC template, which helps your practical use an HPC cluster on Cloud. So, let’s dive in and talk a little bit more about HPC Templates.
The Challenges of Building an HPC Cluster in the Cloud
Today, public cloud providers provide a few resources that are optimized for HPC, in order to make it easier to build HPC clusters on the public cloud. While this is a welcome trend for HPC experts, it’s not that easy for everyone else. There are many challenges to building an HPC cluster, in most cases requiring many hours of time and much expertise.
For example, you would need to choose and learn about the specific resources required for your job, as well as how to deploy, install, and configure them. And, you would need to know how to tune any device accessed via public cloud. This is not an easy job, and often there are pitfalls.
Therefore, building and operating an HPC cluster takes a lot of working hours, during which you’d still have to pay for your cloud subscription even though you aren’t able to run your jobs until your cluster is properly configured.
HPC Templates Solve These Challenges
HPC templates provide an automated way to configure and test a cluster to ensure that it has the correct configuration for operation in about ten minutes. You’re immediately saving money, both in labor and in cloud usage fees, because every process needed for deployment is pre-configured in each template. And you’re not required to be a cluster expert — our HPC architects based these templates on their extensive experience designing and building HPC clusters.
There are two phases to consider when building an HPC cluster on a public cloud. The first phase is deploying resources. We are continually researching and redesigning HPC cluster architecture on public clouds so that it makes the most of available resources. We review network design, interconnects, storage, accelerators, and so on. These are different between each public cloud, but our HPC templates contain underlying architectures to deploy and provision clusters on Microsoft Azure, Google Public Cloud, and Amazon Web Services.
The second phase is provisioning these deployed resources. XTREME-DNA’s software consists of drivers, middleware, applications, configurations, and more. As you know, every piece of software has different versions and different dependencies. And software conflicts often cause problems with installation and usage. But with XTREME-D you don’t have to worry — our HPC templates have already resolved these problems and have been thoroughly tested.
How Were the HPC Templates Created?
HPC templates are like recipes written by HPC architects, who use their know-how and experience to construct the optimal OS, application, middleware, and setup necessary for specific execution or testing. We are preparing many componentized small recipes. These combinations make it possible to produce many useful HPC templates. And we pre-build Virtual Machine images as much as possible for most recipes in order to provide quick provisioning.
Of course, we can also create custom HPC templates for your specific needs. Please reach out if you’d like to learn more about that service.
By the way, our HPC templates are implemented with our original framework that is based on standard OSS software. We’ll talk about that some other time.
Thank you very much for reading through to the end! We are looking forward to seeing you use HPC clusters built from our HPC templates.