High Performance Computing (HPC) refers to the practice of aggregating computing power in a way that delivers much higher horsepower than traditional computers and servers. HPC, or supercomputing, is like everyday computing, only more powerful. It is a way of processing huge volumes of data at very high speeds using multiple computers and storage devices as a cohesive fabric. HPC makes it possible to explore and find answers to some of the world’s biggest problems in science, engineering, and business.
Today, HPC is used to solve complex, performance-intensive problems—and organizations are increasingly moving HPC workloads to the cloud. HPC in the cloud is changing the economics of product development and research because it requires fewer prototypes, accelerates testing, and decreases time to market.
Some workloads, such as DNA sequencing, are simply too immense for any single computer to process. HPC or supercomputing environments address these large and complex challenges with individual nodes (computers) working together in a cluster (connected group) to perform massive amounts of computing in a short period of time. Creating and removing these clusters is often automated in the cloud to reduce costs.
HPC can be run on many kinds of workloads, but the two most common are embarrassingly parallel workloads and tightly coupled workloads.
Are computational problems divided into small, simple, and independent tasks that can be run at the same time, often with little or no communication between them. For example, a company might submit 100 million credit card records to individual processor cores in a cluster of nodes. Processing one credit card record is a small task, and when 100 million records are spread across the cluster, those small tasks can be performed at the same time (in parallel) at astonishing speeds. Common use cases include risk simulations, molecular modeling, contextual search, and logistics simulations.
Typically take a large shared workload and break it into smaller tasks that communicate continuously. In other words, the different nodes in the cluster communicate with one another as they perform their processing. Common use cases include computational fluid dynamics, weather forecast modeling, material simulations, automobile collision emulations, geospatial simulations, and traffic management.
HPC has been a critical part of academic research and industry innovation for decades. HPC helps engineers, data scientists, designers, and other researchers solve large, complex problems in far less time and at less cost than traditional computing.
The primary benefits of HPC are:
Fortune 1000 companies in nearly every industry employ HPC, and its popularity is growing. According to Hyperion Research, the global HPC market is expected to reach US$44 billion by 2022.
The following are some of the industries using HPC and the types of workloads HPC is helping them perform:
HPC can be performed on premise, in the cloud, or in a hybrid model that involves some of each.
In an on-premise HPC deployment, a business or research institution builds an HPC cluster full of servers, storage solutions, and other infrastructure that they manage and upgrade over time. In a cloud HPC deployment, a cloud service provider administers and manages the infrastructure, and organizations use it on a pay-as-you-go model.
Some organizations use hybrid deployments, especially those that have invested in an on-premise infrastructure but also want to take advantage of the speed, flexibility, and cost savings of the cloud. They can use the cloud to run some HPC workloads on an ongoing basis, and turn to cloud services on an ad hoc basis, whenever queue time becomes an issue on premise.
Organizations with on-premise HPC environments gain a great deal of control over their operations, but they must contend with several challenges, including
In part because of the costs and other challenges of on-premise environments, cloud-based HPC deployments are becoming more popular, with Market Research Future anticipating 21% worldwide market growth from 2017 to 2023. When businesses run their HPC workloads in the cloud, they pay only for what they use and can quickly ramp up or down as their needs change.
To win and retain customers, top cloud providers maintain leading-edge technologies that are specifically architected for HPC workloads, so there is no danger of reduced performance as on-premise equipment ages. Cloud providers offer the newest and fastest CPUs and GPUs, as well as low-latency flash storage, lightning-fast RDMA networks, and enterprise-class security. The services are available all day, every day, with little or no queue time.
All cloud providers are not created equal. Some clouds are not designed for HPC and can’t provide optimal performance during peak periods of demanding workloads. The four traits to consider when selecting a cloud provider are
Generally, it’s best to look for bare metal cloud services that offer more control and performance. Combined with RDMA cluster networking, bare metal HPC provides identical results to what you get with similar hardware on premise.
Businesses and institutions across multiple industries are turning to HPC, driving growth that is expected to continue for many years to come. The global HPC market is expected to expand from US$31 billion in 2017 to US$50 billion in 2023. As cloud performance continues to improve and become even more reliable and powerful, much of that growth is expected to be in cloud-based HPC deployments that relieve businesses of the need to invest millions in data center infrastructure and related costs.
In the near future, expect to see big data and HPC converging, with the same large cluster of computers used to analyze big data and run simulations and other HPC workloads. As those two trends converge, the result will be more computing power and capacity for each, leading to even more groundbreaking research and innovation.
i Earl Joseph, Steve Conway, Bob Sorensen, Alex Norton. Hyperion Research Update: ISC19. https://hyperionresearch.com/wp-content/uploads/2019/06/Hyperion-Research-ISC19-Breakfast-Briefing-Presentation-June-2019.pdf