Since the H100 GPU from NVIDIA is now accessible through Cloud Service Providers (CSPs), it was only a matter of time before someone decided to benchmark its performance and contrast it with the A100 GPU from the previous generation. We now have a comparison between these two GPUs with an intriguing look into the cost issue courtesy to the benchmarks of MosaicML, a startup business founded by the former CEO of Nervana and GM of Artificial Intelligence (AI) at Intel, Naveen Rao. First, MosaicML trained Generative Pre-trained Transformer (GPT) models using bfloat16 and FP8 Floating Point precision formats on models of various sizes. On cloud GPU instances powered by CoreWeave, all training was done.

Performance-wise, the NVIDIA H100 GPU increased speed by 2.2x to 3.3x. Comparing the price of running various GPUs in the cloud reveals an intriguing result, nevertheless. The H100 SXM GPUs are priced at $4.76/hr/GPU by CoreWeave, compared to $2.21/hr/GPU for the A100 80 GB SXM. It takes less time to train a model on the H100 and costs less to do so, despite the fact that it is 2.2 times more expensive. For researchers and businesses looking to train Large Language Models (LLMs), this inherently makes H100 more appealing and makes selecting the more recent GPU more practical, despite the higher price. Tables of training time, speedup, and cost comparisons between two GPUs are shown below.