Several performance comparisons have been run and the results are presented below. They focus on two different aspects.
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | Google BigQuery | Azure Synapse | |
---|---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.4xlarge | - | - | - |
Cluster size | 15 + 1 MDS | 8 | X-Large (16) | 800 slots | DW 2,500c |
Geomean time | 5.19 seconds | 8.2 seconds | 13.2 seconds | 20.4 seconds | 23.2 seconds |
Annual cost | USD$49,561 | USD$150,784 | USD$280,320 | USD$163,200 | USD$165,575 |
Note: Redshift and Snowflake numbers for 10 TB TPC-DS are provided by a third party.
Note: Redshift pricing is based on one-year reserved instance pricing (paid all upfront). Snowflake pricing is based on standard edition on-demand pricing. Google BigQuery pricing is based on annual flat-rate commitment (per 100 slots), and Azure Synapse pricing is based on the 1 year reserved instance.
MySQL HeatWave | Amazon Redshift | Amazon Aurora | Amazon RDS for MySQL | Snowflake on AWS | |
---|---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.4xlarge | db.r5.24xlarge | db.r5.24xlarge | - |
Cluster size | 5 + 1 MDS | 2 | 1 | 1 | Medium (4) |
Total elapsed time | 381 seconds | 4,189 seconds | 130 hours | 338 hours | 3,183 seconds |
Annual cost | USD$18,585 | USD$37,696 | USD$67,843 | USD$54,393 | USD$70,080 |
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | Azure Synapse | Google BigQuery | |
---|---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.4xlarge | - | DW 1,500c | 400 slots |
Cluster size | 5 + 1 MDS | 2 | Medium (4) | - | - |
Geomean time | 11.6 seconds | 130 seconds | 107 seconds | 31 seconds | 109 seconds |
Annual cost | USD$18,585 | USD$37,696 | USD$70,080 | USD$99,345 | USD$81,600 |
Note: 4 TB TPC-H numbers for Amazon Redshift, Snowflake, Azure Synapse, and Google BigQuery are derived from independent benchmark testing in March 2022.
Note: Redshift pricing is based on one-year reserved instance pricing (paid all upfront). Snowflake pricing is based on standard edition on-demand pricing. Google BigQuery pricing is based on annual flat-rate commitment (per 100 slots), and Azure Synapse pricing is based on the 1 year reserved instance.
MySQL HeatWave | Amazon Redshift | Amazon Aurora | Amazon RDS for MySQL | |
---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
dc2.8xlarge | db.r5.24xlarge | db.r5.24xlarge |
Cluster size | 10 + 1 MDS | 4 | 1 | 1 |
Total elapsed time | 224 seconds | 728 seconds | 130 hours | 338 hours |
Annual cost | USD$34,073 | USD$110,560 | USD$67,843 | USD$54,393 |
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | |
---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.4xlarge | - |
Cluster size | 12 + 1 MDS | 8 | X-Large (16) |
Total elapsed time | 504 seconds | 2,380.4 seconds | 2,350.5 seconds |
Annual cost | USD$40,268 | USD$150,784 | USD$280,320 |
Note: Redshift and Snowflake numbers for 4TB TPC-H and 10 TB TPC-H are provided by a third party.
Note: Redshift pricing is based on one-year reserved instance pricing (paid all upfront). Snowflake pricing is based on standard edition on-demand pricing.
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | |
---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.4xlarge | - |
Cluster size | 25 + 1 MDS | 8 | X-Large (16) |
Total elapsed time | 346.5 seconds | 2,380.4 seconds | 2,350.5 seconds |
Annual cost | USD$80,536 | USD$150,784 | USD$420,480 |
Note: Redshift and Snowflake numbers for 10 TB TPC-H are provided by a third party. Snowflake pricing is based on the enterprise edition.
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | Azure Synapse | Google BigQuery | |
---|---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.16xLarge | - | DW 15,000c | 5,000 slots |
Cluster size | 42 + 1 MDS (MySQL.HeatWave. BM.Standard.E3) |
20 | 3X-Large (64) | - | - |
Geomean time | 20.59 seconds | 32.32 seconds | 78.17 seconds | 35.44 seconds | 108.5 seconds |
Annual cost | USD$149,495 | USD$1,507,840 | USD$1,681,920 | USD$1,249,956 | USD$1,020,000 |
Note: 30 TB TPC-H numbers for Amazon Redshift, Snowflake, Azure Synapse, and Google BigQuery are derived from independent benchmark testing in October 2020.
* Disclaimer: Benchmark queries are derived from the TPC-H and TPC-DS benchmarks, but results are not comparable to published TPC-H and TPC-DS benchmark results since these do not comply with the TPC-H and TPC-DS specifications.
MySQL HeatWave | Amazon Redshift | Snowflake on AWS | Azure Synapse | Google BigQuery | |
---|---|---|---|---|---|
Instance shape | MySQL.HeatWave. VM.Standard.E3 |
ra3.16xLarge | - | DW 15,000c | 5,000 slots |
Cluster size | 75 + 1 MDS (MySQL.HeatWave. BM.Standard.E3) |
20 | 3X-Large (64) | - | - |
Geomean time | 11.4 seconds | 32.32 seconds | 78.17 seconds | 35.44 seconds | 108.5 seconds |
Annual cost | USD$251,714 | USD$1,507,840 | USD$1,681,920 | USD$1,249,956 | USD$1,020,000 |
Note: 30 TB TPC-H numbers for Amazon Redshift, Snowflake, Azure Synapse, and Google BigQuery are derived from independent benchmark testing in October 2020.
* Disclaimer: Benchmark queries are derived from the TPC-H benchmark, but results are not comparable to published TPC-H benchmark results since these do not comply with the TPC-H specification.
MySQL HeatWave | Amazon Aurora | |
---|---|---|
Instance shape | MySQL.HeatWave.VM.Standard.E3 | db.r5.8xlarge |
Cluster size | 2 + 1 MDS | 1 |
OLTP throughput (transactions per minute) | 30,000 | 30,000 |
OLTP latency | 0.02 seconds | 0.02 seconds |
OLAP throughput (transactions per minute) | 6.6 | 0.06 |
OLAP latency | 35 seconds | 637 seconds |
Annual cost | USD$9,293 | USD$22,614 |
** Disclaimer: CH-benCHmark queries were derived from the TPC-C and CH-benCH queries specified in the OLTPBench framework and are not comparable to any published TPC-C or TPC-H benchmark results since these do not comply with the TPC specifications.
Notes: All costs include only the cost of compute. Storage costs are not included and are extra.
Setup for the comparison of two different ML problems: classification and regression. For detailed setup, reference the HeatWave AutoML code for performance benchmarks on GitHub.
Dataset | Explanation | Rows (Training set) | Features |
---|---|---|---|
Airlines | Predict flight delays. | 377,568 | 8 |
Bank Marketing | Direct marketing—banking products. | 31,648 | 17 |
CNAE-9 | Documents with free text business descriptions of Brazilian companies. | 757 | 857 |
Connect-4 | 8-ply positions in the game of connect-4, in which neither player has won yet—predict win/loss. | 47,290 | 43 |
Fashion MNIST | Clothing classification problem. | 60,000 | 785 |
Nomao | Active learning is used to efficiently detect data that refers to the same place, based on the Nomao browser. | 24,126 | 119 |
Numerai | Data is cleaned, regularized, and encrypted global equity data. | 67,425 | 22 |
Higgs | Monte Carlo simulations. | 10,500,000 | 29 |
Census | Determine if a person’s income exceeds $50,000 a year. | 32,561 | 15 |
Titanic | Survival status of individuals. | 917 | 14 |
Credit Card Fraud | Identify fraudulent transactions. | 199,364 | 30 |
KDD Cup (appetency) | Predict the propensity of customers to buy new products. | 35,000 | 230 |
Dataset | Explanation | Rows (Training set) | Features |
---|---|---|---|
Black Friday | Customer purchases on Black Friday. | 116,774 | 10 |
Diamonds | Predict the price of a diamond. | 37,758 | 17 |
Mercedes | Time the car took to pass testing. | 2,946 | 377 |
News Popularity | Predict the number of times articles were shared on social networks. | 27,750 | 60 |
NYC Taxi | Predict the tip amount for a New York City taxicab. | 407,284 | 15 |
The popularity of a topic on social media. | 408,275 | 78 |
Dataset | Accuracy | Training time (minutes) | Speedup | ||
---|---|---|---|---|---|
Redshift ML | HeatWave AutoML | Redshift ML | HeatWave AutoML | ||
Airlines | 0.5 | 0.6524 | 90.00 | 2.71 | 33.21 |
Bank | 0.8378 | 0.7115 | 90.00 | 3.72 | 24.19 |
CNAE-9 | X | 0.9167 | X | 5.91 | X |
Connect-4 | 0.6752 | 0.6970 | 90.00 | 7.13 | 12.62 |
Fashion MNIST | X | 0.9073 | X | 181.85 | X |
Nomao | 0.9512 | 0.9602 | 90.00 | 3.30 | 27.27 |
Numerai | 0.5 | 0.5184 | 90.00 | 0.34 | 264.71 |
Higgs | 0.5 | 0.758 | 90.00 | 68.58 | 1.31 |
Census | 0.7985 | 0.7946 | 90.00 | 1.22 | 73.77 |
Titanic | 0.9571 | 0.7660 | 90.00 | 0.47 | 191.49 |
CC Fraud | 0.9154 | 0.9256 | 90.00 | 29.06 | 3.10 |
KDD Cup | X | 0.5 | X | 3.55 | X |
Geomean | 0.712 | 0.754 | 90.00 | 3.561 | 25.271 |
Dataset | Training cost ($) | Speedup | ||
---|---|---|---|---|
Redshift ML list | Redshift ML with one-year plan |
HeatWave AutoML | ||
Airlines | 20.00 | 6.23 | 0.0479 | 130.03 |
Bank | 10.76 | 5.68 | 0.0658 | 86.30 |
CNAE-9 | 12.97 | X | 0.10458 | X |
Connect-4 | 20.00 | 6.18 | 0.1261 | 49.05 |
Fashion MNIST | 20.00 | X | 3.2151 | X |
Nomao | 20.00 | 5.96 | 0.0583 | 102.14 |
Numerai | 20.00 | 5.49 | 0.0060 | 913.49 |
Higgs | 20.00 | 7.27 | 1.2125 | 5.99 |
Census | 9.77 | 6.12 | 0.0216 | 283.95 |
Titanic | 0.26 | 5.60 | 0.0083 | 674.32 |
CC Fraud | 20.00 | 6.70 | 0.0083 | 13.03 |
KDD Cup | 20.00 | X | 0.5138 | X |
Geomean | 10.62 | 6.115 | 0.063 | 97.13 |
Dataset | Accuracy | Training time (minutes) | Speedup | ||
---|---|---|---|---|---|
Redshift ML | HeatWave AutoML | Redshift ML | HeatWave AutoML | ||
Black Friday | 0.54 | 0.53 | 90.00 | 1.14 | 78.80 |
Diamonds | 0.98 | 0.98 | 90.00 | 2.40 | 37.42 |
Mercedes | X | 0.61 | X | 1.16 | X |
News Popularity | 0.02 | 0.01 | 90.00 | 0.60 | 149.13 |
NYC Taxi | 0.19 | 0.25 | 90.00 | 7.34 | 12.26 |
0.88 | 0.93 | 90.00 | 44.24 | 2.03 | |
Geomean | 0.27 | 0.26 | 90.00 | 3.52 | 25.58 |
Dataset | Training cost ($) | Lower cost | ||
---|---|---|---|---|
Redshift ML list |
Redshift ML cost with one-year plan |
HeatWave AutoML | Scalability | |
Black Friday | 20.00 | 2.95 | 0.02 | 146.10 |
Diamonds | 7.55 | 5.13 | 0.04 | 120.61 |
Mercedes | 20.00 | X | 0.02 | X |
News Popularity | 20.00 | 4.15 | 0.01 | 389.08 |
NYC Taxi | 20.00 | 2.82 | 0.13 | 21.76 |
20.00 | 3.64 | 0.78 | 4.66 | |
Geomean | 17.00 | 3.64 | 0.06 | 58.66 |
Several performance comparisons have been run and the results are presented below. They focus on two different aspects.
MySQL HeatWave on AWS | Amazon Redshift | Snowflake on AWS | Azure Synapse | Google BigQuery | |
---|---|---|---|---|---|
Instance shape | HeatWave.256GB + MySQL.32.256GB | ra3.4xlarge | - | - | - |
Cluster size | 10 + 1 MySQL node | 2 nodes | Medium | DW 1500c | 400 slots |
Geomean time | 6.53 seconds | 130.62 seconds | 107.27 seconds | 31.8 seconds | 109.47 seconds |
Price performance | US$0,023 | US$0,156 | US$0,238 | US$0,1 | US$0,283 |
Note: Redshift, Snowflake, Synapse and Google BigQuery numbers for 4 TB TPC-H are provided by a third party.
Note: Redshift pricing is based on one-year reserved instance pricing (paid all upfront). Snowflake pricing is based on standard edition on-demand pricing. Google BigQuery pricing is based on annual flat-rate commitment (per 100 slots), and Azure Synapse pricing is based on the one-year reserved instance.
*Disclaimer: Benchmark queries are derived from the TPC-H benchmarks, but results aren’t comparable to published TPC-H benchmark results since these don't comply with the TPC-H specifications.
MySQL HeatWave on AWS node type: MySQL.32.256GB.
Amazon Aurora node type: db.r5.8xlarge.
Concurrency | 1 | 4 | 16 | 64 | 128 | 256 | 512 | 1,024 | 2,048 | 4,096 |
---|---|---|---|---|---|---|---|---|---|---|
Amazon Aurora throughput | 116 | 471 | 1,411 | 3,138 | 4,615 | 5,081 | 4,784 | 2,487 | 574 | 245 |
MySQL HeatWave throughput | 86 | 322 | 1,040 | 3,314 | 5,198 | 6,192 | 6,195 | 5,953 | 6,080 | 6,001 |
Note: Aurora numbers for TPC-C are provided by a third party.
**Disclaimer: Benchmark queries are derived from the TPC-C benchmarks, but results aren’t comparable to published TPC-C benchmark results since these don’t comply with the TPC-C specifications.