MySQL HeatWave Performance

Performance comparison of MySQL HeatWave on OCI with Snowflake, Amazon Redshift, Amazon Aurora, and Amazon RDS for MySQL

Several performance comparisons have been run and the results are presented below. They focus on two different aspects.

• Query processing
  The performance comparison encompasses a variety of benchmarks—TPC-H, TPC-DS, and CH-benCHmark with different dataset sizes (4 TB, 10 TB, and 30 TB) to validate the speedup provided by HeatWave.
• Machine learning
  The performance experiments use a wide variety of publicly known datasets for machine learning classification and regression problems.

1. Common setup for analytic (OLAP) workload test for TPC-H and TPC-DS

• The workload is derived from the TPC's TPC-H and TPC-DS benchmarks.*
• Generate data using the corresponding data generation tool.
• Provision and configure the target service.
• Create the corresponding schema on the target service instance.
• Import the data generated to the target service instance.
• Run queries derived from TPC-H or TPC-DS to test the performance.
• For best performance numbers, always do multiple runs of the query and ignore the first (cold) run.
• You can always do an explain plan to make sure that you get the best expected plan.

2. Common setup for mixed workload test

• The workload is derived from CH-benCHmark.**
• The OLTPBench framework (with changes made to support HeatWave) was used to run the mixed workload benchmark.
• Provision and configure the target service instance.
• Create the mixed workload schema (TPC-C and CH-benCH) on the target service instance.
• Generate and load the 1000 W (100 GB) dataset to the target service instance.
• Set incoming rate of TPC-C transactions at 500 per second (30K transactions per minute) and incoming rate of CH-benCH transactions at 4 per second (240 transactions per minute).
• Run 128 concurrent sessions of TPC-C (OLTP) and 4 concurrent sessions of CH-benCH (OLAP).
• Note the OLTP and OLAP throughput and average latency as reported.

3. HeatWave specific setup

• Optimal encodings are used for certain columns that will be loaded into HeatWave.
• Custom data placement is used for certain tables that will be loaded into HeatWave.
• Mark the tables as offloadable and load them into HeatWave.
• For each query, force offload to HeatWave using the hint (set_var(use_secondary_engine=forced)).
• A straight_join hint is required for certain queries to get the optimal query plan for HeatWave.
• For detailed setup, reference the following:
  • HeatWave TPC-H GitHub for TPC-H.
  • HeatWave GitHub for TPC-DS and CH-benCHmark.

4. Snowflake on AWS specific setup

• For TPC-H, use clustering on ORDERS(o_orderdate) and LINEITEM(l_shipdate) tables.
• You can also try without clustering and pick the scheme that gives better performance.

5. Amazon Redshift specific setup

• Determine the best shape and cluster size for the experiments. In our experiments, we used ra3.4xlarge instance shape.
• For efficient data ingestion, follow the guidelines for enhanced VPC routing.
• Use the default parameters as specified by the Amazon documentation.
• Make sure that the sort keys and distribution keys for each table are optimal for queries. The scripts provided by awslabs can be used for reference.

6. Amazon Aurora specific setup

• Use the largest shape possible so that as much of the data can fit into the buffer cache as possible.
• For the 4 TB TPC-H datasets
  • Use the db.r5.24xlarge shapes.
  • Set the innodb_buffer_pool size to 630 GB.
  • Here are other settings that were modified from their default value in our TPC-H experiments: innodb_sort_buffer_size=67108864; lock_wait_timeout =86400; max_heap_table_size=103079215104; tmp_table_size=103079215104.
• For the 1,000 W TPC-C dataset (used in mixed workload tests)
  • Use the db.r5.8xlarge shape.
  • Use the default parameter settings for innodb_buffer_pool and other parameters.
  • Disable result cache.
  • Enable parallel query.
• Set aurora_disable_hash_join = 0 and aurora_parallel_query = ON to use parallel query.
• Follow the best practices for Aurora database configuration for any other tuning.
• For parallel query to work, make sure that none of the tables are partitioned.
• A straight_join hint can be used if the query plan looks suboptimal.

7. Amazon Aurora specific setup

• Use the largest shape possible so that as much of the data can fit into the buffer cache as possible.
• For the 4 TB TPC-H datasets, use the db.r5.24xlarge shape.
• Set the innodb_buffer_pool size to 650 GB.
• Here are other settings that were modified from their default value in our TPC-H experiments: innodb_buffer_pool_instances=64; innodb_page_cleaners=64; innodb_read_io_threads=64; temptable_max_ram=64424509440; max_heap_table_size=103079215104; tmp_table_size=103079215104.
• Follow the best practices for Amazon RDS for any other tuning.
• Make sure all the primary and foreign key indexes are created.
• A straight_join hint can be used if the query plan looks suboptimal.

8. Results

Performance comparison of HeatWave AutoML with Redshift ML

Setup for the comparison of two different ML problems: classification and regression. For detailed setup, reference the HeatWave AutoML code for performance benchmarks on GitHub.

1. Common setup

• Datasets listed below are publicly available. Select a dataset to start your test.

  Datasets for classification

  Dataset	Explanation	Rows (Training set)	Features
  Airlines	Predict flight delays.	377,568	8
  Bank Marketing	Direct marketing—banking products.	31,648	17
  CNAE-9	Documents with free text business descriptions of Brazilian companies.	757	857
  Connect-4	8-ply positions in the game of connect-4, in which neither player has won yet—predict win/loss.	47,290	43
  Fashion MNIST	Clothing classification problem.	60,000	785
  Nomao	Active learning is used to efficiently detect data that refers to the same place, based on the Nomao browser.	24,126	119
  Numerai	Data is cleaned, regularized, and encrypted global equity data.	67,425	22
  Higgs	Monte Carlo simulations.	10,500,000	29
  Census	Determine if a person’s income exceeds $50,000 a year.	32,561	15
  Titanic	Survival status of individuals.	917	14
  Credit Card Fraud	Identify fraudulent transactions.	199,364	30
  KDD Cup (appetency)	Predict the propensity of customers to buy new products.	35,000	230

  Datasets for regression

  Dataset	Explanation	Rows (Training set)	Features
  Black Friday	Customer purchases on Black Friday.	116,774	10
  Diamonds	Predict the price of a diamond.	37,758	17
  Mercedes	Time the car took to pass testing.	2,946	377
  News Popularity	Predict the number of times articles were shared on social networks.	27,750	60
  NYC Taxi	Predict the tip amount for a New York City taxicab.	407,284	15
  Twitter	The popularity of a topic on social media.	408,275	78

• For each dataset, use 70% of the data for training and 30% of the data for testing. We recommend randomly selecting training data from the data set.
• Compute balanced accuracy for classification and R2 regression for regression on the test set.

2. HeatWave AutoML specific setup

• Use a HeatWave cluster with two HeatWave nodes.
• Run ML_TRAIN with the default arguments, and set task to classification or regression based on the dataset type.
• Score the generated model using ML_SCORE, with the metric set to balanced_accuracy for classification and R2 for regression.

3. Amazon Redshift ML specific setup

• Use a Redshift cluster with two nodes on ra3.4xlarge shape.
• Run CREATE_MODEL with the default values of MAX_CELLS (1M) and MAX_RUNTIME (5400s).
  • For classification dataset, do not specify problem_type and let SageMaker detect the problem type.
  • For regression dataset, run CREATE_MODEL with PROBLEM_TYPE being REGRESSION.
• Create custom SQL scoring functions to calculate balanced accuracy for classification and R2 for regression.

4. Setup for scalability test

• Test with 2, 4, 8, and 16 HeatWave node cluster.
• Run ML_TRAIN on all datasets described in common setup section.
• Compute HeatWave AutoML versus Redshift ML speedup for training on every dataset via the following:
  • (Redshift ML default training time budget (90 minutes)) / (HeatWave AutoML end-to-end ML_TRAIN time)
• Compute geometric mean of speedup across the datasets for each of the cluster configurations.
• Once all five geometric mean speedup values are obtained, plot geomean speedup (y-axis) versus cluster size (x-axis).

5. Results

Performance comparison of MySQL HeatWave on AWS with Amazon Aurora, Amazon Redshift, Snowflake, Azure Synapse Analytics, and Google BigQuery

Several performance comparisons have been run and the results are presented below. They focus on two different aspects.

• OLAP performance
  The performance comparison includes the TPC-H benchmark* with a dataset of 4 TB, to validate the speedup provided by MySQL HeatWave on AWS.
• OLTP performance
  The performance comparison includes the TPC-C benchmark** with a dataset of 10 GB to validate the throughput provided by MySQL HeatWave on AWS.

1. Common setup for analytic (OLAP) workload test for TPC-H

• The workload is derived from TPC's TPC-H benchmarks.*
• Generate data using the corresponding data generation tool.
• Provision and configure the target service.
• Create the corresponding schema on the target service instance.
• Import the data generated to the target service instance.
• Run queries derived from TPC-H to test the performance.
• For best performance numbers, always do multiple runs of the query and ignore the first (cold) run.
• You can always do an explain plan to make sure you get the best expected plan.

2. Common setup for transactional (OLTP) workload test for TPC-C

• The workload is derived from the TPC-C benchmark.**
• The sysbench framework (with additional Lua scripts for TPC-C) is used to run the TPC-C benchmark.
• Provision and configure the target service instance.
• Create the OLTP workload schema (TPC-C) on the target service instance.
• Generate and load the 100 W (10 GB) dataset to the target service instance.
• Start with one concurrent session and gradually increase the number of concurrent sessions to see the impact on throughput and latency.
• Note the OLTP throughput as reported.

3. HeatWave specific setup

• MySQL.32.256GB shape is used for the MySQL node and HeatWave.256GB shape is used for the HeatWave nodes.
• For TPC-H benchmark
  • Optimal encodings are used for certain columns that will be loaded into HeatWave.
  • Custom data placement is used for certain tables that will be loaded into HeatWave.
  • Mark the tables as offloadable and load them into HeatWave.
  • For each query, force offload to HeatWave using the hint (set_var(use_secondary_engine=forced)).
  • A straight_join hint is required for certain queries to get the optimal query plan for HeatWave.
• For TPC-C benchmark
  • Make sure the client and HeatWave instance are in the same physical AZ.
• For detailed setup, reference the following:
  • HeatWave TPC-H GitHub for TPC-H.
  • HeatWave GitHub for TPC-C benchmark.

4. Snowflake on AWS specific setup

• In the experiments, a medium-size cluster is used.
• For TPC-H, use clustering on ORDERS(o_orderdate) and LINEITEM(l_shipdate) tables.
• You can also try without clustering and pick the scheme that gives better performance.

5. Amazon Redshift specific setup

• Determine the best shape and cluster size for the experiments. In the experiments, two nodes of ra3.4xlarge instance shape were used.
• For efficient data ingestion, follow the guidelines for enhanced VPC routing.
• Use the default parameters as specified by the Amazon documentation.
• Make sure the sort keys and distribution keys for each table are optimal for queries. The scripts provided by awslabs can be used for reference.

6. Microsoft Azure Synapse specific setup

• In the experiments, DW 1500c cluster is used.

7. Google BigQuery specific setup

• In the experiments, 400 slots are used for running the queries.

8. Amazon Aurora specific setup for TPC-C benchmark

• Use the db.r5.8xlarge shape.
• Use the default parameter settings for innodb_buffer_pool and other parameters.
• Set adaptive hash index = off; replication = off; redo_log = off.
• Follow the best practices for Aurora database configuration for any other tuning.

9. Results