Merge DM dev docs to TiDB dev docs #8042
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| --- | ||
| title: DM 1.0-GA 性能测试报告 | ||
|
||
| aliases: ['/docs-cn/tidb-data-migration/dev/benchmark-v1.0-ga/'] | ||
| --- | ||
|
|
||
| # DM 1.0-GA 性能测试报告 | ||
|
|
||
| 本报告记录了对 1.0-GA 版本的 DM 进行性能测试的目的、环境、场景和结果。 | ||
|
|
||
| ## 测试目的 | ||
|
|
||
| 该性能测试用于评估使用 DM 进行全量数据导入和增量数据复制的性能上限，并根据测试结果提供 DM 迁移任务的参考配置。 | ||
|
|
||
| ## 测试环境 | ||
|
|
||
| ### 测试机器信息 | ||
|
|
||
| 系统信息： | ||
|
|
||
| | 机器 IP | 操作系统 | 内核版本 | 文件系统类型 | | ||
| | :---------: | :---------------------------: | :-----------------------: | :--------------: | | ||
| | 172.16.4.39 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
| | 172.16.4.40 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
| | 172.16.4.41 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
| | 172.16.4.42 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
| | 172.16.4.43 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
| | 172.16.4.44 | CentOS Linux release 7.6.1810 | 3.10.0-957.1.3.el7.x86_64 | ext4 | | ||
|
|
||
| 硬件信息： | ||
|
|
||
| | 类别 | 指标 | | ||
| | :----------: | :-------------------------------------------------: | | ||
| | CPU | 40 vCPUs, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | | ||
| | 内存 | 192GB, 12 条 16GB DIMM DDR4 2133 MHz | | ||
| | 磁盘 | Intel DC P4510 4TB NVMe PCIe 3.0 | | ||
| | 网卡 | 万兆网卡 | | ||
|
|
||
| 其他： | ||
|
|
||
| * 服务器间网络延迟：rtt min/avg/max/mdev = 0.074/0.088/0.121/0.019 ms | ||
|
|
||
| ### 集群拓扑 | ||
|
|
||
| | 机器 IP | 部署的服务 | | ||
| | :---------: | :--------------------------------: | | ||
| | 172.16.4.39 | PD1, DM-worker1, DM-master | | ||
| | 172.16.4.40 | PD2, MySQL1 | | ||
| | 172.16.4.41 | PD3, TiDB | | ||
| | 172.16.4.42 | TiKV1 | | ||
| | 172.16.4.43 | TiKV2 | | ||
| | 172.16.4.44 | TiKV3 | | ||
|
|
||
| ### 各服务版本信息 | ||
|
|
||
| - MySQL 版本: 5.7.27-log | ||
| - TiDB 版本: v4.0.0-alpha-198-gbde7f440e | ||
| - DM 版本: v1.0.1 | ||
| - Sysbench 版本: 1.0.17 | ||
|
|
||
| ## 测试场景 | ||
|
|
||
| 可以参考[性能测试](/dm/dm-performance-test.md)中介绍的测试场景，测试单个 MySQL 实例到 TiDB 的数据迁移： MySQL1 (172.16.4.40) -> DM-worker -> TiDB (172.16.4.41)。 | ||
|
|
||
| ### 全量导入性能测试 | ||
|
|
||
| 可以参考[全量导入性能测试用例](/dm/dm-performance-test.md#全量导入性能测试用例)中介绍的方法进行测试。 | ||
|
|
||
| #### 全量导入性能测试结果 | ||
|
|
||
| 在 `mydumpers` 中使用 `--rows` 选项可以开启单表多线程并发导出（当前 `mydumpers` 在 MySQL 的单表并发会优先选出一列做拆分基准，选择优先级为主键>唯一索引>普通索引，选出目标列后需保证该列为 INT 类型；针对 `TiDB` 的并发导出则会优先尝试 `_tidb_rowid` 列），以下的第一项测试用于验证开启单表并发导出可以提高数据导出性能。 | ||
|
|
||
| | 测试项 | 导出线程数 | mydumpers extra-args 参数 | 导出速度 (MB/s) | | ||
| | :----------------: | :---------: | :---------------------------------: | :---------------: | | ||
| | 开启单表并发导出 | 32 | "--rows 320000 --regex '^sbtest.*'" | 191.03 | | ||
| | 不开启单表并发导出 | 4 | "--regex '^sbtest.*'" | 72.22 | | ||
|
|
||
| | 测试项 | 事务执行延迟 (s) | 每条插入语句包含的行数 | 导入数据量 (GB) | 导入时间 (s) | 导入速度 (MB/s) | | ||
| | :-------: | :--------------: | :--------------------: | :-------------: | :----------: | :-------------: | | ||
| | load data | 1.737 | 4878 | 38.14 | 2346.9 | 16.64 | | ||
|
|
||
| #### 在 load 处理单元使用不同 pool size 的性能测试对比 | ||
|
|
||
| 该测试中使用 `sysbench` 全量导入的数据量为 3.78 GB，测试数据如下所示： | ||
|
|
||
| | load 处理单元 pool size | 事务执行最大延迟 (s) | 导入时间 (s) | 导入速度 (MB/s) | TiDB 99 duration (s) | | ||
| | :---------------------: | :--------------: | :----------: | :-------------: | :------------------: | | ||
| | 2 | 0.250 | 425.9 | 9.1 | 0.23 | | ||
| | 4 | 0.523 | 360.1 | 10.7 | 0.41 | | ||
| | 8 | 0.986 | 267.0 | 14.5 | 0.93 | | ||
| | 16 | 2.022 | 265.9 | 14.5 | 2.68 | | ||
| | 32 | 3.778 | 262.3 | 14.7 | 6.39 | | ||
| | 64 | 7.452 | 281.9 | 13.7 | 8.00 | | ||
|
|
||
| #### 导入数据时每条插入语句包含行数不同的情况下的性能测试对比 | ||
|
|
||
| 该测试中全量导入的数据量为 3.78 GB，load 处理单元 `pool-size` 大小为 32。插入语句包含行数通过 dump 处理单元的 `--statement-size` 来控制。 | ||
|
|
||
| | 每条语句中包含的行数 | mydumper extra-args 参数 | 事务执行最大延迟 (s) | 导入时间 (s) | 导入速度 (MB/s) | TiDB 99 duration (s) | | ||
| | :------------------------: | :-----------------------: | :---------------: | :----------: | :-------------: | :------------------: | | ||
| | 7426 | -s 1500000 -r 320000 | 6.982 | 258.3 | 15.0 | 10.34 | | ||
| | 4903 | -s 1000000 -r 320000 | 3.778 | 262.3 | 14.7 | 6.39 | | ||
| | 2470 | -s 500000 -r 320000 | 1.962 | 271.36 | 14.3 | 2.00 | | ||
| | 1236 | -s 250000 -r 320000 | 1.911 | 283.3 | 13.7 | 1.50 | | ||
| | 618 | -s 125000 -r 320000 | 0.683 | 299.9 | 12.9 | 0.73 | | ||
| | 310 | -s 62500 -r 320000 | 0.413 | 322.6 | 12.0 | 0.49 | | ||
|
|
||
| ### 增量复制性能测试用例 | ||
|
|
||
| 使用[增量复制性能测试用例](/dm/dm-performance-test.md#增量复制性能测试用例)中介绍的方法进行测试。 | ||
|
|
||
| #### 增量复制性能测试结果 | ||
|
|
||
| 该性能测试中复制任务 `sync` 处理单元 `worker-count` 设置为 32，`batch` 大小设置为 100。 | ||
|
|
||
| | 组件 | qps | tps | 95% 延迟 | | ||
| | :------------------------: | :----------------------------------------------------------: | :-------------------------------------------------------------: | :--------------------------: | | ||
| | MySQL | 42.79k | 42.79k | 1.18ms | | ||
| | DM relay log unit | - | 11.3MB/s | 45us (binlog 读延迟) | | ||
| | DM binlog replication unit | 22.97k (单位时间内接收到的不被忽略的 binlog event 数量) | - | 20ms (事务执行时间) | | ||
| | TiDB | 31.30k (Begin/Commit 3.93k Insert 22.76k) | 4.16k | 95%: 6.4ms 99%: 9ms | | ||
|
|
||
| #### 在 sync 处理单元使用不同并发度的性能测试对比 | ||
|
|
||
| | sync 处理单元 worker-count 数 | DM 内部 job tps | 事务执行最大延迟 (ms) | TiDB qps | TiDB 99 duration (ms) | | ||
| | :---------------------------: | :-------------: | :-----------------------: | :------: | :-------------------: | | ||
| | 4 | 7074 | 63 | 7.1k | 3 | | ||
| | 8 | 14684 | 64 | 14.9k | 4 | | ||
| | 16 | 23486 | 56 | 24.9k | 6 | | ||
| | 32 | 23345 | 28 | 29.2k | 10 | | ||
| | 64 | 23302 | 30 | 31.2k | 16 | | ||
| | 1024 | 22225 | 70 | 56.9k | 70 | | ||
|
|
||
| #### 不同数据分布的增量复制性能测试对比 | ||
|
|
||
| | sysbench 语句类型| DM relay log 持久化速率 (MB/s) | DM 内部 job tps | 事务执行最大延迟 (ms) | TiDB qps | TiDB 99 duration (ms) | | ||
| | :--------------: | :----------------------------: | :-------------: | :------------------: | :------: | :-------------------: | | ||
| | insert_only | 11.3 | 23345 | 28 | 29.2k | 10 | | ||
| | write_only | 18.7 | 33470 | 129 | 34.6k | 11 | | ||
|
|
||
| ## 推荐迁移任务参数配置 | ||
|
|
||
| ### dump 处理单元 | ||
|
|
||
| 推荐每一条插入语句的大小在 200KB ~ 1MB 之间，相应每条语句包含的行数大约在 1000-5000（具体包含的语句行数与实际场景中每行数据大小有关）。 | ||
|
|
||
| ### load 处理单元 | ||
|
|
||
| 推荐 `pool-size` 设置为 16。 | ||
|
|
||
| ### sync 处理单元 | ||
|
|
||
| 推荐将 `batch` 设置为 100，`worker-count` 设置为 16 ~ 32。 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.