Skip to content

Fix migration might skip some records on big table after job restarting#36878

Merged
sandynz merged 1 commit intoapache:masterfrom
sandynz:pipeline-1
Oct 14, 2025
Merged

Fix migration might skip some records on big table after job restarting#36878
sandynz merged 1 commit intoapache:masterfrom
sandynz:pipeline-1

Conversation

@sandynz
Copy link
Contributor

@sandynz sandynz commented Oct 14, 2025

Related to #33996.

Changes proposed in this pull request:

  • Remove InventoryDumperContext.firstDump

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.
  • I have updated the Release Notes of the current development version. For more details, see Update Release Note

@sandynz sandynz merged commit a65eb5f into apache:master Oct 14, 2025
24 checks passed
@sandynz sandynz deleted the pipeline-1 branch October 14, 2025 08:34
@sandynz
Copy link
Contributor Author

sandynz commented Nov 12, 2025

InventoryDumperContext.firstDump might cause bug.

Test case

Table and configuration:

  • tableA with 300 million records, with integer primary key id AUTO_INCREMENT. Use default SHARDING_SIZE 10 million, then there could be 30 records shards.
  • Use default WORKER_THREAD 20.

Steps to reproduce

  1. Start migration job. (There will be 20 threads to do migration concurrently, and 10 shards tasks are queued)
  2. After job running a very short time, run show migration status {jobId};, make sure processed_records_count is greater than 0 and inventory_finished_percentage is less than 10%.
  3. Restart migration job.

Reason analyze

When job is stopped and there are still some shards tasks are queued, after job is started again, InventoryDumperContext.firstDump will return false (check on job item level, not on every shard task). When queued shards tasks running, query SQL condition will use >, so a shard task will skip one record, >= should be used at the first dump for every shard task.

@sandynz sandynz changed the title Improve InventoryDumper query range after job restarting Fix migration might skip some records on big table after job restarting Nov 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants