Reduce commit() calls in compression scheduler to improve transaction handling and performance

[coderabbitai]

Context

During the review of PR #1637, it was observed that compression_scheduler.py calls .commit() immediately after every .execute() statement. This pattern prevents batching of SQL statements and causes two issues:

1. Loss of atomicity: Write-after-read operations may lose atomicity if commits occur between reads and writes, potentially allowing overwrites. Currently mitigated by having a single compression scheduler instance per deployment.
2. Performance overhead: Frequent commits add overhead that hurts performance.

Goal

Investigate and implement a solution to reduce the frequency of .commit() calls in the compression scheduler, enabling better use of transactions and improving performance.

References

• PR feat(job-orchestration): Allow concurrent compression job processing by processing batches of compression tasks per job. #1637: feat(job-orchestration): Allow concurrent compression job processing by processing batches of compression tasks per job. #1637
• Requested by: @junhaoliao
• Discussion: feat(job-orchestration): Allow concurrent compression job processing by processing batches of compression tasks per job. #1637 (comment)