feat: Reuse existing file instead of reopening during shuffle write

[zuston]

Which issue does this PR close?

Closes #.

Rationale for this change

Reduce the system call cost to reuse the existing file

What changes are included in this PR?

How are these changes tested?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.19%. Comparing base (f09f8af) to head (3ec2047).
⚠️ Report is 614 commits behind head on main.

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #2577       +/-   ##
=============================================
- Coverage     56.12%   42.19%   -13.94%     
- Complexity      976     1093      +117     
=============================================
  Files           119      146       +27     
  Lines         11743    13747     +2004     
  Branches       2251     2353      +102     
=============================================
- Hits           6591     5800      -791     
- Misses         4012     6978     +2966     
+ Partials       1140      969      -171     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wForget]

spill_data.file is opened in write-only mode.

datafusion-comet/native/core/src/execution/shuffle/shuffle_writer.rs

Lines 1169 to 1180 in 8f89c1c

	let spill_data = OpenOptions::new()
	.write(true)
	.create(true)
	.truncate(true)
	.open(spill_file.path())
	.map_err(|e| {
	DataFusionError::Execution(format!("Error occurred while spilling {e}"))
	})?;
	self.spill_file = Some(SpillFile {
	temp_file: spill_file,
	file: spill_data,
	});

[zuston]

Nice catch

[wForget]

Also, could you provide more information about the Rationale for this change?

[zuston]

Also, could you provide more information about the Rationale for this change?

updated.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.