feat: provide feature flag for shallow cloning

[majin1102]

Context:

#4257 (comment)

This is simpler than I thought...

[Xuanwo]

Is this related to our storage version? Do we need to add it in 2.2 instead of 2.1?

I ask this question since we are very close to stabilize format 2.1 and we don't want to introduce any breaking changes after that.

Cc @jackye1995 @westonpace @wjones127

[majin1102]

Is this related to our storage version? Do we need to add it in 2.2 instead of 2.1?

From my perspective, I recommend including the shallow_clone feature in ​​v2.1​​ for the following reasons:

1. Schedule Uncertainty​​: The timeline for v2.2 is unclear, whereas v2.1 is already in progress.
2. Non-intrusive Design​​: This feature is purely a table-format enhancement, fully backward-compatible with existing datasets（if you don't use old versions read or write cloned dataset）.
3. No Conflict Risk​​: It does not interfere with the main futures intruduced by v2.1 (As I know, 2.1 mainly enhanced file format?).

However, if there are broader compatibility or dependency concerns I might have missed, I’m open to deferring it to v2.2. Would appreciate your thoughts!

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.06%. Comparing base (7480c9c) to head (9de404c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4552      +/-   ##
==========================================
- Coverage   81.06%   81.06%   -0.01%     
==========================================
  Files         308      308              
  Lines      114382   114421      +39     
  Branches   114382   114421      +39     
==========================================
+ Hits        92727    92758      +31     
- Misses      18385    18392       +7     
- Partials     3270     3271       +1     

Flag	Coverage Δ
unittests	81.06% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[majin1102]

@jackye1995 This PR is ready for review

[westonpace]

Is this related to our storage version? Do we need to add it in 2.2 instead of 2.1?

I ask this question since we are very close to stabilize format 2.1 and we don't want to introduce any breaking changes after that.

I have not been following the shallow clone feature very closely but I think it is a table format feature and not a file format feature. The 2.1 stabilization effort is for the file format.

We have not yet had a need to version the table format. We require all changes are backwards compatible and have gone to some substantial effort to avoid any kind of breaking changes. Not all changes are forwards compatible however.

[westonpace]

This looks good to me. An attempt to open a dataset that has shallow clones should fail if the Lance version is too old to understand shallow clones.

I don't think we have any "forwards compatibility" testing framework (e.g. no way for a CI test to confirm that an old reader will fail to read a cloned dataset with the right error message). Did you confirm this manually?

[jackye1995]

Overall looks good to me.

We require all changes are backwards compatible and have gone to some substantial effort to avoid any kind of breaking changes. Not all changes are forwards compatible however.

Yeah I think this is more of a forward compatibility problem of the table format.