Epic: from shallow_clone to branch

[majin1102]

This epic captures the evolution from shallow_clone to branch in Lance, based on the design discussion in #4256 #3861. It clarifies how shallow_clone provides the foundational mechanics for branch, how branch brings difference with shallow_clone and outlines the tracked implementation work across runtimes and storage management.

Relationship between shallow_clone and branch

• shallow_clone creates a dataset derived from a base dataset without copying the underlying data files. It references the base data and maintains its own metadata and snapshots, enabling quick duplication with copy-on-write behavior for subsequent writes.
• A branch is effectively a shallow_clone with additional branch metadata
• shallow_clone is a operation across datasets and branch operations(create/delete/checkout) are within one dataset
• Branch makes global file lifetime management possible as discussed in From shallow clone to branching #4256

Scope

• Deliver branch feature built atop shallow_clone mechanics across supported runtimes (Java, Python).
• Ensure shallow_clone supports fragment operations and indexes consistently.
• Implement cross-branch file lifetime management with safe retention and cleanup policies.

Out of Scope

• This epic does not track discussions or any design directly. Only issues with some meaningful PRs are listed in the tracked checklists.

clone Issues

• Dataset support shallow clone (Dataset support basic shallow_clone #4255)
• shallow_clone support indexes (shallow_clone support indexes #4863)
• Provide feature flag for shallow_clone (feat: provide feature flag for shallow cloning #4552
• Support shallow_clone in Java (Support shallow_clone in Java #4854)
• Support shallow_clone in Python (Support shallow_clone in Python #4856)
• Dataset supports deep_clone (Support deep_clone like Deta lake #5249)

Branch issues

• Support basic branch operations(Support basic branch operations #4862)
• Support file lifetime management across branches (Support file lifetime management across branches #4858)
• Support branch in Java (Support branch in Java #4855)
• Support branch in Python (Support branch in Python #4857)
• Add docs for branches(Add docs for branch #5073)
• Support global tags retrieval(feat: support global tag retrieval and improve tag api #5088)

[jackye1995]

What is this issue tracking?

[majin1102]

What is this issue tracking?

Sorry, lately edited the content