feat: dataset supports deep_clone #5250
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| pub fn collect_paths(&self) -> Vec<PathSpec> { | ||
| let mut specs = Vec::new(); | ||
| // Data files: ensure dataset-relative path prefixed with "data/" | ||
| for df in &self.files { | ||
| specs.push(PathSpec { | ||
| path_kind: PathKind::Data, | ||
| path: format!("data/{}", df.path), |
There was a problem hiding this comment.
The new deep-clone flow builds the list of files to copy by calling frag.collect_paths() and then copying the returned dataset-relative paths (see dataset.rs lines 2021‑2044). However, collect_paths (shown here) simply prefixes every data file with data/… (and synthesizes _deletions/…) without considering DataFile.base_id or DeletionFile.base_id. For any dataset that contains fragments stored via Manifest::base_paths—for example a shallow clone referencing another dataset—the actual files live under the referenced base path, not under the shallow clone’s own data/ or _deletions/ directories. As soon as deep_clone tries to copy such a fragment it gets Error::NotFound and the clone fails, so the feature cannot convert a shallow clone into a fully materialized dataset, which is the primary use case. The path collector needs to resolve base_id to the correct root before emitting copy specs.
Useful? React with 👍 / 👎.
majin1102
force-pushed
the
deep_clone
branch
from
be7b215 to
2ff4807
Compare
1 participant
Close #5249