Skip to content

[ML-10118] Preserve spark dataframe schema order when create petastorm dataset/dataloader #513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
selitvin merged 23 commits into from
Mar 30, 2020
Merged

[ML-10118] Preserve spark dataframe schema order when create petastorm dataset/dataloader #513

selitvin merged 23 commits into from
Mar 30, 2020

Conversation

@WeichenXu123
Copy link
Collaborator

@WeichenXu123 WeichenXu123 commented Mar 23, 2020 •
edited
Loading

  • Preserve spark dataframe schema order when create petastorm dataset/dataloader
  • Add selected fields for TransformSpec

@WeichenXu123 WeichenXu123 mentioned this pull request Mar 23, 2020
Closed
@WeichenXu123 WeichenXu123 reopened this Mar 24, 2020
@WeichenXu123 WeichenXu123 changed the title [WIP][ML-10118] Keep petastorm dataset/dataloader schema fields order the same with spark dataframe [ML-10118] Preserve spark dataframe schema order when create petastorm dataset/dataloader Mar 26, 2020
@codecov
Copy link

codecov bot commented Mar 26, 2020 •
edited
Loading

Codecov Report

Merging #513 into master will increase coverage by 0.10%.
The diff coverage is 77.77%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #513      +/-   ##
==========================================
+ Coverage   86.02%   86.13%   +0.10%     
==========================================
  Files          81       81              
  Lines        4402     4435      +33     
  Branches      704      713       +9     
==========================================
+ Hits         3787     3820      +33     
  Misses        504      504              
  Partials      111      111
Impacted Files Coverage Δ
petastorm/reader.py 90.99% <ø> (ø)
petastorm/transform.py 85.18% <60.00%> (-14.82%) ⬇️
petastorm/unischema.py 94.71% <100.00%> (+0.13%) ⬆️
petastorm/spark/spark_dataset_converter.py 92.73% <0.00%> (+2.11%) ⬆️
petastorm/pytorch.py 92.68% <0.00%> (+2.43%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b70510...08329f1. Read the comment docs.

liangz1
liangz1 reviewed Mar 26, 2020
View reviewed changes
Copy link
Collaborator

@liangz1 liangz1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I left some questions for discussion.

@selitvin selitvin self-requested a review March 26, 2020 19:57
selitvin
selitvin reviewed Mar 26, 2020
View reviewed changes
selitvin
selitvin approved these changes Mar 26, 2020
View reviewed changes
Copy link
Collaborator

@selitvin selitvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

liangz1
liangz1 approved these changes Mar 27, 2020
View reviewed changes
Copy link
Collaborator

@liangz1 liangz1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@selitvin selitvin merged commit 4a99b9b into uber:master Mar 30, 2020
tkakantousis pushed a commit to logicalclocks/petastorm that referenced this pull request Sep 16, 2020
@WeichenXu123 @tkakantousis
[ML-10118] Preserve spark dataframe schema order when create petastor…
2af1d35 
…m dataset/dataloader (uber#513)

Preserve spark dataframe schema order when create petastorm dataset/dataloader
Add selected fields for TransformSpec
kashishmittal55 pushed a commit to kashishmittal55/petastorm that referenced this pull request Aug 15, 2025
@WeichenXu123
[ML-10118] Preserve spark dataframe schema order when create petastor…
9820f02 
…m dataset/dataloader (uber#513)

Preserve spark dataframe schema order when create petastorm dataset/dataloader
Add selected fields for TransformSpec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@selitvin selitvin selitvin approved these changes

+1 more reviewer

@liangz1 liangz1 liangz1 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@WeichenXu123 @selitvin @liangz1