Skip to content

fix preprocess script with processbar, integrity check and logs#16608

Merged
luotao1 merged 2 commits intoPaddlePaddle:developfrom
lidanqing-vv:lidanqing/preprocess_after1.4
Apr 2, 2019
Merged

fix preprocess script with processbar, integrity check and logs#16608
luotao1 merged 2 commits intoPaddlePaddle:developfrom
lidanqing-vv:lidanqing/preprocess_after1.4

Conversation

@lidanqing-vv
Copy link
Copy Markdown
Contributor

@lidanqing-vv lidanqing-vv commented Apr 1, 2019

Fix reviews of PR 16529, rebased on release 1.4

  1. process bar added
  2. add check size and check integrity for existing binary file, for comment: Exisiting file is incomplete
  3. Write clear logging
  4. The unzipped file is in .cache/paddle/dataset/int8/download/full_data, for comment: unzip not inplace

@lidanqing-vv
Copy link
Copy Markdown
Contributor Author

@kbinias @bingyanghuang @Sand3r- @luotao1 Please review this new PR rebased on release 1.4. (old one closed). Thank you

Comment thread paddle/fluid/inference/tests/api/full_ILSVRC2012_val_preprocess.py
Copy link
Copy Markdown
Contributor

@kbinias kbinias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this code on my developer machine, the expected result is binary file.

Comment thread paddle/fluid/inference/tests/api/full_ILSVRC2012_val_preprocess.py Outdated
Comment thread paddle/fluid/inference/tests/api/full_ILSVRC2012_val_preprocess.py Outdated
Comment thread paddle/fluid/inference/tests/api/full_ILSVRC2012_val_preprocess.py Outdated
@luotao1
Copy link
Copy Markdown
Contributor

luotao1 commented Apr 2, 2019

The log is very nice, thanks very much!

Start to download and convert 50000 images to binary file...
Downloading full ImageNet Validation dataset ...

Downloaded part /dev/shm/.cache/paddle/dataset/int8/download/ILSVRC2012_img_val.tar.gz.partaa


Downloaded part /dev/shm/.cache/paddle/dataset/int8/download/ILSVRC2012_img_val.tar.gz.partab

Extracting...

Extracted. Full Imagenet Validation dataset is located at /dev/shm/.cache/paddle/dataset/int8/download/full_data/ILSVRC2012

Converting 50000 images to binary file ...

[===========                                       ]22%

Copy link
Copy Markdown
Contributor

@luotao1 luotao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please cherry-pick to release/1.4, and note that the commit info should have test=release/1.4.

@luotao1 luotao1 merged commit 2ca0de3 into PaddlePaddle:develop Apr 2, 2019
@luotao1 luotao1 added the int8 label Apr 2, 2019
yihuaxu pushed a commit to yihuaxu/Paddle that referenced this pull request Apr 2, 2019
…lePaddle#16608)

test=release/1.4

* fix preprocess script with processbar, integrity check and logs

* delete unnecessary empty lines, change function name
yihuaxu pushed a commit to yihuaxu/Paddle that referenced this pull request Apr 2, 2019
…lePaddle#16608)

test=release/1.4

* fix preprocess script with processbar, integrity check and logs

* delete unnecessary empty lines, change function name
luotao1 added a commit that referenced this pull request Apr 3, 2019
@lidanqing-vv lidanqing-vv deleted the lidanqing/preprocess_after1.4 branch April 16, 2019 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants