SVHN dataset for torchvision #98

uridah · 2017-03-10T20:25:10Z

In reference to #59
cc: @mjpieters

Replacing scipy.misc.lena() with scipy.misc.face()

Syncing forked repository

Adding SVHN dataset http://ufldl.stanford.edu/housenumbers/ (Format 2) for torchvision

soumith · 2017-03-10T20:31:58Z

@uridah do you have a notebook that checks for sanity of this code, like the notebook that loads some CIFAR10 images and displays them:
https://github.com/pytorch/vision/blob/master/test/sanity_checks.ipynb

uridah · 2017-03-10T20:49:53Z

@soumith I don't but I can create one.

mjpieters · 2017-03-10T21:52:10Z

I don't know if PyTorch follows PEP8 as a styleguide? If so, you way want to run flake8 on your code and fix the issues that finds; I see a few whitespace issues in it. :-) If not, I'll leave it to the project owners to set the requirements.

soumith · 2017-03-10T21:54:05Z

thanks @mjpieters . We have a LINT check as part of the contbuild, and it is failing:

https://travis-ci.org/pytorch/vision/builds/209884377

Locally doing:

pip install flake8
flake8 .

will show exact LINT errors you need to fix @uridah

torchvision/datasets/svhn.py

+                raise RuntimeError('Dataset not found or corrupted.' +
+                               ' You can use download=True to download it')
+
+            self.train_data = []


torchvision/datasets/svhn.py

+        self.dataset = dataset  # training set or test set or extra set
+
+        # download and load the data 
+        if self.dataset=='train':


torchvision/datasets/svhn.py

+
+    def __len__(self):
+        if self.dataset == 'train':
+            return 73257


torchvision/datasets/svhn.py

+        self.root = root
+        self.transform = transform
+        self.target_transform = target_transform
+        self.dataset = dataset  # training set or test set or extra set


fmassa · 2017-03-10T23:16:42Z

@uridah I made some small inline comments (very minor and are only style changes). The PR looks good, thanks!

- now using dictionary for urls, filenames and md5s - updated len function - renamed 'dataset' keyword to split - fixed whitespaces using flake8

uridah · 2017-03-13T22:12:45Z

@fmassa Your comments very very useful and really helped concise the code
@soumith I added a new notebook named sanity_checks1.ipynb which is basically same as the sanity_checks.ipynb notebook, but in addition, is also calling SVHN datasets for transformations https://github.com/uridah/vision/blob/master/test/sanity_checks1.ipynb
Also, ran the code through flake8
@mjpieters

torchvision/datasets/svhn.py

+            # reading(loading) mat file as array
+            loaded_mat = sio.loadmat(os.path.join(root, self.filename))
+
+            if self.split != 'test':


torchvision/datasets/svhn.py

+                self.test_labels = loaded_mat['y']
+                self.test_data = np.transpose(self.test_data, (3, 2, 1, 0))
+        else:
+            print ("Wrong dataset entered! Please use split=train or split=extra or split=test")


torchvision/datasets/svhn.py

+        self.target_transform = target_transform
+        self.split = split  # training set or test set or extra set
+
+        if self.split in self.split_list:


torchvision/datasets/svhn.py

+            print ("Wrong dataset entered! Please use split=train or split=extra or split=test")
+
+    def __getitem__(self, index):
+        if self.split == 'train' or self.split == 'extra':


torchvision/datasets/svhn.py

+        return img, target
+
+    def __len__(self):
+        return len(self.train_data)


fmassa · 2017-03-13T22:22:36Z

Hi @uridah
I made some more comments. The code is looking very nice!

uridah · 2017-03-13T22:32:39Z

Thanks @fmassa. Updated according to your suggestions

torchvision/datasets/svhn.py

+
+        if self.split not in self.split_list:
+            raise ValueError('Wrong split entered! Please use split=train or split=extra or split=test')
+        else:


torchvision/datasets/svhn.py

+            self.download()
+
+        if not self._check_integrity():
+                raise RuntimeError('Dataset not found or corrupted.' +


fmassa · 2017-03-13T22:51:39Z

About the check_sanity1.pynb file, maybe you could integrate your changes into the check_sanity.pynb file? What do you think?

uridah · 2017-03-13T22:53:37Z

I will, if you guys think it's good enough.

fmassa · 2017-03-13T23:00:01Z

@uridah another thing, it would be great if you could add an entry in the doc in README.rst, like the one in Cifar10 for example. Maybe an example in the doc would be enough?
cc @soumith

uridah · 2017-03-15T18:05:57Z

here is the pull request for the change in documentation: #104
@fmassa

fmassa · 2017-03-15T23:12:18Z

Hi @uridah
So, looking at the notebook that you added, I have the impression that the numbers are rotated. Maybe the x and y axis in the dataset are inverted, and you need to transpose them?
Also, could you please fix the indentation in the line I commented?
Once those are fixed, I think there is no need for the notebook (but @soumith knows better), so could you please remove it?
After that, the PR is good to be merged!

Patch 2

uridah · 2017-03-16T07:45:56Z

@fmassa as it turns out I needed it to transpose along 3,2,0,1 axis instead of 3,2,1,0. I have fixed that and updated the indentation and sanity_checks1.ipynb. Please have a look and let me know if anything else needs to be changed.
Also, I created a separate request to update the documentation but now I have merged it with this one.

soumith · 2017-03-16T21:38:16Z

Thanks Uridah, as you saw the last 4 commits, I made some minor changes to your PR. But it looked great. Merged into master now!!!

vabh · 2017-07-03T14:54:50Z

Hello,

It seems that the labels returned are in the range 1-10 (the data set assigns class 10 to the digit 0 and class d for all other digits, d [ http://ufldl.stanford.edu/housenumbers/ see section overview ]).

Given that some of the loss functions (CELoss, NLLLoss) expect the class labels to be in the range [0, C-1], shouldn't that be reflected here as well?

Another thing is that the the returned labels are of type ByteTensor and of size batchSize x 1. Again, CELoss, etc, expect it to be a 1d LongTensor.

I can make a PR if the current behaviour is inconsistent and should be made similar to the other data sets, for example as in CIFAR10

soumith · 2017-07-03T16:22:27Z

@vabh if you could make a PR to reflect this, that'd be great. Thanks.

vabh · 2017-07-03T18:45:29Z

@soumith I made one here: #194

…mlperf/community (pytorch#98)

uridah added 5 commits March 8, 2017 19:35

lena() is no longer included in SciPy, replacing it with face()

bff9af0

Merge pull request #1 from uridah/uridah-vision

00a6350

Replacing scipy.misc.lena() with scipy.misc.face()

Merge pull request #2 from pytorch/master

db5a830

Syncing forked repository

SVHN dataset

460f3a8

Adding SVHN dataset http://ufldl.stanford.edu/housenumbers/ (Format 2) for torchvision

Adding svhn.py to __init__.py

f189f56

fmassa reviewed Mar 10, 2017

View reviewed changes

torchvision/datasets/svhn.py Outdated

raise RuntimeError('Dataset not found or corrupted.' +

' You can use download=True to download it')

self.train_data = []

This comment was marked as off-topic.

Sign in to view

fmassa reviewed Mar 10, 2017

View reviewed changes

torchvision/datasets/svhn.py Outdated

self.dataset = dataset # training set or test set or extra set

# download and load the data

if self.dataset=='train':

This comment was marked as off-topic.

Sign in to view

fmassa reviewed Mar 10, 2017

View reviewed changes

torchvision/datasets/svhn.py Outdated

def __len__(self):

if self.dataset == 'train':

return 73257

This comment was marked as off-topic.

Sign in to view

fmassa reviewed Mar 10, 2017

View reviewed changes

torchvision/datasets/svhn.py Outdated

self.root = root

self.transform = transform

self.target_transform = target_transform

self.dataset = dataset # training set or test set or extra set

This comment was marked as off-topic.

Sign in to view

uridah added 2 commits March 14, 2017 03:04

Updating svhn.py based on the comments in last PR

060a170

- now using dictionary for urls, filenames and md5s - updated len function - renamed 'dataset' keyword to split - fixed whitespaces using flake8

Add files via upload

a04a59a