ToTensor aliases input data and operations like Normalize work in-place

[jatentaki]

ToTensor aliases input data and operations like Normalize work in-place, which leads to a nasty footgun when implementing one's own data loaders with the data residing in RAM - as training progresses, the dataset becomes more and more "normalized" which is difficult to debug. Also, the fact that the current implementation of datasets.MNIST doesn't display this behavior seems to depend on an (arguably buggy) implementation detail of PIL.Image, as I argue in my answer to this stackoverflow question.

I propose that the in-place operation is at least made explicit in docs on Normalize, or (a breaking change) make it work out of place by default and add a inplace flag.

[soumith]

we should make Normalize out of place by default

[fmassa]

After TensorIterator was merged in PyTorch, an approach like #519 now is as efficient as the multiple for loops

[vishwakftw]

This can be closed now, since #699 has been merged.