Okay so it took me another few hours to figure that one out. I know that in a previous issue #38 we talked about exporting your images from Lightroom in ProPhotoRGB. This comes from the fact that the on the website of the Adobe 5K dataset (https://data.csail.mit.edu/graphics/fivek/) they say that they provide the images in ProPhotoRGB format.
BUT...when I use pro photo RGB my images look like this:

And the original label looks like this:

So after training the supervised model I get a very nice PSNR, but unfortunately the images look very strange...So even though Adobe says they use ProPhotoRGB images, in my case this does not work. I don't know if this is general or if this is specific to my case(as another user from #38 says he uses ProPhotoRGB). After googling around one also finds that sRGB is a format which makes the image independent of the screen...so that doesn't sound too bad an intuition why one should use it. Maybe try both.