Feel free to provide suggestions for the next open-source super-powerful generic restoration model

[AIisCool]

Sorry I know this is off-topic, but I asked over at the official repo ByteDance-Seed/SeedVR#35 but had no response. Any idea if they're working on a newer model?

[adrientoupet]

Not that I'm aware of - and even if they did not sure if they would disclose it.

[AIisCool]

Fair enough, was just curious and thought perhaps you might've had some idea.

[IceClear]

Hi~ @AIisCool and @adrientoupet. Thank you for the enthusiasm of SeedVR series.
Also thank @adrientoupet and @numz for the continuous contributions to the applications of SeedVR.

I may begin my work on the next-generation restoration approach in the next few months, and I would like to hear from the community about the expectations of the next open-source super-powerful generic restoration model : )
Feel free to add comments below to provide any suggestions and feedback.
They are really valuable to me >_<

BTW, it is always not easy to open-source large vision models from the company, but I promise to try my best to keep contributing to the community and I pretty appreciate the love of this project from the community!

[adrientoupet]

Hi @IceClear - that's very exciting, you're going to make a lot of people really happy there, myself included.

Some suggestions from me:

• Work at very high resolution: do not limit training data to HD, but also train at 2k, 4k, 8k, 16k
• Include damaged data in the training data, to teach the model to restore heavily damaged inputs (e.g. interlaced and other artifacts that are not just low resolution)
• High color fidelity with the input source
• Full linear workflow support (32b linear input returns a 32b linear output with no compression loss)
• Arbitrary channel support (not limited to rgb, but could include rgba + arbitrary layers)
• Faster VAE
• Keep the DiT upscaling as a single step for speed

I'm going to continue to improve the current implementation of SeedVR2 to tackle some of those but anything you can do on your side for the next generation of model would be much appreciated. And please do not hesitate to reach out if I can help in any way. Thank you!

[IceClear]

Hi. @adrientoupet, really appreciate your feedback. They are very helpful.
May I ask for more details related to the following items?

• ''Full linear workflow support (32b linear input returns a 32b linear output with no compression loss)'' --> what does this linear workflow mean? Any difference from current one?

• ''Arbitrary channel support (not limited to rgb, but could include rgba + arbitrary layers)'' --> I understand the rgba but what do the arbitrary layers do here?

Thanks!

[adrientoupet]

Thank you sir!

• Linear workflow: Correct me if I'm wrong, but I'm pretty sure we're currently clamping input color range to 0-1 values? https://github.com/ByteDance-Seed/SeedVR/blob/6b061f4670599178df207d85a52a8212dd37c541/projects/inference_seedvr_7b.py#L240 If my source is an animated sequence of 32-bits exr, I don't want any clamping to happen, I want to preserve the full colorspace values.
• Arbitrary channel is a bit of a dream request, rgba would already be a good improvement. When working in the VFX industry you often have additional utility layers attached to your image sequence, in addition to your RGB values. You could have Alpha, Depth, Matt, Lighting pass... Being able to upscale of those at once in a temporally consistent way with the same upscale settings would be extremely helpful. Otherwise you can't use your utility layers anymore, because they won't perfectly lineup with your RGB layer once upscaled.

[AIisCool]

I agree 100% with the suggestions by @adrientoupet, tackling degradations would be great as that's one thing Topaz's Starlight model can do, but the results are nowhere near as sharp as SEEDVR2. Also perhaps if it is possible better temporal stability. The results can look great but then there is ghosting or distortions present.

Very excited to hear you're considering working on a newer model, as SEEDVR2 does do incredible things and a newer improved one can certainly yield more impressive results!