Skip to content

Validation should allow specifying paths to input images without requiring validation dataset #1814

@bghira

Description

@bghira

A complete validation dataset is kind-of a bulky requirement for setting up simple I2V validations.

We can simply take in a list of dicts as --validation_input=[{"path": "...", "prompt": "..."}]

  • The prompts must be collected with other validation prompts and cached to the default text embed backend
    • The validation module must be able to retrieve these cached text embeds
  • If the current model requires image embeds (eg. wan 2.1 i2v or qwen-edit), we run into a unique issue where there is generally no "default conditioning embeds backend" for storage - there's no defined location to store these to or retrieve them from. This is where the "validation images require their own dataset" approach wins, because all of this complexity stays bundled to the same initialisation flow
  • Some models require VAE outputs to be fed as inputs, but this isn't as big of a deal (the VAE will be loaded during validation)

One alternative to duplicating this logic for validation init is to create a virtual dataset that contains the paths and captions the user provided. This is still a strange approach, but matches how we're handling conditioning data already for i2v.

Metadata

Metadata

Assignees

No one assigned

    Labels

    apiThis issue or enhancement impacts the API.difficult-featureenhancementNew feature or requestwebuiThis issue or enhancement impacts the web interface.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions