Skip to content

Resuming training seems unstable #419

@roedoejet

Description

@roedoejet

when I resume training, there's an initial spike in the loss that I don't expect:

image

Hypothesis

  • Random seeds not been properly initialized. Note that there are several seeds that need to be initialized
  • The eval is calulated over a sample of validation

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghelp wantedExtra attention is needed

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions