Use templates as parameters #588
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rather than requiring shape/type we can just use templates. The benefit is that this work supports using any Nx.Container as a model parameter. For right now we still support the old style under the
paramfunction, but this defers to the newparameterwhich requires templates.There are certain places where using a container/composite parameter in place of a regular parameter makes sense. For example, if we want to initialize quantized models then we can create a quantized tensor container which represents a quantized parameter that can be converted back to a regular parameter.
In the future, I plan to unify the currently separated shape calculation and initialization process for a parameter. Realistically each parameter should just take input templates, do the shape calculation, and then initialize directly without any intermediate steps