Add ability to derive variables and add selected derived forcings

[ealerskans]

Implements the ability to derive fields from the input datasets, as discussed in Deriving forcings #29.

At the moment, I have only added the possibility to derive the following forcings:

• top-of-atmosphere radiation
• hour of day (cyclically encoded)
• day of year (cyclically encoded)
• time of year (cyclically encoded)

But additional variables, such as boundary and land-sea masks, should be added. But I think that is for another PR.

[joeloskarsson]

Hi @ealerskans, just sneaking in a question about this. Me and @sadamov are eager to add on derived forcing inputs to some experiments. I see that there is still discussion ongoing above about the code structure and tests, but is all the actual functionality in place here? I.e. could we check out this branch and use it in the meantime to create the forcings we need?

[ealerskans]

I have tried to update the PR now based on all the great suggestions and feedback I have gotten. What I think is missing now are

• functionality of how to wrap a derived variable as an xr.DataArray if it is e.g. a np.ndarray
• add more tests

With respect to tests, I think that the following would be nice to test, but I am not sure how to do it:

• That _check_and_get_required_attributes in mllam_data_prep.ops.derive_variable.main works as it should (i.e. how we handle attributes)
• That _get_derived_variable_function in mllam_data_prep.ops.derive_variable.main works as it should (i.e. that we get the correct function, and that it also works for functions outside of mllam_data_prep)

@mafdmi do you perhaps have any good ideas on 1) if it would be good to test them, and 2) how to do it?

Is the functionality to wrap in xr.DataArray something that should be included in this PR?

Then there is also the task of adding other derived fields, such as creating a land-sea mask or a boundary mask (https://github.com/mllam/neural-lam/blob/feature_calculate_forcings/create_forcings.py). But I guess these can be added in another PR.

[ealerskans]

Hi @ealerskans, just sneaking in a question about this. Me and @sadamov are eager to add on derived forcing inputs to some experiments. I see that there is still discussion ongoing above about the code structure and tests, but is all the actual functionality in place here? I.e. could we check out this branch and use it in the meantime to create the forcings we need?

Hi @joeloskarsson and @sadamov If you are only interested in adding the datetime forcings (hour of day and day of year) and top-of-the-atmosphere radiation), then you should be able to use this branch as it is now. I believe that I at one point also tested training neural-lam on a small test dataset which included derived variables as well. However no guarantees ;)

[mfroelund]

I have tried to update the PR now based on all the great suggestions and feedback I have gotten. What I think is missing now are

* functionality of how to wrap a derived variable as an `xr.DataArray` if it is e.g. a `np.ndarray`

* add more tests

With respect to tests, I think that the following would be nice to test, but I am not sure how to do it:

* That `_check_and_get_required_attributes` in `mllam_data_prep.ops.derive_variable.main` works as it should (i.e. how we handle attributes)

* That `_get_derived_variable_function` in `mllam_data_prep.ops.derive_variable.main` works as it should (i.e. that we get the correct function, and that it also works for functions outside of `mllam_data_prep`)

@mafdmi do you perhaps have any good ideas on 1) if it would be good to test them, and 2) how to do it?

Is the functionality to wrap in xr.DataArray something that should be included in this PR?

Then there is also the task of adding other derived fields, such as creating a land-sea mask or a boundary mask (https://github.com/mllam/neural-lam/blob/feature_calculate_forcings/create_forcings.py). But I guess these can be added in another PR.

Yes, I think we can test them. For _check_and_get_required_attributes I would test, that it returns the correct attributes depending on the different cases, e.g. if attribute is in field.attrs or not, the attrs is correctly updated. For _get_derived_variable_function we can setup tests to check that it can import various functions from the standard library, and test what happens with non-existing module. Do you want me to write a proposal for the tests or do want to give it a try from this?

[ealerskans]

Yes, I think we can test them. For _check_and_get_required_attributes I would test, that it returns the correct attributes depending on the different cases, e.g. if attribute is in field.attrs or not, the attrs is correctly updated. For _get_derived_variable_function we can setup tests to check that it can import various functions from the standard library, and test what happens with non-existing module. Do you want me to write a proposal for the tests or do want to give it a try from this?

If you have time it would be really helpful if you could give it a go. Then I can try and learn from that :)

[mfroelund]

Yes, I think we can test them. For _check_and_get_required_attributes I would test, that it returns the correct attributes depending on the different cases, e.g. if attribute is in field.attrs or not, the attrs is correctly updated. For _get_derived_variable_function we can setup tests to check that it can import various functions from the standard library, and test what happens with non-existing module. Do you want me to write a proposal for the tests or do want to give it a try from this?

If you have time it would be really helpful if you could give it a go. Then I can try and learn from that :)

I've opened ealerskans#1 with tests added:)

[ealerskans]

@leifdenby I have implemented what we talked about in 0ecfcca.

[mfroelund]

Looks great!! Only minor suggestions/fixes

[sadamov]

Hi @joeloskarsson and @sadamov If you are only interested in adding the datetime forcings (hour of day and day of year) and top-of-the-atmosphere radiation), then you should be able to use this branch as it is now. I believe that I at one point also tested training neural-lam on a small test dataset which included derived variables as well. However no guarantees ;)

This is brilliant, worked like a charm for my Swiss dataset! Thank you 🙏

[leifdenby]

I just have one suggestion for the docstring for that lat/lon function. Otherwise this is ready to merge 🥳

oh and the change in the README needs changing back I think :)

[leifdenby]

I think this example in the README still needs changing back :)