Integrate nerc-rates for dynamic outage information loading

[jimmysway]

• Add load_outages_from_nerc_rates() to fetch outages from nerc-rates repository
• Update calculate_storage_gb_hours to auto-load outages for "stack" and "ocp-prod" services
• Maintain backwards compatibility with --excluded-time-ranges flag
• Add test coverage for nerc-rates integration

Closes #256

[jimmysway]

Based on previous discussions I think the service mapping was something like

OpenStack resources → stack
OpenShift resources → ocp-prod (or should it be both ocp-prod and ocp-test?)

Is there also other services that need to be included?

[knikolla]

Based on previous discussions I think the service mapping was something like

OpenStack resources → stack OpenShift resources → ocp-prod (or should it be both ocp-prod and ocp-test?)

Is there also other services that need to be included?

stack only applies to the OpenStack ColdFront Resource with name NERC
ocp-prod only applies to the OpenShift ColdFront Resource with name NERC-OCP
academic only applies to the OpenShift ColdFront Resource with name NERC-OCP-EDU

The rest aren't in ColdFront at the moment.

Please create a map as a global variable in the calculate_storage_gb_hours.py file to do the translation so you can make the appropriate call to nerc-rates.

[knikolla]

Created a related issue as a second step, after this PR is merged, to remove the PR and have the mapping be stored as attributes in the Resources themselves. For now, a dict map in calculate_storage_hours is fine.

#262

[knikolla]

Review in prior comments. Please implement a dictionary map to fetch the appropriate outage based on cluster name map.

[knikolla]

@jimmysway actually, seeing the asymmetry between providing CLI args (apply to all resources) and loaded from nerc-rates (apply to the specific resources as listed in the specific outage,) I'm okay with dropping the possibility to provide excluded intervals from the CLI.

[jimmysway]

@jimmysway actually, seeing the asymmetry between providing CLI args (apply to all resources) and loaded from nerc-rates (apply to the specific resources as listed in the specific outage,) I'm okay with dropping the possibility to provide excluded intervals from the CLI.

@knikolla Are there any downstream consequences for something like this that I should be aware of?

[knikolla]

@jimmysway actually, seeing the asymmetry between providing CLI args (apply to all resources) and loaded from nerc-rates (apply to the specific resources as listed in the specific outage,) I'm okay with dropping the possibility to provide excluded intervals from the CLI.

@knikolla Are there any downstream consequences for something like this that I should be aware of?

No, the command is either executed manually through a terminal, or through a cronjob that doesn't provide any excluded intervals when executing the command.

EDIT TO ADD: Furthermore the command is very specific to how we bill, so I'm not worried about potentially breaking someone who might be using it.

[knikolla]

I don't think the current testing is sufficient. You haven't added any new test while adding functionality.

Please add a test that mocks nerc-rates and verifies that get_outages_during is called with the appropriate parameters. You can patch the mapping dictionary to make things easier for resources that don't exist.

You can also test that the mocked output of get_outages_during is correctly used.

[knikolla]

One final comment, otherwise looks ready to go. Good work.

[knikolla]

@QuanMPhm were your comments resolved?

[QuanMPhm]

@knikolla I've resolved my comments and approved

[jimmysway]

@knikolla @QuanMPhm I've just rebased