AzureML Tests

[pradnyeshjoshi]

Description

Main contributions of this PR:

• Added nightly and unit-test GitHub workflows that trigger execution of tests on AzureML compute cluster.
• Added a script to bring up an AzureML compute cluster instance and start an experiment run.
• Tests are divided into into groups, and the groups are executed in parallel to reduce execution time.

Improvements:

• AzureML compute is better suited to handle heavy machine learning workloads.
• Since an instance can be brought up on-demand and automatically, this resolves the need to have self managed hosts up and running 24x7.
• Distributing tests to multiple jobs and executing them parallelly brings down the execution time by ~50%, compared to the current setup. (Unit tests workflow takes ~19 min and integration tests take ~2 hrs for completion.)

Related Issues

Feature request: #995

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• This PR is being made to staging branch and not to main branch.

[codecov-commenter]

Codecov Report

Merging #1717 (1b711cf) into staging (3b38062) will increase coverage by 23.20%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           staging    #1717       +/-   ##
============================================
+ Coverage     0.00%   23.20%   +23.20%     
============================================
  Files           87       88        +1     
  Lines         9132     9142       +10     
============================================
+ Hits             0     2121     +2121     
- Misses           0     7021     +7021     

Flag	Coverage Δ
nightly	?
pr-gate	23.20% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
recommenders/datasets/mind.py	0.00% <0.00%> (ø)
recommenders/datasets/criteo.py	56.60% <0.00%> (+56.60%)	⬆️
recommenders/datasets/movielens.py	66.37% <0.00%> (+66.37%)	⬆️
recommenders/datasets/download_utils.py	90.00% <0.00%> (+90.00%)	⬆️
recommenders/models/newsrec/io/mind_iterator.py	0.00% <0.00%> (ø)
recommenders/models/newsrec/models/base_model.py	0.00% <0.00%> (ø)
recommenders/__init__.py	100.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b77bc33...1b711cf. Read the comment docs.

[miguelgfierro]

Adding some comments

One question, for these tests to be enabled, do you need them to be in main?

[miguelgfierro]

Looks great @pradnyeshjoshi,

Did you get the final time we have now?

[anargyri]

Great work @pradnyeshjoshi ! ❤

[pradnyeshjoshi]

Looks great @pradnyeshjoshi,

Did you get the final time we have now?

@miguelgfierro time taken for last runs: unit tests - 28 minutes, nightly - 42 minutes

[miguelgfierro]

For reference, the latest release took 9h 49m (see run), with this system, all the tests will take 42min.

OH YEAH!!!

big congrats @pradnyeshjoshi!