Use managed identity with OpenID Connect for Azure login

[SimonYansenZhao]

Description

This PR resolves

• the issue [FEATURE] Change the current service principal to managed identity for the AzureML tests #2171 by updating the testing workflows to use the Azure managed identity with OpenID Connect for login.
• and the issue that breaks the testing workflow, which is caused by the Python packages spacy and thinc increase the version number of numpy in their latest version. space and thinc is depended by fastai.

• Additional federated identity credential may be required. See Configure a federated identity credential on a user-assigned managed identity

Related Issues

• [FEATURE] Change the current service principal to managed identity for the AzureML tests #2171
• Use VM managed identity for login #2175

References

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• I have signed the commits, e.g. git commit -s -m "your commit message".
• This PR is being made to staging branch AND NOT TO main branch.

[miguelgfierro]

@SimonYansenZhao I added the federated credential, let me know if it works

[miguelgfierro]

@SimonYansenZhao it seems it is working now. I had to go to the Azure portal and add a federated credential with the name of your branch. I tried with the pull request option, and didn't work, and tried to add * as a wildcard branch, but it doesn't work either. It seems there is a way to programmatically create federeated credentials using a request:

POST https://graph.microsoft.com/v1.0/applications/{application-id}/federatedIdentityCredentials
Content-Type: application/json
Authorization: Bearer {access-token}

Replace {application-id} with your Azure AD application's object ID and {access-token} with a valid Microsoft Graph access token with the required permissions.

Request Body Example
json
Copy code
{
"name": "GitHubActionsFederatedCredential",
"issuer": "https://token.actions.githubusercontent.com",
"subject": "repo:recommenders-team/recommenders:ref:refs/heads/NAME_OF_THE_CURRENT_BRANCH",
"description": "Federated credential for GitHub Actions",
"audiences": [
"api://AzureADTokenExchange"
]
}
Explanation of the Request Body Fields
name: A unique name for the federated identity credential within your application.
issuer: Set to https://token.actions.githubusercontent.com for GitHub Actions.
subject: The subject claim format, e.g., repo:owner/repo:ref:refs/heads/branch.
description: (Optional) A description of the credential.
audiences: Specifies who can use this credential; for GitHub Actions, use "api://AzureADTokenExchange".

One unkonwn still is how to enable this solution to execute the nightly builds in staging and main.

[miguelgfierro]

There is a new error, but not of authentication:

Pytest logs
  /opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/azureml/mlflow/_protos/aml_service_pb2.py:10: UserWarning: google.protobuf.service module is deprecated. RPC implementations should provide code generator plugins which generate code specific to the RPC implementation. service.py will be removed in Jan 2025
    from google.protobuf import service as _service
  Downloading artifacts:   0%|          | 0/1 [00:00<?, ?it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Traceback (most recent call last):
    File "/home/runner/work/recommenders/recommenders/tests/ci/azureml_tests/post_pytest.py", line 93, in <module>
      with open(pathlib.Path(args.log_dir) / log_path, "r") as file:
  FileNotFoundError: [Errno 2] No such file or directory: 'test_logs/user_logs/std_log.txt'

[SimonYansenZhao]

There is a new error, but not of authentication:

Pytest logs
  /opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/azureml/mlflow/_protos/aml_service_pb2.py:10: UserWarning: google.protobuf.service module is deprecated. RPC implementations should provide code generator plugins which generate code specific to the RPC implementation. service.py will be removed in Jan 2025
    from google.protobuf import service as _service
  Downloading artifacts:   0%|          | 0/1 [00:00<?, ?it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00,  1.33it/s]
  Traceback (most recent call last):
    File "/home/runner/work/recommenders/recommenders/tests/ci/azureml_tests/post_pytest.py", line 93, in <module>
      with open(pathlib.Path(args.log_dir) / log_path, "r") as file:
  FileNotFoundError: [Errno 2] No such file or directory: 'test_logs/user_logs/std_log.txt'

Not sure what's going on. I downloaded the artifact generated by the test, and did find std_log.txt in it. 🤣

[SimonYansenZhao]

@SimonYansenZhao it seems it is working now. I had to go to the Azure portal and add a federated credential with the name of your branch. I tried with the pull request option, and didn't work, and tried to add * as a wildcard branch, but it doesn't work either. It seems there is a way to programmatically create federeated credentials using a request:

POST https://graph.microsoft.com/v1.0/applications/{application-id}/federatedIdentityCredentials
Content-Type: application/json
Authorization: Bearer {access-token}

Replace {application-id} with your Azure AD application's object ID and {access-token} with a valid Microsoft Graph access token with the required permissions.

Request Body Example json Copy code { "name": "GitHubActionsFederatedCredential", "issuer": "https://token.actions.githubusercontent.com", "subject": "repo:recommenders-team/recommenders:ref:refs/heads/NAME_OF_THE_CURRENT_BRANCH", "description": "Federated credential for GitHub Actions", "audiences": [ "api://AzureADTokenExchange" ] } Explanation of the Request Body Fields name: A unique name for the federated identity credential within your application. issuer: Set to https://token.actions.githubusercontent.com for GitHub Actions. subject: The subject claim format, e.g., repo:owner/repo:ref:refs/heads/branch. description: (Optional) A description of the credential. audiences: Specifies who can use this credential; for GitHub Actions, use "api://AzureADTokenExchange".

One unkonwn still is how to enable this solution to execute the nightly builds in staging and main.

Ah, I see. This PR does not trigger the workflow, so manual triggering does not match the subject repo:recommenders-team/recommenders:pull_request.