NonCompliant: DataAccessError in an isolated environment

[chengyu-liu-cs]

Describe the bug

I am having error when I ran the MNIST example pipeline in the isolated environment, in particular in the aggregation step. The same pipeline works in an open environment (without private endpoints). The isolated environment was created via 'One click ARM deployment' from github link. The environment creation was successful without any errors.

To Reproduce
Steps to reproduce the behavior:

1. Clone the repo
2. cd inside the repo
3. Create a virtual environment python3 -m venv ~/.virtualenv-default
4. source ~/.virtualenv-default/bin/activate
5. pip3 install -r ./examples/pipelines/fl_cross_silo_literal/requirements.txtments.txt
6. python ./examples/pipelines/fl_cross_silo_literal/submit.py --subscription_id <subscription_id> --resource_group <resource_group> --workspace_name <workspace_name> --example MNIST --submit
8. See error

{"NonCompliant":"DataAccessError(ConnectionFailure { source: Some(Custom { kind: TimedOut, error: "Request timeout" }) })"}
{
"code": "data-capability.UriMountSession.PyFuseError",
"target": "",
"category": "UserError",
"error_details": [
{
"key": "NonCompliantReason",
"value": "DataAccessError(ConnectionFailure { source: Some(Custom { kind: TimedOut, error: "Request timeout" }) })"
},
{
"key": "StackTrace",
"value": " File "/opt/miniconda/envs/data-capability/lib/python3.7/site-packages/data_capability/capability_session.py", line 70, in start\n (data_path, sub_data_path) = session.start()\n\n File "/opt/miniconda/envs/data-capability/lib/python3.7/site-packages/data_capability/data_sessions.py", line 386, in start\n options=mnt_options\n\n File "/opt/miniconda/envs/data-capability/lib/python3.7/site-packages/azureml/dataprep/fuse/dprepfuse.py", line 696, in rslex_uri_volume_mount\n raise e\n\n File "/opt/miniconda/envs/data-capability/lib/python3.7/site-packages/azureml/dataprep/fuse/dprepfuse.py", line 690, in rslex_uri_volume_mount\n mount_context = RslexDirectURIMountContext(mount_point, uri, options)\n"
}
]
}

Expected behavior
The pipeline should succeed without error

Screenshots

Additional context
Is this related to this issue?

[jfomhover]

Thank you for reporting this issue, we'll try to repro. Just to make sure, you're running from the main branch?

[chengyu-liu-cs]

Thank you for reporting this issue, we'll try to repro. Just to make sure, you're running from the main branch?

Yes. Main branch was cloned.

[garg-amit]

This is fixed and the changes will be soon pushed to the main branch.

[jfomhover]

Leaving some notes to explain the issue:

Our provisioning script had a wrong test condition. In the setup used in this issue, the private endpoint to the orchestrator storage from the orchestrator compute/vnet was supposed to be a static IP 10.0.0.50 (to allow for several endpoints to the same storage on different vnets in one unique private dns zone).

Because the test condition was wrong, this IP was allocated dynamically instead, causing a mismatch between the private DNS zone config (set correctly to 10.0.0.50) and the endpoint NIC ip address set dynamically to 10.0.0.*).