[BUG] Error in the AzureML tests with error client is not permitted to update status for run. Status for this run is system managed #1876
Description
Description
In all AzureML tests, we get a similar error (see https://github.com/microsoft/recommenders/actions/runs/4049332367/jobs/6965640900)
Streaming user_logs/std_log.txt
===============================
INFO:submit_groupwise_azureml_pytest.py:Tests to be executed
INFO:submit_groupwise_azureml_pytest.py:['tests/smoke/examples/test_notebooks_python.py::test_baseline_deep_dive_smoke', 'tests/integration/examples/test_notebooks_python.py::test_baseline_deep_dive_integration', 'tests/smoke/examples/test_notebooks_python.py::test_surprise_svd_smoke', 'tests/integration/examples/test_notebooks_python.py::test_surprise_svd_integration', 'tests/integration/examples/test_notebooks_python.py::test_geoimc_integration', 'tests/integration/examples/test_notebooks_python.py::test_benchmark_movielens_cpu']
INFO:azureml._restclient.clientbase:Created a worker pool for first use
INFO:submit_groupwise_azureml_pytest.py:Python version
INFO:submit_groupwise_azureml_pytest.py:3.7.13 (default, Mar 29 2022, 02:18:16)
[GCC 7.5.0]
INFO:submit_groupwise_azureml_pytest.py:Executing tests now...
============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0
rootdir: /mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd, configfile: pyproject.toml
plugins: anyio-3.6.2, cov-4.0.0, hypothesis-6.60.0, rerunfailures-10.3, mock-3.10.0
collected 6 items
tests/smoke/examples/test_notebooks_python.py . [ 16%]
tests/integration/examples/test_notebooks_python.py
tests/smoke/examples/test_notebooks_python.py
tests/integration/examples/test_notebooks_python.py
=============================== warnings summary ===============================
../../../../../../../azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/ansiwrap/core.py:6
/azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/ansiwrap/core.py:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
tests/smoke/examples/test_notebooks_python.py::test_baseline_deep_dive_smoke
tests/integration/examples/test_notebooks_python.py::test_baseline_deep_dive_integration[1m-expected_values0]
/azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/02_model_collaborative_filtering/baseline_deep_dive.ipynb').
object.__init__() takes exactly one argument (the instance to initialize)
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
super().__init__(**kwargs)
tests/smoke/examples/test_notebooks_python.py::test_surprise_svd_smoke
tests/integration/examples/test_notebooks_python.py::test_surprise_svd_integration[1m-expected_values0]
/azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb').
object.__init__() takes exactly one argument (the instance to initialize)
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
super().__init__(**kwargs)
tests/integration/examples/test_notebooks_python.py::test_geoimc_integration[expected_values0]
/azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/00_quick_start/geoimc_movielens.ipynb').
object.__init__() takes exactly one argument (the instance to initialize)
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
super().__init__(**kwargs)
tests/integration/examples/test_notebooks_python.py::test_benchmark_movielens_cpu[size0-algos0-expected_values_ndcg0]
/azureml-envs/azureml_362c648550ddf3444cbaf086714b5547/lib/python3.7/site-packages/traitlets/config/configurable.py:85: DeprecationWarning: Passing unrecognized arguments to super(PapermillNotebookClient).__init__(input_path='/mnt/azureml/cr/j/82c5a50f0b2d4c9881f6deb1628b75cc/exe/wd/examples/06_benchmarks/movielens.ipynb').
object.__init__() takes exactly one argument (the instance to initialize)
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.
super().__init__(**kwargs)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================== 6 passed, 7 warnings in 1552.75s (0:25:52) ==================
INFO:submit_groupwise_azureml_pytest.py:Test execution completed!
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.5428390502929688 seconds
Execution Summary
=================
RunId: nightly_tests_group_cpu_002_1675123876_8a2b565c
Web View: https://ml.azure.com/runs/nightly_tests_group_cpu_002_1675123876_8a2b565c?wsid=/subscriptions/***/resourcegroups/recommenders_project_resources/workspaces/azureml-test-workspace&tid=***
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 589, in _execute_with_arguments
return self._call_api(func, *args_list, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 245, in _call_api
return self._execute_with_base_arguments(func, *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 333, in _execute_with_base_arguments
return ClientBase._execute_func_internal(
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 367, in _execute_func_internal
left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 426, in _handle_retry
raise error
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 358, in _execute_func_internal
response = func(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/operations/events_operations.py", line 95, in post
raise models.ErrorResponseException(self._deserialize, response)
azureml._restclient.models.error_response.ErrorResponseException: (ValidationError) Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1 azureml._restclient/1.5.0 azureml-sdk-core/1.48.0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 499, in <module>
run.complete()
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/core/run.py", line [1411](https://github.com/microsoft/recommenders/actions/runs/4049332367/jobs/6965640900#step:3:1420), in complete
self._client.complete(_set_status=_set_status)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py", line 621, in complete
self.set_completed_status(_set_status=_set_status)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py", line 625, in set_completed_status
self.run.post_event_completed(caller=self.identity)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 384, in post_event_completed
return self.post_event(event, is_async=is_async, caller=caller, custom_headers=custom_headers)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 316, in post_event
return self._execute_with_run_arguments(self._client.events.post, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/run_client.py", line 564, in _execute_with_run_arguments
return self._execute_with_arguments(func, copy.deepcopy(self._run_arguments), *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 591, in _execute_with_arguments
raise ServiceException(e)
azureml._restclient.exceptions.ServiceException: ServiceException:
Code: 400
Message: (ValidationError) Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1 azureml._restclient/1.5.0 azureml-sdk-core/1.48.0
Details:
Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1 azureml._restclient/1.5.0 azureml-sdk-core/1.48.0
Headers: ***
"Date": "Tue, 31 Jan 2023 00:41:13 GMT",
"Content-Type": "application/json; charset=utf-8",
"Content-Length": "1330",
"Connection": "keep-alive",
"Request-Context": "appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d",
"x-ms-response-type": "error",
"Strict-Transport-Security": "max-age=15724800; includeSubDomains; preload",
"X-Content-Type-Options": "nosniff",
"x-aml-cluster": "vienna-eastus-02",
"x-request-time": "0.052"
***
InnerException: ***
"additional_properties": ***,
"error": ***
"additional_properties": ***
"debugInfo": null
***,
"code": "ValidationError",
"severity": null,
"message": "Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1 azureml._restclient/1.5.0 azureml-sdk-core/1.48.0",
"message_format": null,
"message_parameters": null,
"reference_code": null,
"details_uri": null,
"target": "Status",
"details": [
***
"additional_properties": ***
"debugInfo": null
***,
"code": "Immutable",
"severity": null,
"message": "Run nightly_tests_group_cpu_002_1675123876_8a2b565c status cannot be updated by python/3.8.16 (Linux-5.15.0-1031-azure-x86_64-with-glibc2.34) msrest/0.7.1 azureml._restclient/1.5.0 azureml-sdk-core/1.48.0",
"message_format": null,
"message_parameters": ***,
"reference_code": null,
"details_uri": null,
"target": "Status",
"details": [],
"inner_error": null,
"additional_info": null
***
],
"inner_error": null,
"additional_info": null
***,
"correlation": ***
"operation": "f7b112a337eb8db68a8f472148b4d221",
"request": "5600226ee0f6b4f9"
***,
"environment": "eastus",
"location": "eastus",
"time": ***,
"component_name": "run-history"
***
Error: Process completed with exit code 1.
In which platform does it happen?
AzureML tests