Multithread issues (NPE and deadlocks) with Allure.XUnit 2.9.5-preview.1 and AllureStep annotations

[886-59-936]

I'm submitting a ...

• bug report
• feature request
• support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

We are activelly using [AllureStep(".")] annotations and Steps.Step(".", () => {}); All of our tests and most of steps are async.
When we tried to update to Allure.XUnit 2.9.5-preview.1 we faced random intermittent errors: either deadlocks during tests execution or 25 - 30% of tests randomly marked as "Broken" due to exceptions (NPE, type cast exceptions) in steps. Please see examples below in "other information". Sorry for poor formatting - it is due to numerous quotation marks inside stack-trace

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

1. Create a test project (we have ~400). Tests should include Steps.Step(".", () => {}); and\or call methods marked with [AllureStep(".")] annotation
2. Tests and steps methods must be async. (If methods are syncronious or steps not used - no bug)
3. Update Allure.XUnit 2.9.5-preview.1, remove old library with Step extensions (Allure.Xunit.StepExtensions package)
4. Run all tests with "dotnet test some_project.csproj -c Release"
5. Examine generated allure results.

What is the expected behavior?

Normal report generated. No broken tests reported.

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

• Test framework: xunit 2.4.2
• Allure adaptor: Allure.XUnit 2.9.5-preview.1
• Additional packages used:
• xunit.runner.visualstudio Version="2.4.5"
• Divergic.Logging.Xunit Version="4.2.0"
• Xunit.SkippableFact Version="1.4.13"
• FluentAssertions Version="6.8.0"
• xunit.runner.json:
  {
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "diagnosticMessages": true,
  "longRunningTestSeconds": 120,
  "maxParallelThreads": 4,
  "parallelizeTestCollections": true
  }

Other information

The given key 'c72c4a28-aa57-486e-b12a-4c7d7a6e8ae1-order: somename + InternalStatus: Pending.' was not present in the dictionary.
at System.Collections.Concurrent.ConcurrentDictionary2.ThrowKeyNotFoundException(TKey key)
at Allure.Net.Commons.Storage.AllureStorage.AddStep(String parentUuid, String uuid, StepResult stepResult)
at Allure.Net.Commons.AllureLifecycle.StartStep(String parentUuid, String uuid, StepResult stepResult)
at Allure.Net.Commons.AllureLifecycle.StartStep(StepResult result, String& uuid)
at Allure.Xunit.Steps.StartStep(String name, Action1 updateResults) at Allure.Xunit.Steps.Step[T](String name, Func1 action)
at Allure.Xunit.Steps.Step(String name, Func1 action) at AutoTests.Tests..TestNameTest() in /builds/......./NameTest.cs:line 57 at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_0.<b__1>d.MoveNext() in //src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 273
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 54 at Xunit.Sdk.ExceptionAggregator.RunAsync(Func1 code) in //src/xunit.core/Sdk/ExceptionAggregator.cs:line 96`

Value cannot be null. (Parameter 'key')
at System.ThrowHelper.ThrowArgumentNullException(String name) at System.Collections.Concurrent.ConcurrentDictionary2.TryGetValue(TKey key, TValue& value)
at Allure.Net.Commons.Storage.AllureStorage.AddStep(String parentUuid, String uuid, StepResult stepResult)
at Allure.Net.Commons.AllureLifecycle.StartStep(String parentUuid, String uuid, StepResult stepResult)
at Allure.Net.Commons.AllureLifecycle.StartStep(StepResult result, String& uuid)
at Allure.Xunit.Steps.StartStep(String name, Action1 updateResults) at Allure.XUnit.AllureStepAspect.StartStep(MethodBase metadata, String stepName, List1 stepParameters)
at Allure.XUnit.AllureStepAspect.Around(String name, Object[] args, Func2 target, MethodBase metadata, Type returnType) at Framework.Steps.Http.HttpSteps.__a$_around_CheckStatusCode_100663407_w_0(Object[] ) at Framework.Steps.Http.HttpSteps.CheckStatusCode(HttpResponseMessage response, HttpStatusCode code) at AutoTests.Tests.....SomeOurMethodName(String input, OstrovokLanguage lang) in /builds/......../Test.cs:line 205 at AutoTests.Tests.....SomeOurTest.CityBristolTest() in /builds/............./Test.cs:line 86 at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_0.<b__1>d.MoveNext() in //src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48 at Xunit.Sdk.ExceptionAggregator.RunAsync(Func1 code) in //src/xunit.core/Sdk/ExceptionAggregator.cs:line 90`

Object reference not set to an instance of an object.
at Allure.Net.Commons.AllureLifecycle.<>c__DisplayClass39_0.<StartTestCase>b__0(TestResultContainer c) at Allure.Net.Commons.AllureLifecycle.UpdateTestContainer(String uuid, Action1 update)
at Allure.Net.Commons.AllureLifecycle.StartTestCase(String containerUuid, TestResult testResult)
at Allure.Xunit.Steps.StopFixtureSuppressTestCase(Action1 updateResults) at Allure.Xunit.AllureStepBase1.Dispose()
at AutoTests.Tests.Smoke.Test..ctor(ITestOutputHelper output) in /builds/............./Test.cs:line 40`

Nullable object must have a value.
at System.Nullable1.get_Value()
at AutoTests.Tests..........TestClass.Test() in /builds/......../Test.cs:line 200
at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 273 --- End of stack trace from previous location --- at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 54
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 96

Unable to cast object of type 'Allure.Net.Commons.TestResult' to type 'Allure.Net.Commons.StepResult'.
at Allure.Net.Commons.Storage.AllureStorage.Get[T](String uuid) at Allure.Net.Commons.AllureLifecycle.UpdateStep(Action1 update)
at Allure.Net.Commons.AllureLifecycle.StopStep(Action1 beforeStop) at Allure.Xunit.Steps.PassStep(Action1 updateResults)
at Allure.Xunit.Steps.Execute[T](Func1 action) at AutoTests.Tests...........TestClass.EmptyInputWrapper() in /builds/.........../AnotherTest.cs:line 44 at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_0.<b__1>d.MoveNext() in //src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48 at Xunit.Sdk.ExceptionAggregator.RunAsync(Func1 code) in //src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
`

[delatrie]

Hi, @886-59-936 !
Thanks for the report.

Yes, there are some issues with multithreading here. This will be addressed in the near future.

[delatrie]

Hello, @886-59-936 !
We've fixed a lot of concurrency-related issues in the recent Allure.XUnit 2.10.0 including multithreading issues. Give it a try!

If the issue persists, please, leave us a comment here.