Runtime API

[delatrie]

Context

This PR introduces the runtime API that targets authors of tests. Prior to this, they had to use the low-level API of AllureLifecycle (the API designed primarily for authors of integrations).

Motivation

1. Make report enhancements at runtime more convenient and straightforward for test authors.

Currently, they have to use a lot of boilerplate code to enhance the report. For example, adding a set of BDD labels to the test at runtime requires the following code:

AllureLifecycle.Instance.UpdateTestCase(
    t => t.labels.Add(
        Label.Feature("My Feature")
    )
);

The test author's runtime API allows it to be done with this instead:

AllureApi.AddFeature("My Feature");

Only the function call and the name of the feature must be expressed now.

2. Provide an intermediate API layer that can be used by higher level APIs.

Example of such high-level API could be the attribute-based (i.e., compile time) API. Currently, we have two different implementations of such API for NUnit and xUnit.net.

API changes

The Runtime API is split into two sets of functions. The most commonly used are defined in AllureApi. The low-level functions and rarely used ones are defined in ExtendedApi.

All examples have been updated to use those functions.

The Allure.Net.Commons.AllureApi class

This is a facade that provides a set of commonly used functions designed for test authors. All functions are implemented as static methods. Some functions are brand new. Attachment-related functions were moved from AllureLifecycle. Step-related functions were moved from CoreStepsHelper.

Note

The class is named AllureApi instead of just Allure as in allure-java because otherwise, it would conflict with the Allure namespace.

Here is the complete list of the runtime API functions in AllureApi:

• Metadata API
  • SetTestName(string newName)
  • SetFixtureName(string newName)
  • SetStepName(string newName)
  • SetDescription(string description)
  • SetDescriptionHtml(string descriptionHtml)
  • Labels API
    • AddLabels(params Label[] labels)
    • AddLabel(string name, string value)
    • AddLabel(Label newLabel)
    • SetSeverity(SeverityLevel severity)
    • SetOwner(string owner)
    • SetAllureId(int allureId)
    • AddTags(params string[] tags)
  • Links API
    • AddLink(string url)
    • AddLink(string name, string url)
    • AddIssue(string url)
    • AddIssue(string name, string url)
    • AddTmsItem(string url)
    • AddTmsItem(string name, string url)
    • AddLink(string name, string type, string url)
    • AddLinks(params Link[] links)
• Hierarchies API
  • Suites hierarchy API
    • AddParentSuite(string additionalParentSuite)
    • AddSuite(string additionalSuite)
    • AddSubSuite(string additionalSubSuite)
  • BDD hierarchy API
    • AddEpic(string additionalEpic)
    • AddFeature(string additionalFeature)
    • AddStory(string additionalStory)
• Steps API
  • Step(string name)
  • Lambda steps API
    • Step(string name, Action action)
    • Step<T>(string name, Func<T> function): T
    • async Step(string name, Func<Task> action): Task
    • async Step<T>(string name, Func<Task<T>> function): Task<T>
  • Attachments API
    • AddAttachment(string name, string type, string path)
    • AddAttachment(string name, string type, byte[] content, string fileExtension = "")
    • AddAttachment(string path, string? name = null)
    • AddScreenDiff(string expectedPng, string actualPng, string diffPng)
  • Parameters API
    • AddTestParameter(string name, object? value)
    • AddTestParameter(string name, object? value, ParameterMode mode)
    • AddTestParameter(string name, object? value, bool excluded)
    • AddTestParameter(string name, object? value, ParameterMode mode, bool excluded)
    • AddTestParameter(Parameter parameter)

The Allure.Net.Commons.ExtendedApi class

Here is the complete list of the runtime API functions in ExtendedApi:

• Explicit fixtures
  • StartBeforeFixture(string name)
  • StartAfterFixture(string name)
  • PassFixture()
  • PassFixture(Action<FixtureResult> updateResults)
  • FailFixture()
  • FailFixture(Action<FixtureResult> updateResults)
  • BreakFixture()
  • BreakFixture(Action<FixtureResult> updateResults)
• Explicit steps
  • StartStep(string name)
  • StartStep(string name, Action<StepResult> updateResults)
  • PassStep()
  • PassStep(Action<StepResult> updateResults)
  • FailStep()
  • FailStep(Action<StepResult> updateResults)
  • BreakStep()
  • BreakStep(Action<StepResult> updateResults)
• Lambda fixtures
  • Before(string name, Action action)
  • Before<T>(string name, Func<T> function): T
  • async Before(string name, Func<Task> action): Task
  • async Before<T>(string name, Func<Task<T>> function): Task<T>
  • After(string name, Action action)
  • After<T>(string name, Func<T> function): T
  • async After(string name, Func<Task> action): Task
  • async After<T>(string name, Func<Task<T>> function): Task<T>

Deprecations

All public members inside Allure.Net.Commons.Steps.CoreStepsHelper are obsolete now. Additionally, all attachment-related functions inside Allure.Net.Commons.AllureLifecycle and the Allure.Net.Commons.TestResult.AddParameter method are all deprecated.

The following functions were deprecated by this PR in favor of the new test author's runtime API:

• Allure.Net.Commons.Steps.CoreStepsHelper.StartBeforeFixture(string name) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.StartAfterFixture(string name) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.PassFixture() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.PassFixture(Action<Allure.Net.Commons.FixtureResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.FailFixture() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.FailFixture(Action<Allure.Net.Commons.FixtureResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.BrokeFixture() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.BrokeFixture(Action<Allure.Net.Commons.FixtureResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.StartStep(string name) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.StartStep(string name, Action<Allure.Net.Commons.StepResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.PassStep() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.PassStep(Action<Allure.Net.Commons.StepResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.FailStep() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.FailStep(Action<Allure.Net.Commons.StepResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.BrokeStep() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.BrokeStep(Action<Allure.Net.Commons.StepResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.Step(string name) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.Step(string name, Action action) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.Step<T>(string name, Func<T> action) : T
• Allure.Net.Commons.Steps.CoreStepsHelper.Step(string name, Func<Task> action) : Task
• Allure.Net.Commons.Steps.CoreStepsHelper.Step<T>(string name, Func<Task<T>> action) : Task<T>
• Allure.Net.Commons.Steps.CoreStepsHelper.Before(string name, Action action) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.Before<T>(string name, Func<T> action) : T
• Allure.Net.Commons.Steps.CoreStepsHelper.Before(string name, Func<Task> action) : Task
• Allure.Net.Commons.Steps.CoreStepsHelper.Before<T>(string name, Func<Task<T>> action) : Task<T>
• Allure.Net.Commons.Steps.CoreStepsHelper.After(string name, Action action) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.After<T>(string name, Func<T> action) : T
• Allure.Net.Commons.Steps.CoreStepsHelper.After(string name, Func<Task> action) : Task
• Allure.Net.Commons.Steps.CoreStepsHelper.After<T>(string name, Func<Task<T>> action) : Task<T>
• Allure.Net.Commons.AllureLifecycle.AddAttachment(string name, string type, string path) : void
• Allure.Net.Commons.AllureLifecycle.AddAttachment(string name, string type, byte[] content, string fileExtension = "") : void
• Allure.Net.Commons.AllureLifecycle.AddAttachment(string path, string? name = null) : void
• Allure.Net.Commons.AllureLifecycle.AddScreenDiff(string expectedPng, string actualPng, string diffPng) : void
• Allure.Net.Commons.TestResult.AddParameter(string name, object value, IReadOnlyDictionary<Type, Allure.Net.Commons.ITypeFormatter> formatters) : void

We advise users to move to their Allure.Net.Commons.AllureApi's counterparts. The functions mostly remained the name and the parameters. The only exceptions are the BrokeX methods what were renamed to BreakX in AllureApi..

The following functions were deprecated by this PR as duplicates of the existing AllureLifecycle's methods:

• Allure.Net.Commons.Steps.CoreStepsHelper.StopFixture() : void
• Allure.Net.Commons.Steps.CoreStepsHelper.StopFixture(Action<Allure.Net.Commons.FixtureResult> updateResults) : void
• Allure.Net.Commons.Steps.CoreStepsHelper.UpdateTestResult(Action<Allure.Net.Commons.TestResult> update) : void

We advise test authors to consider using the new API instead and only if that's not possible - the corresponding methods of the current lifecycle object (referenced by the AllureLifecycle.Instance property).

The following property was deprecated:

• Allure.Net.Commons.Steps.CoreStepsHelper.StepLogger { get; set; } : IStepLogger?

It will be replaced with event listeners in the future.

Other changes

The PR contains the following additional changes:

Parameter's mode and exclude options (#425)

Parameter.mode: ParameterMode and Parameter.exclude: bool were added to the object model. IdFunctions.CreateHistoryId was adjusted to not take into account excluded parameters.

Xml documentation for Allure.Net.Commons (#426)

The Allure.Net.Commons package now includes an XML doc file that is used by IDE's to provide users with API discovery and code completion. Structured comments were added to most public functions of the package.

XML doc files for other packages will be added later.

historyId calculation fix

Now historyId is calculated lately, in StopTestCase based on fullName and parameters of the test. This allows parameters added at runtime to be taken into account. Additionally, that fixes missing historyId for skipped xUnit theories (see #422).

Fix encoding for JSON schemas (see #415)

To comply with RFC 8259, byte order marks were removed from the JSON schemas:

• Allure.Net.Commons/Schemas/allureConfig.schema.json
• Allure.Net.Commons/Schemas/testplan.schema.json
• Allure.NUnit/Schemas/allureConfig.schema.json
• Allure.SpecFlowPlugin/Schemas/allureConfig.schema.json
• Allure.XUnit/Schemas/allureConfig.schema.json

Fix link factory methods (#416)

Allure.Net.Commons.Link.Issue(string name): Allure.Net.Commons.Link and Allure.Net.Commons.Link.Tms(string name): Allure.Net.Commons.Link were fixed to fill the url field. The methods essentially use the provided URL as the name of the link, so in both cases, the parameter was renamed from name to url.

Fix flaky InvalidOperationException when running SpecFlow tests (#433)

This error occurs because of some rudimentary code that was left from an earlier attempt to implement a selective run in Allure SpecFlow. The code isn't needed now and was removed in this PR.

Checklist

• Sign Allure CLA
• Provide unit tests

Closes #404
Closes #415
Closes #416
Closes #422
Closes #425
Partially implements #426
Closes #433