Skip to content

README.md

Latest commit

 

History

History
380 lines (263 loc) · 11.4 KB

README.md

File metadata and controls

380 lines (263 loc) · 11.4 KB

To Install

In your project, run this npm command:

npm install @villedemontreal/concurrent-api-tests

Concurrent API Test Functions

aFewSeconds

Some test cases must rely on the timing between API requests. These test cases are likely to be flaky if the timing is not managed with care.

If the precision of the timing has to be less than a second, then concurrent-api-tests is not the right tool for this test case.

Arguments

  • delayInSeconds: The number of seconds to wait for. (Ex: 5).

Returns

void

Example

await aFewSeconds(5);

defineCopyTemplate

Define a function that provide a default payload template and that allow to specify only the parts of the payload that are meaningful for the test case. This way, test cases are easier to read since only the parts that matter are specified. Moreover, if a change in a payload is required, only the default payload template and the related tests need to be changed.

Arguments

  • template: A default payload template.

Returns

A function that provide a default payload template and allow to specify only the parts of the payload that are meaningful for the test case.

Example

import { defineCopyTemplate } from "@villedemontreal/concurrent-api-tests";

interface BlogPost {
  title: string;
  content: string;
  keywords: string[];
  category: string;
}

export const copyBlogPostTemplate = defineCopyTemplate<BlogPost>({
  title: "titleDefault",
  content: "contentDefault",
  keywords: [],
  category: "categoryDefault",
});

// In test — only override what matters
const request = copyBlogPostTemplate((x) => {
  x.title = "My Custom Title";
});

defineCopyTemplateVariation

Define a copy template variation to avoid duplication when the same template is used in many test cases.

Arguments

  • originalCopyTemplate: The original copy template function. See defineCopyTemplate.
  • variation: A function that specifies only the parts of the payload that are meaningful for the variation.

Returns

A function that provide a default payload template and allow to specify only the parts of the payload that are meaningful for the test case.

Example

import { defineCopyTemplateVariation } from "@villedemontreal/concurrent-api-tests";

// Create a variation for blog posts with a specific category
export const copyTechBlogPostTemplate = defineCopyTemplateVariation(
  copyBlogPostTemplate,
  (x) => {
    x.category = "tech";
  }
);

// In test — the variation already has category set
const request = copyTechBlogPostTemplate((x) => {
  x.title = "Tech Article";
});
// request.category is already "tech"

defineGetSharedFixture

Define a function that perform lazy initialization of a fixture. This allow to share the same fixture between many tests cases and to initialize it only once.

A fixture may be shared between the test cases of the same test run if

  1. the attribute of the user are not meaningful for the test
  2. the fixture is immutable

Although shared fixture can speed up test runs and reduce the amount of data created on the server, they must be used with care since they can produce flaky test cases if the two points above are not respected.

In doubt, create a new fixture for each test case.

Fast tests are important, but reliable tests are even more important.

Arguments

  • createSharedFixture: A function that initialize the shared fixture.

Returns

A function that perform lazy initialization of the shared fixture.

Example

import { defineGetSharedFixture } from "@villedemontreal/concurrent-api-tests";

interface JwtToken {
  token: string;
  expiresAt: Date;
}

// Shared fixture — caches the JWT token for all tests
export const getAdminJwtToken = defineGetSharedFixture<JwtToken>(
  () => fetchJwtToken("admin") // Called only once, then cached
);

// In test
it("Admin can create blog post", async () => {
  const token = await getAdminJwtToken();
  // token is reused across all tests that call getAdminJwtToken()
});

defineGetSharedFixtureByKey

Same as defineGetSharedFixture, but allow to pass a key as argument. Useful when there are many similar shared fixture to be defined.

Arguments

  • createSharedFixtureByKey: A function that initialize the shared fixture for a specific key.

Returns

A function that perform lazy initialization of the shared fixture for a specific key.

Example

import { defineGetSharedFixtureByKey } from "@villedemontreal/concurrent-api-tests";

type UserRole = "admin" | "editor" | "reader";

interface JwtToken {
  token: string;
  expiresAt: Date;
}

// Shared by key — caches one JWT token per role
export const getJwtTokenFor = defineGetSharedFixtureByKey<UserRole, JwtToken>(
  (role) => fetchJwtToken(role) // Called once per unique role
);

// In tests
it("Editor can create blog post", async () => {
  const token = await getJwtTokenFor("editor");
  // First call authenticates; subsequent calls reuse cached token
});

it("Admin can delete blog post", async () => {
  const token = await getJwtTokenFor("admin");
  // Different key, so authenticates separately from "editor"
});

getTestRunId

Get a unique identifier for the current test run. This identifier is used to partition data between concurrent test runs, ensuring test isolation.

The identifier is prefixed with zApiTest- followed by a UUID. The z prefix ensures that data created by API tests appears at the end of alphabetically sorted lists, making it easier to distinguish test data from manually created data.

Arguments

None.

Returns

A unique string identifier for the current test run (e.g., zApiTest-550e8400-e29b-41d4-a716-446655440000).

Example

import { getTestRunId } from "@villedemontreal/concurrent-api-tests";

it("Search blog posts by keyword", async () => {
  // Use getTestRunId() to create a unique keyword for this test run
  const keyword = `${getTestRunId()}-testing`;

  // Create posts with the partitioned keyword
  await postBlogPost({ title: "Post 1", keywords: [keyword] });
  await postBlogPost({ title: "Post 2", keywords: [keyword] });

  // Search only returns posts from this test run
  const results = await getBlogPosts(keyword);
  assert.strictEqual(results.length, 2);
});

shouldThrow

Assert against an API request that is expected to throw an error.

Arguments

  • act: A function that send the API request.
  • customAssert: A function that assert against the error.

Returns

void

Example

import { shouldThrow } from "@villedemontreal/concurrent-api-tests";
import { assert } from "chai";

it("Title is required", async () => {
  const request = copyBlogPostTemplate((x) => {
    x.title = null;
  });

  await shouldThrow(
    () => postBlogPost(request),
    (err) => {
      assert.strictEqual(err.status, 400);
      assert.include(err.message, "title");
    }
  );
});

FlakyTestReporter

A custom Vitest reporter that highlights flaky tests. A flaky test is one that fails on initial attempts but eventually passes after retries. The reporter extends Vitest's VerboseReporter to display detailed information about each flaky test, including the number of failures and the full error stack traces.

Configuration

Add the reporter to your vitest.config.ts and enable retries:

import { defineConfig } from "vitest/config";
import { FlakyTestReporter } from "@villedemontreal/concurrent-api-tests";

export default defineConfig({
  test: {
    reporters: [new FlakyTestReporter()],
    retry: 2, // Number of retry attempts for failed tests
  },
});

Example Output

When flaky tests are detected, the reporter displays a dedicated section showing each flaky test with its failure history:

 RUN  v4.0.3 /workspaces/concurrent-api-tests/flakyTestReporterDemo

 ✓ test/flakyTests.apiTestSuite.ts > Flaky > Test pass 1ms
 × test/flakyTests.apiTestSuite.ts > Flaky > Test with error 3ms (retry x2)
   → Pow!
   → Pow!
   → Pow!
 ✓ test/flakyTests.apiTestSuite.ts > Flaky > Flaky once 0ms (retry x1)
 ✓ test/flakyTests.apiTestSuite.ts > Flaky > Flaky twice 1ms (retry x2)
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
 Flaky Tests 2
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
 FLAKY  Flaky > Flaky once: failed 1 time and then passed.
Error: Pow!Flaky:1
 ❯ test/flakyTests.apiTestSuite.ts:16:21
     14|     iFlakyOnce++;
     15|     if (iFlakyOnce <= 1) {
     16|       const error = new Error("Pow!Flaky:" + iFlakyOnce) as any;
       |                     ^
     17|       error.additionnalAttribute = "The key to understand this bug.";
     18|       throw error;

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { additionnalAttribute: 'The key to understand this bug.' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
 FLAKY  Flaky > Flaky twice: failed 2 times and then passed.
Error: Pow!Flaky:1
 ❯ test/flakyTests.apiTestSuite.ts:25:21
     23|     iFlakyTwice++;
     24|     if (iFlakyTwice <= 2) {
     25|       const error = new Error("Pow!Flaky:" + iFlakyTwice) as any;
       |                     ^
     26|       error.additionnalAttribute =
     27|         "The key to understand this bug. Flaky:" + iFlakyTwice;

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { additionnalAttribute: 'The key to understand this bug. Flaky:1' }
Error: Pow!Flaky:2
 ❯ test/flakyTests.apiTestSuite.ts:25:21
     23|     iFlakyTwice++;
     24|     if (iFlakyTwice <= 2) {
     25|       const error = new Error("Pow!Flaky:" + iFlakyTwice) as any;
       |                     ^
     26|       error.additionnalAttribute =
     27|         "The key to understand this bug. Flaky:" + iFlakyTwice;

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { additionnalAttribute: 'The key to understand this bug. Flaky:2' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯

 FAIL  test/flakyTests.apiTestSuite.ts > Flaky > Test with error
 FAIL  test/flakyTests.apiTestSuite.ts > Flaky > Test with error
 FAIL  test/flakyTests.apiTestSuite.ts > Flaky > Test with error
Error: Pow!
 ❯ test/flakyTests.apiTestSuite.ts:8:19
      6|   });
      7|   it("Test with error", () => {
      8|     const error = new Error("Pow!") as any;
       |                   ^
      9|     error.additionnalAttribute = "The key to understand this bug.";
     10|     throw error;

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { additionnalAttribute: 'The key to understand this bug.' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/3]⎯


 Test Files  1 failed (1)
      Tests  1 failed | 3 passed (4)
   Start at  22:40:24
   Duration  204ms (transform 29ms, setup 0ms, collect 39ms, tests 6ms, environment 0ms, prepare 9ms)

Testing concurrent-api-tests itself

Run all unit tests, run this npm command:

npm start

Debug all unit tests, run this npm command:

npm run watch-no-emit (to activate incremental transpilation) and use a JavaScript Debug Terminal.

Lint, run this npm command:

npm run lint-fix