Skip to content
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
9c2f2ab
fix(runner): apply fails before retrying
hi-ogawa Sep 9, 2026
326c386
test(runner): cover fails with retries and repeats
hi-ogawa Sep 9, 2026
2d62978
test(runner): reproduce lost failure across repeats
hi-ogawa Sep 9, 2026
1851ea8
test(runner): contrast ordinary and fails repeats
hi-ogawa Sep 9, 2026
7b05bbc
test(runner): record current repeat results
hi-ogawa Sep 9, 2026
5f9d131
test(runner): assert completed diagnostics exist
hi-ogawa Sep 9, 2026
7d9ec54
fix(runner): retain failures across repeats
hi-ogawa Sep 9, 2026
5716687
test(runner): cover repeat failure aggregation
hi-ogawa Sep 9, 2026
fdd91ab
test(runner): consolidate retry coverage
hi-ogawa Sep 10, 2026
829b6d3
test(runner): update fails lifecycle expectations
hi-ogawa Sep 10, 2026
47f5304
test(runner): clarify retry count across repeats
hi-ogawa Sep 10, 2026
eb84349
test(runner): show retry count across repeats
hi-ogawa Sep 10, 2026
4070bb6
docs(runner): clarify retry and repeat counts
hi-ogawa Sep 10, 2026
129a889
test(runner): assert retry result exists
hi-ogawa Sep 10, 2026
d63d16c
test(runner): reproduce lost repeat failures
hi-ogawa Sep 10, 2026
444d633
fix(runner): retain failures across repeats
hi-ogawa Sep 10, 2026
4d9658d
test(runner): cover repeat failure hooks
hi-ogawa Sep 10, 2026
15a8597
chore: stack repeat results fix
hi-ogawa Sep 10, 2026
203953e
docs(runner): clarify retry and repeat counts
hi-ogawa Sep 10, 2026
7e998d3
chore: update repeat results base
hi-ogawa Sep 10, 2026
dedca80
test(runner): show retry counts across repeats
hi-ogawa Sep 10, 2026
a756764
chore: update repeat results base
hi-ogawa Sep 10, 2026
1b5bc45
test(runner): alternate repeat outcomes
hi-ogawa Sep 10, 2026
11f3e68
chore: update repeat results base
hi-ogawa Sep 10, 2026
88cf0a9
test(runner): move repeat retries to e2e
hi-ogawa Sep 10, 2026
a9178d6
chore: update repeat results base
hi-ogawa Sep 10, 2026
8073168
docs(runner): explain state widening
hi-ogawa Sep 10, 2026
c39b768
chore: update repeat results base
hi-ogawa Sep 10, 2026
a32d934
docs(runner): clarify state widening
hi-ogawa Sep 10, 2026
2908873
chore: update repeat results base
hi-ogawa Sep 10, 2026
9eeb240
chore: merge latest main
hi-ogawa Sep 10, 2026
2579faf
chore: merge latest main
hi-ogawa Sep 10, 2026
7537f8d
test(runner): remove redundant repeat retry test
hi-ogawa Sep 10, 2026
5d96a2a
test(runner): remove duplicate repeat coverage
hi-ogawa Sep 10, 2026
3ed4c4a
test(runner): focus retry coverage on fails
hi-ogawa Sep 10, 2026
4d9a3ed
test(runner): assert expected failure run counts
hi-ogawa Sep 10, 2026
a4f5b30
test(runner): assert retry runs per repeat
hi-ogawa Sep 10, 2026
a8ac00e
test(runner): remove redundant repeat retention
hi-ogawa Sep 10, 2026
4723b9d
test(runner): replace retry diagnostics with runs
hi-ogawa Sep 10, 2026
3d83a66
test(runner): cover fails across repeats
hi-ogawa Sep 10, 2026
83e2109
test(runner): retain retry errors across repeats
hi-ogawa Sep 11, 2026
3c0f998
fix(runner): track failed repeats separately
hi-ogawa Sep 11, 2026
cf6fa5b
chore: update repeat results base
hi-ogawa Sep 11, 2026
efe3cff
test(runner): retain expected failure attempts
hi-ogawa Sep 11, 2026
8e226db
fix(runner): preserve expected failure attempts
hi-ogawa Sep 11, 2026
2793d0a
feat(runner): expose test attempts
hi-ogawa Sep 11, 2026
d84b37a
test(runner): cover reported attempts
hi-ogawa Sep 11, 2026
bf606d5
refactor(runner): use attempt indexes
hi-ogawa Sep 11, 2026
64c7fd8
docs(runner): stabilize attempt API
hi-ogawa Sep 11, 2026
dfd1172
chore: merge main into repeat-results
hi-ogawa Sep 28, 2026
9b73534
chore: merge repeat-results into fails-retry
hi-ogawa Sep 28, 2026
d482e6e
chore: merge fails-retry into attempts api
hi-ogawa Sep 28, 2026
becc514
Merge remote-tracking branch 'upstream/main' into attempts-api-prototype
hi-ogawa Oct 1, 2026
06f96ac
refactor(reporters): share attempt state type
hi-ogawa Oct 1, 2026
68368d0
feat(runner): grow attempts while the test runs
hi-ogawa Oct 1, 2026
62f00aa
docs: document test attempts API
hi-ogawa Oct 1, 2026
f372728
test(reporters): cover repeated test attempts
hi-ogawa Oct 1, 2026
86827cd
test(runner): make growing attempts test readable
hi-ogawa Oct 1, 2026
330cb80
test(runner): name attempt fields in growing attempts test
hi-ogawa Oct 1, 2026
ccad3c2
test(runner): name observation fields in growing attempts test
hi-ogawa Oct 1, 2026
d9837ba
test(runner): observe growing attempts from afterEach
hi-ogawa Oct 1, 2026
17a9892
test(runner): cover skipped attempts
hi-ogawa Oct 1, 2026
8651715
test(runner): note afterEach sees previous attempts only
hi-ogawa Oct 1, 2026
effbcfb
test(runner): use attempt field names as is
hi-ogawa Oct 1, 2026
2bd2784
test(runner): diagram attempts in growing attempts test
hi-ogawa Oct 1, 2026
318a66e
test(runner): use minimal retry in growing attempts test
hi-ogawa Oct 1, 2026
94fbd47
test(runner): diagram skipped attempts test
hi-ogawa Oct 1, 2026
eed88dc
Merge remote-tracking branch 'upstream/main' into attempts-api-prototype
hi-ogawa Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/api/advanced/runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,37 @@ export interface TaskResult {
* The zero-based index of the current repeat.
*/
repeatCount?: number
/**
* Individual results for every retry and repeat attempt.
*/
attempts?: TaskResultAttempt[]
}

export interface TaskResultAttempt {
/**
* The state after applying the test's expected failure option.
*/
state: 'pass' | 'fail' | 'skip'
/**
* Errors produced by this attempt.
*/
errors?: TestError[]
/**
* How long in milliseconds the attempt took to run.
*/
duration: number
/**
* Time in milliseconds when the attempt started running.
*/
startTime: number
/**
* The zero-based retry index within the repeat.
*/
retryIndex: number
/**
* The zero-based repeat index.
*/
repeatIndex: number
}
```

Expand Down
54 changes: 53 additions & 1 deletion docs/api/advanced/test-case.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ interface TestResultPassed {
```

::: warning
Note that the test with `passed` state can still have errors attached - this can happen if `retry` was triggered at least once.
Note that the test with `passed` state can still have errors attached - this can happen if `retry` was triggered at least once. Use [`attempts()`](#attempts) to see which run produced each error.
:::

## diagnostic
Expand Down Expand Up @@ -278,6 +278,58 @@ interface TestDiagnostic {
`diagnostic()` will return `undefined` if the test was not scheduled to run yet.
:::

## attempts <Version>5.1.0</Version> {#attempts}

```ts
function attempts(): ReadonlyArray<TestAttempt>
```

Results of every run of the test, in execution order. A test runs more than once when [`retry`](/config/retry) or [`repeats`](/api/test#repeats) is configured, and each run produces one entry. Unlike `result()`, which only reports the final outcome and merges errors from all runs, each attempt keeps its own state, errors and timing.

```ts
interface TestAttempt {
/**
* The state of this attempt. `fails` tests are already inverted,
* so an expected failure is reported as `passed`.
*/
readonly state: 'passed' | 'failed' | 'skipped'
/**
* Errors thrown during this attempt only.
*/
readonly errors: ReadonlyArray<TestError> | undefined
/**
* How long in milliseconds the attempt took to run.
*/
readonly duration: number
/**
* Time in milliseconds when the attempt started.
*/
readonly startTime: number
/**
* The zero-based retry index within the current repeat.
*/
readonly retryIndex: number
/**
* The zero-based repeat index.
*/
readonly repeatIndex: number
}
```

For example, a test with `retry: 2` that passes on its third run returns:

```ts
[
{ state: 'failed', retryIndex: 0, repeatIndex: 0, errors: [/* ... */] },
{ state: 'failed', retryIndex: 1, repeatIndex: 0, errors: [/* ... */] },
{ state: 'passed', retryIndex: 2, repeatIndex: 0, errors: undefined },
]
```

::: info
`attempts()` returns an empty array if the test has not run yet.
:::

## annotations

```ts
Expand Down
35 changes: 34 additions & 1 deletion packages/vitest/src/node/reporters/reported-tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,20 @@ export class TestCase extends ReportedTaskImplementation {
} satisfies TestResultFailed
}

/**
* Individual results for every retry and repeat attempt.
*/
public attempts(): ReadonlyArray<TestAttempt> {
return (this.task.result?.attempts || []).map((attempt) => ({
state: attempt.state === 'pass' ? 'passed' : attempt.state === 'fail' ? 'failed' : 'skipped',
errors: attempt.errors as TestError[] | undefined,
duration: attempt.duration,
startTime: attempt.startTime,
retryIndex: attempt.retryIndex,
repeatIndex: attempt.repeatIndex,
}))
}

/**
* Test annotations added via the `task.annotate` API during the test execution.
*/
Expand Down Expand Up @@ -609,10 +623,29 @@ function buildOptions(task: RunnerTestCase | RunnerTestSuite): TaskOptions {
}
}

export type TestSuiteState = 'skipped' | 'pending' | 'failed' | 'passed'
export type TestAttemptState = 'skipped' | 'failed' | 'passed'
export type TestSuiteState = TestAttemptState | 'pending'
export type TestModuleState = TestSuiteState | 'queued'
export type TestState = TestResult['state']

/**
* The result of an individual test attempt.
*/
export interface TestAttempt {
/** The state after applying the test's expected failure option. */
readonly state: TestAttemptState
/** Errors produced by this attempt. */
readonly errors: ReadonlyArray<TestError> | undefined
/** How long in milliseconds the attempt took to run. */
readonly duration: number
/** Time in milliseconds when the attempt started running. */
readonly startTime: number
/** The zero-based retry index within the repeat. */
readonly retryIndex: number
/** The zero-based repeat index. */
readonly repeatIndex: number
}

export type TestResult = TestResultPassed | TestResultFailed | TestResultSkipped | TestResultPending

export interface TestResultPending {
Expand Down
1 change: 1 addition & 0 deletions packages/vitest/src/public/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ export type {
TaskBase as RunnerTaskBase,
TaskEventPack as RunnerTaskEventPack,
TaskResult as RunnerTaskResult,
TaskResultAttempt as RunnerTaskResultAttempt,
TaskResultPack as RunnerTaskResultPack,
Test as RunnerTestCase,
File as RunnerTestFile,
Expand Down
2 changes: 2 additions & 0 deletions packages/vitest/src/public/node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ export type { Report } from '../node/reporters/report'
export type {
ModuleDiagnostic,
TaskOptions,
TestAttempt,
TestAttemptState,
TestCase,
TestCollection,
TestDiagnostic,
Expand Down
18 changes: 18 additions & 0 deletions packages/vitest/src/runtime/runner/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import type {
Task,
TaskMeta,
TaskResult,
TaskResultAttempt,
TaskResultPack,
TaskState,
TaskUpdateEvent,
Expand Down Expand Up @@ -586,6 +587,7 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
state: 'run',
startTime: unixNow(),
retryCount: 0,
attempts: [],
}
updateTask('test-prepare', test, runner)

Expand All @@ -602,7 +604,20 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
test.result.state = 'run' as TaskState
const retry = getRetryCount(test.retry)
for (let retryCount = 0; retryCount <= retry; retryCount++) {
const attemptStart = now()
const attemptStartTime = unixNow()
const attemptErrorsStart = test.result.errors?.length ?? 0
const recordAttempt = (state: TaskResultAttempt['state']) => {
const errors = test.result!.errors?.slice(attemptErrorsStart)
test.result!.attempts!.push({
state,
errors: errors?.length ? errors : undefined,
duration: now() - attemptStart,
startTime: attemptStartTime,
retryIndex: retryCount,
repeatIndex: repeatCount,
})
}
let beforeEachCleanups: unknown[] = []
// fixtureCheckpoint is passed by callAroundEachHooks - it represents the count
// of fixture cleanup functions AFTER all aroundEach fixtures have been resolved
Expand Down Expand Up @@ -694,12 +709,14 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {

// skipped with new PendingError
if (test.result?.pending || test.result?.state === 'skip') {
recordAttempt('skip')
test.mode = 'skip'
test.result = {
state: 'skip',
note: test.result?.note,
pending: true,
duration: now() - start,
attempts: test.result.attempts,
}
updateTask('test-finished', test, runner)
setCurrentTest(undefined)
Expand All @@ -724,6 +741,7 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
}
}
}
recordAttempt(test.result.state === 'pass' ? 'pass' : 'fail')
if (test.result.state === 'pass') {
break
}
Expand Down
22 changes: 22 additions & 0 deletions packages/vitest/src/runtime/runner/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,24 @@ export interface TaskMeta {
__vitest_label__?: string
}

/**
* The result of an individual test attempt.
*/
export interface TaskResultAttempt {
/** The state after applying the test's expected failure option. */
state: 'pass' | 'fail' | 'skip'
/** Errors produced by this attempt. */
errors?: TestError[]
/** How long in milliseconds the attempt took to run. */
duration: number
/** Time in milliseconds when the attempt started running. */
startTime: number
/** The zero-based retry index within the repeat. */
retryIndex: number
/** The zero-based repeat index. */
repeatIndex: number
}

/**
* The result of calling a task.
*/
Expand All @@ -174,6 +192,10 @@ export interface TaskResult {
* if `expect.soft()` failed multiple times or `retry` was triggered.
*/
errors?: TestError[]
/**
* Individual results for every retry and repeat attempt.
*/
attempts?: TaskResultAttempt[]
/**
* How long in milliseconds the task took to run.
*/
Expand Down
92 changes: 92 additions & 0 deletions test/e2e/test/reported-tasks.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,39 @@ it('correctly reports flaky tests', ({ testModule }) => {
const result = testFlaky.result()!
expect(result.state).toBe('passed')
expect(result.errors).toHaveLength(2)
expect(
testFlaky.attempts().map((attempt) => ({
state: attempt.state,
errors: attempt.errors?.map((error) => error.message),
retryIndex: attempt.retryIndex,
repeatIndex: attempt.repeatIndex,
})),
).toMatchInlineSnapshot(`
[
{
"errors": [
"expected +0 to be 2 // Object.is equality",
],
"repeatIndex": 0,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 0,
"retryIndex": 1,
"state": "failed",
},
{
"errors": undefined,
"repeatIndex": 0,
"retryIndex": 2,
"state": "passed",
},
]
`)
})

it('correctly reports repeated tests', ({ testModule }) => {
Expand All @@ -287,6 +320,65 @@ it('correctly reports repeated tests', ({ testModule }) => {
const result = testRepeated.result()!
expect(result.state).toBe('failed')
expect(result.errors).toHaveLength(6)
expect(
testRepeated.attempts().map((attempt) => ({
state: attempt.state,
errors: attempt.errors?.map((error) => error.message),
retryIndex: attempt.retryIndex,
repeatIndex: attempt.repeatIndex,
})),
).toMatchInlineSnapshot(`
[
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 0,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 1,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 2,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 3,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 4,
"retryIndex": 0,
"state": "failed",
},
{
"errors": [
"expected 1 to be 2 // Object.is equality",
],
"repeatIndex": 5,
"retryIndex": 0,
"state": "failed",
},
]
`)
})

it('correctly passed down metadata', ({ testModule }) => {
Expand Down
Loading
Loading