Skip to content

Commit 1b72c3e

Browse files
AriPerkkioAriPerkkio
committed
docs: add Profiling Test Performance guide
1 parent d8af763 commit 1b72c3e

File tree

12 files changed

+358
-2
lines changed

12 files changed

+358
-2
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ logs
33
npm-debug.log*
44
yarn-debug.log*
55
yarn-error.log*
6+
*.cpuprofile
7+
*.heapprofile
68
lib-cov
79
coverage
810
!**/integrations/coverage

‎docs/.vitepress/config.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,6 +325,10 @@ export default ({ mode }: { mode: string }) => {
325325
text: 'Common Errors',
326326
link: '/guide/common-errors',
327327
},
328+
{
329+
text: 'Profiling Test Performance',
330+
link: '/guide/profiling-test-performance',
331+
},
328332
{
329333
text: 'Improving Performance',
330334
link: '/guide/improving-performance',

‎docs/guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,7 @@ Learn more about [IDE Integrations](/guide/ide)
251251
| `react` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/react) | [Play Online](https://stackblitz.com/fork/github/vitest-dev/vitest/tree/main/examples/react?initialPath=__vitest__/) |
252252
| `solid` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/solid) | [Play Online](https://stackblitz.com/fork/github/vitest-dev/vitest/tree/main/examples/solid?initialPath=__vitest__/) |
253253
| `sveltekit` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/sveltekit) | [Play Online](https://stackblitz.com/fork/github/vitest-dev/vitest/tree/main/examples/sveltekit?initialPath=__vitest__/) |
254+
| `profiling` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/profiling) | Not Available |
254255
| `typecheck` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/typecheck) | [Play Online](https://stackblitz.com/fork/github/vitest-dev/vitest/tree/main/examples/typecheck?initialPath=__vitest__/) |
255256
| `workspace` | [GitHub](https://github.com/vitest-dev/vitest/tree/main/examples/workspace) | [Play Online](https://stackblitz.com/fork/github/vitest-dev/vitest/tree/main/examples/workspace?initialPath=__vitest__/) |
256257

‎docs/guide/profiling-test-performance.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# Profiling Test Performance
2+
3+
When you run Vitest it reports multiple time metrics of your tests:
4+
5+
> ```bash
6+
> RUN v2.1.1 /x/vitest/examples/profiling
7+
>
8+
> ✓ test/prime-number.test.ts (1) 4517ms
9+
> ✓ generate prime number 4517ms
10+
>
11+
> Test Files 1 passed (1)
12+
> Tests 1 passed (1)
13+
> Start at 09:32:53
14+
> Duration 4.80s (transform 44ms, setup 0ms, collect 35ms, tests 4.52s, environment 0ms, prepare 81ms)
15+
> # Time metrics ^^
16+
> ```
17+
18+
- Transform: How much time was spent transforming the files. See [File Transform](#file-transform).
19+
- Setup: Time spent for running the [`setupFiles`](/config/#setupfiles) files.
20+
- Collect: Time spent for colleting all tests in the test files. This includes the time it took to import all file dependencies.
21+
- Tests: Time spent for actually running the test cases.
22+
- Environment: Time spent for setting up the test environment, for example JSDOM.
23+
- Prepare: Time Vitest uses to prepare the test runner.
24+
25+
## Test runner
26+
27+
In cases where your test execution time is high, you can generate a profile of the test runner. See NodeJS documentation for following options:
28+
29+
- [`--cpu-prof`](https://nodejs.org/api/cli.html#--cpu-prof)
30+
- [`--heap-prof`](https://nodejs.org/api/cli.html#--heap-prof)
31+
- [`--prof`](https://nodejs.org/api/cli.html#--prof)
32+
33+
:::warning
34+
The `--prof` option does not work with `pool: 'threads'` due to `node:worker_threads` limitations.
35+
:::
36+
37+
To pass these options to Vitest's test runner, define `poolOptions.<pool>.execArgv` in your Vitest configuration:
38+
39+
::: code-group
40+
```ts [Forks]
41+
import { defineConfig } from 'vitest/config'
42+
43+
export default defineConfig({
44+
test: {
45+
pool: 'forks',
46+
poolOptions: {
47+
forks: {
48+
execArgv: [
49+
'--cpu-prof',
50+
'--cpu-prof-dir=test-runner-profile',
51+
'--heap-prof',
52+
'--heap-prof-dir=test-runner-profile'
53+
],
54+
55+
// To generate a single profile
56+
singleFork: true,
57+
},
58+
},
59+
},
60+
})
61+
```
62+
```ts [Threads]
63+
import { defineConfig } from 'vitest/config'
64+
65+
export default defineConfig({
66+
test: {
67+
pool: 'threads',
68+
poolOptions: {
69+
threads: {
70+
execArgv: [
71+
'--cpu-prof',
72+
'--cpu-prof-dir=test-runner-profile',
73+
'--heap-prof',
74+
'--heap-prof-dir=test-runner-profile'
75+
],
76+
77+
// To generate a single profile
78+
singleThread: true,
79+
},
80+
},
81+
},
82+
})
83+
```
84+
:::
85+
86+
After the tests have run there should be a `test-runner-profile/*.cpuprofile` and `test-runner-profile/*.heapprofile` files generated. See [Inspecting profiling records](#inspecting-profiling-records) for instructions how to analyze these files.
87+
88+
See [Profiling | Examples](https://github.com/vitest-dev/vitest/tree/main/examples/profiling) for example.
89+
90+
## Main thread
91+
92+
Profiling main thread is useful for debugging Vitest's Vite usage and [`globalSetup`](/config/#globalsetup) files.
93+
This is also where your Vite plugins are running.
94+
95+
:::tip
96+
See [Performance | Vite](https://vitejs.dev/guide/performance.html) for more tips about Vite specific profiling.
97+
:::
98+
99+
To do this you'll need to pass arguments to the Node process that runs Vitest.
100+
101+
```bash
102+
$ node --cpu-prof --cpu-prof-dir=main-profile ./node_modules/vitest/vitest.mjs --run
103+
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^
104+
# NodeJS arguments Vitest arguments
105+
```
106+
107+
After the tests have run there should be a `main-profile/*.cpuprofile` file generated. See [Inspecting profiling records](#inspecting-profiling-records) for instructions how to analyze these files.
108+
109+
## File transform
110+
111+
In cases where your test transform and collection time is high, you can use `DEBUG=vite-node:*` environment variable to see which files are being transformed and executed by `vite-node`.
112+
113+
```bash
114+
$ DEBUG=vite-node:* vitest --run
115+
116+
RUN v2.1.1 /x/vitest/examples/profiling
117+
118+
vite-node:server:request /x/vitest/examples/profiling/global-setup.ts +0ms
119+
vite-node:client:execute /x/vitest/examples/profiling/global-setup.ts +0ms
120+
vite-node:server:request /x/vitest/examples/profiling/test/prime-number.test.ts +45ms
121+
vite-node:client:execute /x/vitest/examples/profiling/test/prime-number.test.ts +26ms
122+
vite-node:server:request /src/prime-number.ts +9ms
123+
vite-node:client:execute /x/vitest/examples/profiling/src/prime-number.ts +9ms
124+
vite-node:server:request /src/unnecessary-file.ts +6ms
125+
vite-node:client:execute /x/vitest/examples/profiling/src/unnecessary-file.ts +4ms
126+
...
127+
```
128+
129+
This profiling strategy is a good way to identify unnecessary transforms caused by [barrel files](https://vitejs.dev/guide/performance.html#avoid-barrel-files).
130+
If these logs contain files that should not be loaded when your test is run, you might have barrel files that are importing files unnecessarily.
131+
132+
You can also use [Vitest UI](/guide/ui) to debug slowness caused by barrel file.
133+
The example below shows how importing files without barrel file reduces amount of transformed files by ~85%.
134+
135+
::: code-group
136+
``` [File tree]
137+
├── src
138+
│ └── utils
139+
│ ├── currency.ts
140+
│ ├── formatters.ts <-- File to test
141+
│ ├── index.ts
142+
│ ├── location.ts
143+
│ ├── math.ts
144+
│ ├── time.ts
145+
│ └── users.ts
146+
├── test
147+
│ └── formatters.test.ts
148+
└── vitest.config.ts
149+
```
150+
```ts [example.test.ts]
151+
import { expect, test } from 'vitest'
152+
import { formatter } from '../src/utils' // [!code --]
153+
import { formatter } from '../src/utils/formatters' // [!code ++]
154+
155+
test('formatter works', () => {
156+
expect(formatter).not.toThrow()
157+
})
158+
```
159+
:::
160+
161+
<img src="/module-graph-barrel-file.png" alt="Vitest UI demonstrating barrel file issues" />
162+
163+
To see how files are transformed, you can use `VITE_NODE_DEBUG_DUMP` environment variable to write transformed files in the file system:
164+
165+
```bash
166+
$ VITE_NODE_DEBUG_DUMP=true vitest --run
167+
168+
[vite-node] [debug] dump modules to /x/examples/profiling/.vite-node/dump
169+
170+
RUN v2.1.1 /x/vitest/examples/profiling
171+
...
172+
173+
$ ls .vite-node/dump/
174+
_x_examples_profiling_global-setup_ts-1292904907.js
175+
_x_examples_profiling_test_prime-number_test_ts-1413378098.js
176+
_src_prime-number_ts-525172412.js
177+
```
178+
179+
## Inspecting profiling records
180+
181+
You can inspect the contents of `*.cpuprofile` and `*.heapprofile` with various tools. See list below for examples.
182+
183+
- [Speedscope](https://www.speedscope.app/)
184+
- [Profile Node.js performance with the Performance panel | developer.chrome.com](https://developer.chrome.com/docs/devtools/performance/nodejs#analyze)
185+
- [Memory panel overview | developer.chrome.com](https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots#view_snapshots)

‎docs/public/module-graph-barrel-file.png‎

499 KB
Loading

‎examples/profiling/global-setup.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
import { rmSync } from 'node:fs'
2+
3+
export function setup() {
4+
rmSync('./threads-profile', { force: true, recursive: true })
5+
rmSync('./forks-profile', { force: true, recursive: true })
6+
}

‎examples/profiling/package.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"name": "@vitest/example-profiling",
3+
"type": "module",
4+
"private": true,
5+
"license": "MIT",
6+
"scripts": {
7+
"test": "vitest"
8+
},
9+
"devDependencies": {
10+
"vite": "latest",
11+
"vitest": "latest"
12+
}
13+
}

‎examples/profiling/src/prime-number.ts‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
/* eslint-disable unicorn/no-new-array */
2+
3+
const store: bigint[] = []
4+
5+
export default function getPrimeNumber(bitLength: number): bigint {
6+
if (!bitLength) {
7+
throw new Error('bitLength is required')
8+
}
9+
10+
const number = randomBigInt(bitLength)
11+
12+
if (isPrimeNumber(number) && !store.includes(number)) {
13+
store.push(number)
14+
15+
return number
16+
}
17+
18+
return getPrimeNumber(bitLength)
19+
}
20+
21+
/**
22+
* Generate random `BigInt` with given bit length
23+
* e.g. randomBigInt(8) -> 153n (1001 1001)
24+
*/
25+
function randomBigInt(bitLength: number): bigint {
26+
const binaryString: string = new Array(bitLength)
27+
// MSB should be one to guarantee bit length
28+
.fill('1')
29+
// Fill string with 0s and 1s
30+
.reduce(bin => bin + Math.round(Math.random()).toString())
31+
32+
return BigInt(`0b${binaryString}`)
33+
}
34+
35+
function isPrimeNumber(number: bigint): boolean {
36+
if (number <= 2n) {
37+
return false
38+
}
39+
40+
if (number % 2n === 0n) {
41+
return false
42+
}
43+
44+
if (number === 3n) {
45+
return true
46+
}
47+
48+
const squareRoot = bigIntSquareRoot(number)
49+
50+
// Intentionally unefficient to highlight performance issues
51+
for (let i = 3n; i < squareRoot; i += 2n) {
52+
if (number % i === 0n) {
53+
return false
54+
}
55+
}
56+
57+
return true
58+
}
59+
60+
function bigIntSquareRoot(number: bigint): bigint {
61+
if (number < 0n) {
62+
throw new Error('Negative numbers are not supported')
63+
}
64+
if (number < 2n) {
65+
return number
66+
}
67+
68+
function iterate(value: bigint, guess: bigint): bigint {
69+
const nextGuess = (value / guess + guess) >> 1n
70+
71+
if (guess === nextGuess) {
72+
return guess
73+
}
74+
if (guess === nextGuess - 1n) {
75+
return guess
76+
}
77+
78+
return iterate(value, nextGuess)
79+
}
80+
81+
return iterate(number, 1n)
82+
}

‎examples/profiling/test/prime-number.test.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import { expect, test } from 'vitest'
2+
import getPrimeNumber from '../src/prime-number'
3+
4+
const BITS = 62
5+
6+
test('generate prime number', () => {
7+
const prime = getPrimeNumber(BITS)
8+
9+
expect(prime).toBeGreaterThanOrEqual(0)
10+
expect(prime).toBeLessThanOrEqual(2 ** BITS)
11+
})

‎examples/profiling/vitest.config.ts‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import { defineConfig } from 'vitest/config'
2+
3+
export default defineConfig({
4+
test: {
5+
watch: false,
6+
globalSetup: './global-setup.ts',
7+
8+
// Switch between forks|threads
9+
pool: 'forks',
10+
11+
poolOptions: {
12+
threads: {
13+
execArgv: [
14+
// https://nodejs.org/api/cli.html#--cpu-prof
15+
'--cpu-prof',
16+
'--cpu-prof-dir=threads-profile',
17+
18+
// https://nodejs.org/api/cli.html#--heap-prof
19+
'--heap-prof',
20+
'--heap-prof-dir=threads-profile',
21+
],
22+
23+
// Generate a single profile
24+
singleThread: true,
25+
},
26+
27+
forks: {
28+
execArgv: [
29+
// https://nodejs.org/api/cli.html#--cpu-prof
30+
'--cpu-prof',
31+
'--cpu-prof-dir=forks-profile',
32+
33+
// https://nodejs.org/api/cli.html#--heap-prof
34+
'--heap-prof',
35+
'--heap-prof-dir=forks-profile',
36+
],
37+
38+
// Generate a single profile
39+
singleFork: true,
40+
},
41+
},
42+
},
43+
})

0 commit comments

Comments
 (0)