Skip to content

Docs Search - Add script to generate openai embeddings of the markdown #2806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
NWylynko wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

Docs Search - Add script to generate openai embeddings of the markdown #2806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8e105a9
Add vercel package for pulling project envs
NWylynko Nov 11, 2025
95efbec
load in markdown and chunk it (still need to filter out undesired mar…
NWylynko Nov 11, 2025
b28ece8
run the generate embeddings step
NWylynko Nov 11, 2025
207c920
Cut down the content being chunking by removing code examples and com…
NWylynko Nov 12, 2025
08bd474
Finish script, calling openai api to generate embeddings
NWylynko Nov 12, 2025
5cb63c5
Clean up the markdown for creating embeddings
NWylynko Nov 12, 2025
14c72f3
Add a endpoint to query embeddings
NWylynko Nov 12, 2025
e6d930b
put the embeddings in to the api folder
NWylynko Nov 12, 2025
1e8a2f2
set tsconfig for api routes
NWylynko Nov 12, 2025
3ea5c39
basic search handler
NWylynko Nov 12, 2025
cf11d1a
use type module
NWylynko Nov 12, 2025
8ef5c34
remove 'assert json'
NWylynko Nov 12, 2025
347cb26
use a ts file to import from
NWylynko Nov 12, 2025
7251aaa
OOM during build
NWylynko Nov 12, 2025
34b3d7b
just attempt to load from file system
NWylynko Nov 12, 2025
6785fbd
list the file system
NWylynko Nov 12, 2025
17a9704
list inside the api folder
NWylynko Nov 12, 2025
7cdf24a
Lets handle the search in clerk/clerk
NWylynko Nov 12, 2025
f2d81dc
clean up
NWylynko Nov 12, 2025
2f3eb13
Include more metadata for each chunk
NWylynko Nov 12, 2025
2a4d8ed
move where we switch to paragraph type chunks
NWylynko Nov 12, 2025
3c39912
include search ranking in chunk metadata
NWylynko Nov 12, 2025
e6f2bb3
include search ranking
NWylynko Nov 12, 2025
ee716dc
Update encoding format in generate-embeddings script for optimized ne…
NWylynko Nov 13, 2025
ba6a7dd
include both the available sdks and the active sdk for each chunk
NWylynko Nov 13, 2025
b2008f0
switch to using ai sdk
NWylynko Nov 13, 2025
19f263e
drop embeddings dimensions down to 512 (1/3)
NWylynko Nov 13, 2025
c5d524f
Remove vercel dependency, use ai@5 instead of beta
NWylynko Nov 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion flags.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
{}
{
"use-ai-search": true
}
172 changes: 169 additions & 3 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,15 @@
"move-doc": "node scripts/move-doc.mjs",
"delete-doc": "node scripts/delete-doc.mjs",
"migrate-sdk-scoping": "tsx scripts/migrate-sdk-scoping.ts",
"migrate-api-reference-links": "tsx scripts/migrate-api-reference-links.ts"
"migrate-api-reference-links": "tsx scripts/migrate-api-reference-links.ts",
"generate-embeddings": "bun scripts/generate-embeddings.ts"
},
"devDependencies": {
"@ai-sdk/openai": "^2.0.65",
"@parcel/watcher": "^2.5.1",
"@sindresorhus/slugify": "^2.2.1",
"@types/node": "^22.13.2",
"ai": "^5.0.93",
"bun": "^1.2.20",
"chokidar": "^4.0.3",
"concurrently": "^8.2.2",
Expand All @@ -46,6 +49,7 @@
"remark-mdx": "^3.0.1",
"simple-git": "^3.27.0",
"symlink-dir": "^6.0.5",
"tiktoken": "^1.0.22",
"tsx": "^4.19.2",
"typescript": "^5.7.3",
"unist-builder": "^4.0.0",
Expand Down
Loading
Loading