Skip to content

Commit c6d7ebe

Browse files
bholmesdevbluwysarah11918
authored
Data collections and references (#6850)
* feat: add generated lookup-map * feat: wire up fast getEntryBySlug() lookup * fix: consider frontmatter slugs * chore: changeset * chore: lint no-shadow * fix: revert bad rootRelativePath change * chore: better var name * refactor: generated `.json` to in-memory map * chore: removed unneeded await Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com> * chore: removed unneeded await Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com> * Revert "chore: removed unneeded await" This reverts commit 1b0a8b0. * fix: bad `GetEntryImport` type * chore: remove unused variable * refactor: for -> Promise.all * refactor: replace duplicate parseSlug * refactor: add cache layer * Revert "refactor: add cache layer" This reverts commit 1c3bfdc. * feat: json collection POC * wip: add test json file * wip: playing with api ideas * refactor: extract getCollectionName * feat: add defineDataCollection * refactor: variable destructure * wip: basic data entry pipeline * chore: revert fixture playing * wip: basic entry array parser * feat: basic data type gen * chore: add with-data playground * feat: add error when `defineDataCollection()` isn't used * fix: missing error message * feat: data collections are here! * wip: play with data query APIs * feat: reference() util! * fix: Markdoc `$entry` variable * play: add reference util with markdoc * chore: delete console logs * feat: `src/data/`! * feat: reference() errors * fix: handle hoisted schema parse errors * fix: reload config and invalid on collection changes * feat: separate maps for content and data entries * feat: new `reference()` API that fixes type inference * feat: support `defineCollection()` for data config * fix: defineCollection `type` inferenenceπinference * chore: lock * feat: getCollection() for everything! * feat: get full entry access from reference() * chore: changeset * wip: type error on acorn? * chore: lint * chore: add slugger to data ID processing * chore: astro/zod -> zod * chore: example version * chore: remove slugifier from data id * chore: remove dead getDataCollection * chore: remove dead defineDataCollection * fix: bad collection import * chore: lock * feat: add data collections to lookup map * refactor: stop resolving data from reference * feat: introduce getEntry and new reference() * fix: update config loader * fix: reference() type * feat: test self references (they work 🎉) * fix: use `slug` for content references * fix: bad getEntry content type * chroe: remove console logs * fix: strict null checks on with-data * feat: add getEntries for ref arrays * chore: fix type hints for reference strings * chore: change to type never for clarity * play: try getEntries * Return to "everything goes in `src/content/` This reverts commit cc637ec. * fix: remove old function * chore: update to AstroErrors * chore: remove unused fixture files * play: names * deps: js-yaml * feat: data collection YAML with error handling * refactor: remove console log * refactor: code cleanup * fix: allow mixed content to pass through glob imports * chore: move lookupMap util to virtual-mod * refactor: new lookupMap logic, better errors * chore: change MixedContent title * refactor: remove unneeded try / catch * fix: use `ws.send` for type gen errors * fix: bubble `ws.send` errors from astro sync * refactor: revert verbose astroContentCollectionEntry * fix: bad with-data package name * fix: bad virtual mod flag * chore: remove with-data playground * test: data collection authors * test: translations data collection * fix: add `.yml` support * refactor: mix in `.yaml` just for fun * refactor: i18n -> translations * chore: content-collection-references fixture * chore: bad lockfile * fix: bad ContentLookupMap import * chore: revert back to astroContentCollectionEntry * test: collection references * fix: bad error code override * chore: remove unused asset * test: sync errors * chore: remove stray console log * chore: lock * chore: revert with-markdoc changes * chore: doc error states, remove bad merge code * chore: remove bad `as any` * chore: lint * chore: inline ContentLookupMap comments * chore: settings -> config * fix: put back `defineCollection()` * fix: entry.slug for get content collection * chore: update get-entry-type tests * docs: totally shorten "missing a `type`" Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * docs: truncate share a `schema` Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca> * chore: add `test:unit` and `test:unit:match`to base * chore: update changeset * refactor: cleanup runtime types and inline comments * nit: [0] instead of shift() * refactor: `getRelativeEntryPath()` util * chore: capitalized Collections for test:match * nit: ?? viteId on split * nit: separate Params obj * chore: add try / catch on readFile * nit: `const data` * chore: clean up data collection exceptions * nit: `?? ''` for search params * chore: remove TODO on hoisted error --------- Co-authored-by: Bjorn Lu <bjornlu.dev@gmail.com> Co-authored-by: Sarah Rainsberger <sarah@rainsberger.ca>
1 parent fc52681 commit c6d7ebe

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+1987
-345
lines changed

.changeset/early-eyes-bow.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'astro': minor
3+
'@astrojs/markdoc': minor
4+
---
5+
6+
Content collections now support data formats including JSON and YAML. You can also create relationships, or references, between collections to pull information from one collection entry into another. Learn more on our [updated Content Collections docs](https://docs.astro.build/en/guides/content-collections/).

package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818
"format:imports": "organize-imports-cli ./packages/*/tsconfig.json ./packages/*/*/tsconfig.json",
1919
"test": "turbo run test --concurrency=1 --filter=astro --filter=create-astro --filter=\"@astrojs/*\"",
2020
"test:match": "cd packages/astro && pnpm run test:match",
21+
"test:unit": "cd packages/astro && pnpm run test:unit",
22+
"test:unit:match": "cd packages/astro && pnpm run test:unit:match",
2123
"test:smoke": "pnpm test:smoke:example && pnpm test:smoke:docs",
2224
"test:smoke:example": "turbo run build --concurrency=100% --filter=\"@example/*\"",
2325
"test:smoke:docs": "turbo run build --filter=docs",

packages/astro/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,7 @@
144144
"github-slugger": "^2.0.0",
145145
"gray-matter": "^4.0.3",
146146
"html-escaper": "^3.0.3",
147+
"js-yaml": "^4.1.0",
147148
"kleur": "^4.1.4",
148149
"magic-string": "^0.27.0",
149150
"mime": "^3.0.0",
@@ -181,6 +182,7 @@
181182
"@types/estree": "^0.0.51",
182183
"@types/hast": "^2.3.4",
183184
"@types/html-escaper": "^3.0.0",
185+
"@types/js-yaml": "^4.0.5",
184186
"@types/mime": "^2.0.3",
185187
"@types/mocha": "^9.1.1",
186188
"@types/prettier": "^2.6.3",

packages/astro/src/@types/astro.ts

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1250,12 +1250,22 @@ export type ContentEntryModule = {
12501250
};
12511251
};
12521252

1253+
export type DataEntryModule = {
1254+
id: string;
1255+
collection: string;
1256+
data: Record<string, unknown>;
1257+
_internal: {
1258+
rawData: string;
1259+
filePath: string;
1260+
};
1261+
};
1262+
12531263
export interface ContentEntryType {
12541264
extensions: string[];
12551265
getEntryInfo(params: {
12561266
fileUrl: URL;
12571267
contents: string;
1258-
}): GetEntryInfoReturnType | Promise<GetEntryInfoReturnType>;
1268+
}): GetContentEntryInfoReturnType | Promise<GetContentEntryInfoReturnType>;
12591269
getRenderModule?(
12601270
this: rollup.PluginContext,
12611271
params: {
@@ -1266,7 +1276,7 @@ export interface ContentEntryType {
12661276
contentModuleTypes?: string;
12671277
}
12681278

1269-
type GetEntryInfoReturnType = {
1279+
type GetContentEntryInfoReturnType = {
12701280
data: Record<string, unknown>;
12711281
/**
12721282
* Used for error hints to point to correct line and location
@@ -1278,12 +1288,23 @@ type GetEntryInfoReturnType = {
12781288
slug: string;
12791289
};
12801290

1291+
export interface DataEntryType {
1292+
extensions: string[];
1293+
getEntryInfo(params: {
1294+
fileUrl: URL;
1295+
contents: string;
1296+
}): GetDataEntryInfoReturnType | Promise<GetDataEntryInfoReturnType>;
1297+
}
1298+
1299+
export type GetDataEntryInfoReturnType = { data: Record<string, unknown>; rawData?: string };
1300+
12811301
export interface AstroSettings {
12821302
config: AstroConfig;
12831303
adapter: AstroAdapter | undefined;
12841304
injectedRoutes: InjectedRoute[];
12851305
pageExtensions: string[];
12861306
contentEntryTypes: ContentEntryType[];
1307+
dataEntryTypes: DataEntryType[];
12871308
renderers: AstroRenderer[];
12881309
scripts: {
12891310
stage: InjectedScriptStage;

packages/astro/src/content/consts.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
export const PROPAGATED_ASSET_FLAG = 'astroPropagatedAssets';
2-
export const CONTENT_FLAG = 'astroContent';
2+
export const CONTENT_FLAG = 'astroContentCollectionEntry';
3+
export const DATA_FLAG = 'astroDataCollectionEntry';
4+
export const CONTENT_FLAGS = [CONTENT_FLAG, DATA_FLAG, PROPAGATED_ASSET_FLAG] as const;
5+
36
export const VIRTUAL_MODULE_ID = 'astro:content';
47
export const LINKS_PLACEHOLDER = '@@ASTRO-LINKS@@';
58
export const STYLES_PLACEHOLDER = '@@ASTRO-STYLES@@';

packages/astro/src/content/runtime-assets.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ import type { AstroSettings } from '../@types/astro.js';
44
import { emitESMImage } from '../assets/index.js';
55

66
export function createImage(
7-
settings: AstroSettings,
7+
settings: Pick<AstroSettings, 'config'>,
88
pluginContext: PluginContext,
99
entryFilePath: string
1010
) {

packages/astro/src/content/runtime.ts

Lines changed: 193 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { AstroError, AstroErrorData } from '../core/errors/index.js';
22
import { prependForwardSlash } from '../core/path.js';
3-
3+
import { ZodIssueCode, string as zodString, type z } from 'zod';
44
import {
55
createComponent,
66
createHeadAndContent,
@@ -9,7 +9,10 @@ import {
99
renderTemplate,
1010
renderUniqueStylesheet,
1111
unescapeHTML,
12+
type AstroComponentFactory,
1213
} from '../runtime/server/index.js';
14+
import type { ContentLookupMap } from './utils.js';
15+
import type { MarkdownHeading } from '@astrojs/markdown-remark';
1316

1417
type LazyImport = () => Promise<any>;
1518
type GlobResult = Record<string, LazyImport>;
@@ -37,14 +40,31 @@ export function createCollectionToGlobResultMap({
3740

3841
const cacheEntriesByCollection = new Map<string, any[]>();
3942
export function createGetCollection({
40-
collectionToEntryMap,
43+
contentCollectionToEntryMap,
44+
dataCollectionToEntryMap,
4145
getRenderEntryImport,
4246
}: {
43-
collectionToEntryMap: CollectionToEntryMap;
47+
contentCollectionToEntryMap: CollectionToEntryMap;
48+
dataCollectionToEntryMap: CollectionToEntryMap;
4449
getRenderEntryImport: GetEntryImport;
4550
}) {
4651
return async function getCollection(collection: string, filter?: (entry: any) => unknown) {
47-
const lazyImports = Object.values(collectionToEntryMap[collection] ?? {});
52+
let type: 'content' | 'data';
53+
if (collection in contentCollectionToEntryMap) {
54+
type = 'content';
55+
} else if (collection in dataCollectionToEntryMap) {
56+
type = 'data';
57+
} else {
58+
throw new AstroError({
59+
...AstroErrorData.CollectionDoesNotExistError,
60+
message: AstroErrorData.CollectionDoesNotExistError.message(collection),
61+
});
62+
}
63+
const lazyImports = Object.values(
64+
type === 'content'
65+
? contentCollectionToEntryMap[collection]
66+
: dataCollectionToEntryMap[collection]
67+
);
4868
let entries: any[] = [];
4969
// Cache `getCollection()` calls in production only
5070
// prevents stale cache in development
@@ -54,20 +74,26 @@ export function createGetCollection({
5474
entries = await Promise.all(
5575
lazyImports.map(async (lazyImport) => {
5676
const entry = await lazyImport();
57-
return {
58-
id: entry.id,
59-
slug: entry.slug,
60-
body: entry.body,
61-
collection: entry.collection,
62-
data: entry.data,
63-
async render() {
64-
return render({
77+
return type === 'content'
78+
? {
79+
id: entry.id,
80+
slug: entry.slug,
81+
body: entry.body,
6582
collection: entry.collection,
83+
data: entry.data,
84+
async render() {
85+
return render({
86+
collection: entry.collection,
87+
id: entry.id,
88+
renderEntryImport: await getRenderEntryImport(collection, entry.slug),
89+
});
90+
},
91+
}
92+
: {
6693
id: entry.id,
67-
renderEntryImport: await getRenderEntryImport(collection, entry.slug),
68-
});
69-
},
70-
};
94+
collection: entry.collection,
95+
data: entry.data,
96+
};
7197
})
7298
);
7399
cacheEntriesByCollection.set(collection, entries);
@@ -110,6 +136,121 @@ export function createGetEntryBySlug({
110136
};
111137
}
112138

139+
export function createGetDataEntryById({
140+
dataCollectionToEntryMap,
141+
}: {
142+
dataCollectionToEntryMap: CollectionToEntryMap;
143+
}) {
144+
return async function getDataEntryById(collection: string, id: string) {
145+
const lazyImport =
146+
dataCollectionToEntryMap[collection]?.[/*TODO: filePathToIdMap*/ id + '.json'];
147+
148+
// TODO: AstroError
149+
if (!lazyImport) throw new Error(`Entry ${collection}${id} was not found.`);
150+
const entry = await lazyImport();
151+
152+
return {
153+
id: entry.id,
154+
collection: entry.collection,
155+
data: entry.data,
156+
};
157+
};
158+
}
159+
160+
type ContentEntryResult = {
161+
id: string;
162+
slug: string;
163+
body: string;
164+
collection: string;
165+
data: Record<string, any>;
166+
render(): Promise<RenderResult>;
167+
};
168+
169+
type DataEntryResult = {
170+
id: string;
171+
collection: string;
172+
data: Record<string, any>;
173+
};
174+
175+
type EntryLookupObject = { collection: string; id: string } | { collection: string; slug: string };
176+
177+
export function createGetEntry({
178+
getEntryImport,
179+
getRenderEntryImport,
180+
}: {
181+
getEntryImport: GetEntryImport;
182+
getRenderEntryImport: GetEntryImport;
183+
}) {
184+
return async function getEntry(
185+
// Can either pass collection and identifier as 2 positional args,
186+
// Or pass a single object with the collection and identifier as properties.
187+
// This means the first positional arg can have different shapes.
188+
collectionOrLookupObject: string | EntryLookupObject,
189+
_lookupId?: string
190+
): Promise<ContentEntryResult | DataEntryResult | undefined> {
191+
let collection: string, lookupId: string;
192+
if (typeof collectionOrLookupObject === 'string') {
193+
collection = collectionOrLookupObject;
194+
if (!_lookupId)
195+
throw new AstroError({
196+
...AstroErrorData.UnknownContentCollectionError,
197+
message: '`getEntry()` requires an entry identifier as the second argument.',
198+
});
199+
lookupId = _lookupId;
200+
} else {
201+
collection = collectionOrLookupObject.collection;
202+
// Identifier could be `slug` for content entries, or `id` for data entries
203+
lookupId =
204+
'id' in collectionOrLookupObject
205+
? collectionOrLookupObject.id
206+
: collectionOrLookupObject.slug;
207+
}
208+
209+
const entryImport = await getEntryImport(collection, lookupId);
210+
if (typeof entryImport !== 'function') return undefined;
211+
212+
const entry = await entryImport();
213+
214+
if (entry._internal.type === 'content') {
215+
return {
216+
id: entry.id,
217+
slug: entry.slug,
218+
body: entry.body,
219+
collection: entry.collection,
220+
data: entry.data,
221+
async render() {
222+
return render({
223+
collection: entry.collection,
224+
id: entry.id,
225+
renderEntryImport: await getRenderEntryImport(collection, lookupId),
226+
});
227+
},
228+
};
229+
} else if (entry._internal.type === 'data') {
230+
return {
231+
id: entry.id,
232+
collection: entry.collection,
233+
data: entry.data,
234+
};
235+
}
236+
return undefined;
237+
};
238+
}
239+
240+
export function createGetEntries(getEntry: ReturnType<typeof createGetEntry>) {
241+
return async function getEntries(
242+
entries: { collection: string; id: string }[] | { collection: string; slug: string }[]
243+
) {
244+
return Promise.all(entries.map((e) => getEntry(e)));
245+
};
246+
}
247+
248+
type RenderResult = {
249+
Content: AstroComponentFactory;
250+
headings: MarkdownHeading[];
251+
remarkPluginFrontmatter: Record<string, any>;
252+
};
253+
113254
async function render({
114255
collection,
115256
id,
@@ -118,7 +259,7 @@ async function render({
118259
collection: string;
119260
id: string;
120261
renderEntryImport?: LazyImport;
121-
}) {
262+
}): Promise<RenderResult> {
122263
const UnexpectedRenderError = new AstroError({
123264
...AstroErrorData.UnknownContentCollectionError,
124265
message: `Unexpected error while rendering ${String(collection)}${String(id)}.`,
@@ -186,3 +327,38 @@ async function render({
186327
remarkPluginFrontmatter: mod.frontmatter ?? {},
187328
};
188329
}
330+
331+
export function createReference({ lookupMap }: { lookupMap: ContentLookupMap }) {
332+
return function reference(collection: string) {
333+
return zodString().transform((lookupId: string, ctx) => {
334+
const flattenedErrorPath = ctx.path.join('.');
335+
if (!lookupMap[collection]) {
336+
ctx.addIssue({
337+
code: ZodIssueCode.custom,
338+
message: `**${flattenedErrorPath}:** Reference to ${collection} invalid. Collection does not exist or is empty.`,
339+
});
340+
return;
341+
}
342+
343+
const { type, entries } = lookupMap[collection];
344+
const entry = entries[lookupId];
345+
346+
if (!entry) {
347+
ctx.addIssue({
348+
code: ZodIssueCode.custom,
349+
message: `**${flattenedErrorPath}**: Reference to ${collection} invalid. Expected ${Object.keys(
350+
entries
351+
)
352+
.map((c) => JSON.stringify(c))
353+
.join(' | ')}. Received ${JSON.stringify(lookupId)}.`,
354+
});
355+
return;
356+
}
357+
// Content is still identified by slugs, so map to a `slug` key for consistency.
358+
if (type === 'content') {
359+
return { slug: lookupId, collection };
360+
}
361+
return { id: lookupId, collection };
362+
});
363+
};
364+
}

0 commit comments

Comments
 (0)