Content collections backward-compatibility

[ascorbic]

The Content Layer API is new way to define content collections that is available in Astro 4.14 behind a flag, and stable in Astro 5. It is designed to work alongside existing content collections.

Currently (Astro 4.14/ Astro 5 beta), any folder inside src/content is treated as a legacy content collection and processed in exactly the same way as before. This means there is a restriction on content collections that use the content layer API, meaning that they can't be in that directory. This is quite counter-intuitive.

Proposal

By default, collections that use the the old types (content or data) are implemented using the glob loader, with extra backward-compat handling. This includes any collection without a loader defined.

Any legacy content collections are handled like this:

• a glob loader collection is defined, with patterns that match the previous handling (matches src/content/<collection name>/**/*.md and other content extensions depending on installed integrations, with underscore-prefixed files and folders ignored)
• When used in the runtime, the entries have an ID based on the filename in the same format as legacy collections
• A slug field is added with the same format as before
• A render() method is added to the entry, so they can be called using entry.render()
• getEntryBySlug is supported

Legacy data collections are handled like this:

• a glob loader collection is defined, with patterns that match the previous handling (matches src/content/<collection name>/**/*{.json,.yaml} and other data extensions, with underscore-prefixed files and folders ignored)
• Entries have an ID that is not slugified
• getDataEntryById is supported

While these emulate most of the features of legacy collections, they have these differences:

• No implicit collections. In order to be generated, a collection must be defined in config.ts. For legacy collections these can just be empty declarations: e.g.const blog = defineCollection({}). Removing implicit collections means that we can allow content layer collections in src/content.
• The layout field is not supported in Markdown
• Experimental content collection cache is not supported
• Sort order of generated collections is non-deterministic and platform-dependent.
• image().refine() is not supported
• the key for getEntry is typed as string, rather than having types for every entry.

A new config flag legacy.legacyContentCollections is added for users that need the old behavior. When set, collections in src/content are processed in the same way as before rather than being implemented with glob - including implicit collections. When set, content layer collections are forbidden in src/content, and will fail a build if defined.

There is a WIP implementation of this proposal in withastro/astro#11976

Previous proposals

This discussion is about several related issues around how to handle backward compatibility.

Motivations:

• Disallowing content layer collections in src/content is confusing. We should find a way allow it.
• Implementing legacy collections using the content layer would improve performance, particularly for markdown

Problems:

1. Legacy collections are defined implicitly for any folder in src/content, even if they're not defined in config.ts
2. Legacy collections are created in a fundamentally different way, so sites may behave unexpectedly
3. Data shape is different: slug field is not generated, and id is the slug not the filename.
4. Types are different: there's no auto-complete for each individual entry
5. Query API is different: getEntryBySlug is not supported.
6. Render API is different: It is render(entry) not entry(render)

Possible solutions:

1. scanning the src/content directory and generating collections for each directory does work (tried in PoC), though there's no clean way to exclude directories that are already used in a collection
2. test suites should catch these
3. we could add the slug field to legacy collections and use the file as the id. Handling this via the schema is tough, because of all the ways zod schemas can be defined, so this may need to be handled elsewhere - though this means loader-specific logic outside of the loader which is a bit ick. I did quite a bit of PoC work on doing this via the schema and there are lots of edge cases but it is probably doable.
4. Add types for each entry. Maybe only if collections are small, or just live without it.
5. getEntryBySlug is already deprecated in 4.x, so removing support should be fine
6. we can add the method to the objects without too much trouble.

The biggest problem is the conflicts in 1. There are no clean ways to determine if a folder is meant to use legacy collections if we allow new collections in src/content too.

Options:

1. We keep it like now: no content layer in src/content, and legacy collections handled with old code
2. We put legacy collection support behind a flag. If legacy behaviour is enabled then we make it behave like in experimental (no content layer in src/content, and legacy collections handled with old code), otherwise content layer can use src/content, and no legacy collections are parsed.
3. We remove support for auto-generated collections: we only allow legacy collections if they're defined in config.ts, and then emulate those using glob.

[matthewp]

I'd favor (2), putting legacy support behind a flag, along with a good migration path (mostly docs).

[sarah11918]

I favour option 1 during 5.0 beta, with option 2 being viable for 5.0 stable.

Reminder that historically, it's only been a small percentage of people who have run on beta versions. Many prefer to wait until stable versions, and the ones who do opt in to a beta version are typically the ones who are cool jumping through a few extra hoops to get it working. I would not want to offload the extra friction to existing projects/users!

If, as I was kind of thinking/hoping, this is solved for 5 stable, then people with existing collections in src/content/ only need to update a schema. (Fair, reasonable, not that hard.) Older content that we're NOT yet updating in beta docs (e.g. all our CMS guides that give examples using src/content/ don't have to change because that will be possible then.

So, my preference depends on when you think content layer will be compatible with src/content/

[ascorbic]

What do you mean by "compatible" in "when you think content layer will be compatible with src/content/", and what do you mean by solved in "this is solved for 5 stable"?

[sarah11918]

Sorry, my question was to clarify whether this is a proposal for beta launch next week (which would have me choosing option 1 given the chaos I expect and our ability message it quickly enough), but that for a stable release, Option 2 would be fine with me!

The way I have beta docs oriented now, we're referring to old collections as "legacy." We do have precedent for legacy flags, so I'd say a legacy flag works well from a docs standpoint, because we can shove the differences in legacy.content much like we document features in experimental.contentLayer right now.

[sarah11918]

Will also note with the beta docs, we have a LOT of existing e.g. CMS guides written and submitted by community members that expects your collections are in src/content/ (but don't explain how to set that up -- are just usage guides for other things where you have a local collection already).

Our current strategy in the beta docs, which people aren't going to read more than the super new stuff and upgrade guide anyway, we figure, is to not update all these examples, with the assumption that in stable 5 release, this restriction will be gone and it will be fine to refer to "collections in src/content/ again, creating an eventual consistency.

So we care more about the fact that the restriction will be gone ("backwards compatibility") to prevent all these guides from being out of date, than we care about how both collections are handled simultaneously for a time. Whatever you happen to choose would be less of a docs chore for us than checking and updating all that existing material that would now only work for legacy collections. 😄

But since we're calling them legacy collections anyway, and we do have a section in docs for Legacy flags (of which we currently have none), that seems like a good strategy for us!

[delucis]

I guess content layer makes it kind of impossible to support implicit collections like we currently have, right? (Because there’s no way to know if src/content/something/ is used in an existing glob collection etc. or not.) I’ll admit it was a nice feature for people to be able to just drop folders into src/content/ and not need any config — made for a very smooth onboarding of opt-in layers.

Part of me wishes we could keep offering more or less the current behaviour with only the entry interface changes, so that the migration path for a user was focused entirely on internal code (slug => id, entry.render() => render(entry)) without needing to change config, i.e. a collection without a loader is treated as an implicit glob loader for the directory matching its name.

Not sure if this overlaps with the points you’ve explored @ascorbic, but could we also in theory enable the legacy behaviour on a per-collection basis while still dropping the implicit collection behaviour? i.e. something like

import { defineCollection } from 'astro:content';
import { legacyLoader } from 'astro:loaders';
import { z } from 'astro/zod';

export const collections = {
  docs: defineCollection({
    loader: legacyLoader('docs'),
    schema: z.object({ ... }),
  }),
};

That would make gradual migrations a bit easier rather than forcing into a single on/off flag.

[ascorbic]

The difference from creating duplicates with glob is that it's a choice you're making, rather than magic happening in the background without you realising.

[ascorbic]

I should add @delucis, your suggestion there is pretty close to option 3. It wouldn't need them to manually use a legacy loader though: we could pretty easily just inject one for collections that don't have a loader and match a folder name in src/content. The problem with this though is that emulating legacy collections with glob() risks being almost but not exactly the same, because the implementation is very different under the hood. We'd run the risk of creating a backward compatibility mode that isn't fully compatible.

[delucis]

Right, close to option 3 but presumably with a legacyLoader() that smooths out the differences. Although I guess technically that might be tricky as I’m guessing custom loaders can’t do stuff like add a render() method to the entry or reintroduce slug/id properties as used previously — that would need to happen internally with a loader doing something like setting a secret mode: 'legacy' flag or something.

[ascorbic]

Yeah, it would need to be something internal, because it's doing things that loaders don't have access to

[ascorbic]

Rather than having a legacy loader, we could just say you have to create a collection with type: "content" or "data". My withastro/astro#11976 PR handles this, though with the implicit collections too. I'm quite happy with how it does it now, so I'm leaning more to option 3, with 2 behind a flag:

• by default, implicit collections are disabled, but any "data" and "content" types defined in config.ts are emulated with glob (I think I have this working now).
• there's a legacy mode flag which enables the old implicit collections, which are handled using the old code - which is also used for any explicit legacy collections. This mode also disables support for content layer in src/content.

[lloydjatkinson]

Can you reconsider exporting RenderResult again?

error ts(2305): Module '"astro:content"' has no exported member 'RenderResult'.

import type { RenderResult } from 'astro:content';

[ascorbic]

Good point. Done in withastro/astro#11974

[ascorbic]

I have a prototype of emulating current behaviour using glob. See withastro/astro#11976

[ascorbic]

I've updated this with my proposal, and moved the other ideas behind a details tag

[matthewp]

All looks good to me.