add support parallel builds

[chaegumi]

Changes

• add support parallel builds

Testing

When greater than 1 using parallel builds during the build.

{
  build: {
    concurrency: 1
  }
}

pnpm run build

Docs

add a configuration options concurrency in build options

[changeset-bot]

🦋 Changeset detected

Latest commit: 3e26b4c

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[ematipico]

At the beginning I thought you were implementing the solution using worker threads, but your solution stil uses a main thread. This solution feels more like a queue instead. Is my analysis incorrect?

[matthewp]

Thanks, as we're in the middle of releasing Astro 5 beta we likely won't get to this right away but we'll circle back in a couple of weeks and take a look.

Note that we have had a few attempts to bring this in, but it never quite worked out. Might look at old PRs to see what went wrong in the past. Also, generating pages isn't usually the slow part of the builds, so would love to see some examples of where this has big benefits. cc @bluwy who might have thoughts.

[chaegumi]

My code writing may not be optimal, but this change is really important to me, and I believe it will also be beneficial to Astro. My project is connected to the CMS API, with more than 9,000 URLs. The current Astro build queues one build before generating the next one. My modification allows the setting to generate 10 or more URLs at the same time. Below is a screenshot of my project running results.

Before:

12:40:24 [@astrojs/sitemap] sitemap-index.xml created at dist
12:40:24 [build] 9377 page(s) built in 7158.37s
12:40:24 [build] Complete!
Done in 7289.08s.

After:

21:45:59 [build] Waiting for integration "@astrojs/sitemap", hook "astro:build:done"...
21:46:01 [@astrojs/sitemap] sitemap-index.xml created at dist
21:46:01 [build] 9372 page(s) built in 4003.43s
21:46:01 [build] Complete!
Done in 4063.53s.

[bluwy]

Yeah this seems to still run on the main thread, but what I understand is that it allows multiple fetch calls from different pages to be executed in parallel? Usually for cases like this, we suggest doing all the fetches beforehand (e.g. from a shared file, during build only), that way you're not blocked by the page rendering order and better cache things if needed.

Maybe that's the better approach here for your app? I think there's some value in a concurrency option, but it's hard to tell users what's the ideal value for it. If you set it too high, you can slow down the rendering process too.

[chaegumi]

Assuming I have 10 pages, from page 1 to page 10, is the current program generating page 2 after page 1 is completed, page 3 after page 2 is completed, until page 10 is generated.
My requirement is simple. I need to generate two pages at a time, such as page 1 and page 2, page 3 and page 4 (or the concurrency parameter I added). Is that okay?
So I used it https://lodash.com/docs/4.17.15#chunk , split the pages that need to be generated, and then use promise.all

[gacek1123]

@bluwy could we maybe get an experimental release for this?

I have an app that does all fetching beforehand and then renders pages from a shared cache of sorts so I could maybe see if this makes any difference.

It is quite a bit smaller (around 800 pages) but we should still be able to see if there's any improvement.

[chaegumi]

@gacek1123 you can clone and run test by yourself through pnpm link.

https://github.com/withastro/astro/blob/main/CONTRIBUTING.md#development

https://pnpm.io/cli/link#replace-an-installed-package-with-a-local-version-of-it

[bluwy]

My requirement is simple. I need to generate two pages at a time, such as page 1 and page 2, page 3 and page 4 (or the concurrency parameter I added). Is that okay?
So I used it lodash.com/docs/4.17.15#chunk , split the pages that need to be generated, and then use promise.all

I think using p-limit could be better so that we always generate two pages at a time, so that we don't have pages waiting for each chunks. Before making any changes to that though, I think the team still needs to discuss if this is something we want.

Like I mentioned, if you perform all the fetch calls beforehand, cache it, and share it between pages, does that help with the generation speed? That should make page generation even faster than having a concurrency option.

@bluwy could we maybe get an experimental release for this?

If you've not tested linking locally like @chaegumi mentioned, I can definitely cut a preview release for this. But it looks like the lockfile is borked as it's installed with pnpm v8, so i'm not sure publishing will pass. Would have to fix that first.

[chaegumi]

if you perform all the fetch calls beforehand, cache it, and share it between pages, does that help with the generation speed?

My program was designed to do this, and I started using it https://github.com/11ty/eleventy-fetch I did request caching, but when my program cached the .cache directory for a few hundred MB, it failed to start. Later, I switched to cacache. I am now using keyv and keyv-lru to cache my requests, and I have also done Incremental Static Regeneration (ISR), but none of these can solve my problem. I have no problem obtaining API data, but now I just want to control the number of pages generated simultaneously. One page takes 1 second, and I want to change it to generate 10 pages per second.

But it looks like the lockfile is borked as it's installed with pnpm v8, so i'm not sure publishing will pass. Would have to fix that first.

ok

[bluwy]

!snapshot build-concurrency

[bluwy]

It looks like preview releases doesn't work on forks :(

but none of these can solve my problem. I have no problem obtaining API data, but now I just want to control the number of pages generated simultaneously. One page takes 1 second, and I want to change it to generate 10 pages per second.

Ah so you've already optimized network fetching, that's good to know. I'm curious what's making page rendering slow still. Anyways I can't seem to cut a preview release, so we might have to wait for the rest of the team to discuss about this feature.

[chaegumi]

English is not my native language, so my expression may not be very accurate, causing your misunderstanding. My page rendering should not be slow, 200ms, 300ms, 500ms, 600ms, etc., and some may take about 1 second.
The current build program is one after another
I want multiple at once, don't queue up, because when generating, the order is not important

[chaegumi]

Just like we need to request 10 URLs at the same time and send 10 requests at the same time