storage controller: make proxying of GETs to pageservers more robust

[jcsp]

Problem

These commits are split off from https://github.com/neondatabase/neon/pull/8971/commits where I was fixing this to make a better scale test pass -- Vlad also independently recognized these issues with cloudbench in #9062.

1. The storage controller proxies GET requests to pageservers based on their intent, not the ground truth of where they're really attached.
2. Proxied requests can race with scheduling to tenants, resulting in 404 responses if the request hits the wrong pageserver.

Closes: #9062

Summary of changes

1. If a shard has a running reconciler, then use the database generation_pageserver to decide who to proxy the request to
2. If such a request gets a 404 response and its scheduled node has changed since the request was dispatched.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

4968 tests run: 4804 passed, 0 failed, 164 skipped (full report)

---

Flaky tests (8)

Postgres 17

• test_scrubber_physical_gc[4]: debug-x86-64, release-arm64
• test_subscriber_restart: release-x86-64

Postgres 16

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64
• test_scrubber_physical_gc[4]: release-x86-64

Postgres 14

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64
• test_scrubber_physical_gc[4]: release-x86-64
• test_restart_endpoint_after_switch_wal: release-x86-64

Code coverage* (full report)

• functions: 31.8% (7420 of 23301 functions)
• lines: 49.8% (59726 of 119930 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
4323bdc at 2024-09-19T17:52:22.111Z :recycle:}