Use new conditional writes API on S3 for user-configurable overrides

[kvrhdn]

Is your feature request related to a problem? Please describe.

The user-configurable overrides API allows clients to write and update a subset of Tempo overrides. The overrides are stored in a bucket (separate from the traces). See also Tempo - user-configurable overrides.

User-configurable overrides are versioned and when a client wishes to update them they should supply the current version to avoid concurrent writes. GCS and Azure blob storage support this capability but AWS S3 did not so far, leading to a best-effort implementation:

tempo/tempodb/backend/s3/s3.go

Lines 470 to 473 in 64fd38e

	func (rw *readerWriter) WriteVersioned(ctx context.Context, name string, keypath backend.KeyPath, data io.Reader, version backend.Version) (backend.Version, error) {
	// Note there is a potential data race here because S3 does not support conditional headers. If
	// another process writes to the same object in between ReadVersioned and Write its changes will
	// be overwritten.

Describe the solution you'd like

S3 has announced they now support conditional writes. Docs on adding preconditions.

We should extend our S3 backend implementation and make use of the new preconditions.

⚠ Not all S3-compatible backends might support preconditions. We should take some time to investigate what they are planning to do and ensure those will still work.

Describe alternatives you've considered

We can keep using our best-effort implementation, since volume is low this is unlikely to cause a lot of issues.

Additional context