-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot use a custom Endpoint with SQS #39706
Labels
Team:obs-ds-hosted-services
Label for the Observability Hosted Services team
Comments
strawgate
changed the title
Cannot use a custom Endpoint with SQS
[Draft] Cannot use a custom Endpoint with SQS
May 23, 2024
botelastic
bot
added
the
needs_team
Indicates that the issue/PR needs a Team:* label
label
May 23, 2024
strawgate
changed the title
[Draft] Cannot use a custom Endpoint with SQS
Cannot use a custom Endpoint with SQS
May 24, 2024
13 tasks
proposed PR #39709 |
13 tasks
This was referenced May 24, 2024
cmacknz
added
the
Team:obs-ds-hosted-services
Label for the Observability Hosted Services team
label
May 24, 2024
Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services) |
botelastic
bot
removed
the
needs_team
Indicates that the issue/PR needs a Team:* label
label
May 24, 2024
@pierrehilbert @strawgate With #39709 merged, can this issue be closed now? |
Could be considered as a duplicate taking into account that @strawgate opened another issue for that: #40792 |
Yes this can be closed now in favor of the issue @pierrehilbert referenced |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Specifying a custom Endpoint for SQS does not work.
With the introduction of #36208 we started creating an endpoint resolver that causes every sdk request, regardless of S3 or SQS, to go to
endpoint
. There's a series of bugs related to this. Investigation done on 8.12.1 and Main.TL;DR : When an endpoint is specified, we ask the user to provide the S3 endpoint but this endpoint actually gets applied to all SDK calls and not just S3 calls. This is not a problem when S3 bucket polling but breaks SQS -> S3: https://github.com/elastic/beats/pull/36208/files#diff-589195ecff44ed2c6bf097da676822765ae86c2e73fc5789aaf0a64d8bb26ff1R73-R82
Here we go. First, a couple bugs that got us into this mess:
First, we load the region from the provided credential (not from the beats config)
And then we never insert the actual region provided by the beats config.
When the user specifies a queue url that lives on a custom endpoint, we cannot parse the region from these domains. Our region resolution then returns an empty string and overwrites our region with this invalid value:
This was first introduced in #36034 but this was fixed by @faec about 3 weeks ago: #39327
This shouldn't matter as when we have no region set, the SDK will use the default region from the endpoint. However, we have another bug that causes the default region to be ignored.
We end up ignoring the default_region entirely because we call getRegionFromQueueURL and pass the region to the third parameter, instead of the default region:
This bug was introduced in #36034. This is fixed on Main by @faec, three weeks ago #38958
This bug covers up another bug where parsing the queue_url can work just fine but it will parse to a region that doesn't match the default_region which will cause a warning to print. If there is no region configured at all then we hit an issue where we exit with
failed to get AWS region from queue_url
. This should print the warning on the next line not exitNext, there is a part of getRegionFromQueueURL that is meant to support pulling the region from
endpoint
but it also does not work due to our custom endpoint having a schema (https://) while our parsed hostname has no schema. This means we can never pull the region from theendpoint
field.This bug was introduced with this PR #24861 and is still an issue on main:
beats/x-pack/filebeat/input/awss3/sqs.go
Lines 44 to 57 in 809e001
So when these 8.12.1 bugs are combined: s3sqs input does not honor the value in
default_region
and only the value inregion
is considered. If the user supplied anendpoint
, it's almost certainly in a format we can't parse, so when we try to get the region from the URL, we ruin the region value. This broken region ensures that default endpoint resolution will fail.So, in this PR #36208 when we wanted to support custom endpoints for local testing, we applied an EndpointResolver to the S3 input to work around the above unknown issues, but that EndpointResolver applied to all calls, which works around all of the above issues as an Endpoint Resolver returns an exact URL for the service so the missing region doesn't matter anymore. This has the side effect of forcing everything to a single URL and then our SQS call ends up getting directed to the the S3 endpoint.
Now, instead of getting S3 bucket notifications from the SQS Queue URL specified by the user, the EndpointResolver, when called by the SQS service, replaces the entire hostname with the value in
endpoint
resulting in the SQS receive message going tohttps://s3.us-east-1.amazonaws.com/.....
. :So when these bugs are combined: s3sqs input does not honor the value in
default_region
and only the value inregion
is considered. If the user supplied anendpoint
, it's almost certainly in a format we can't parse, so when we try to get the region from the URL, we ruin theregion
value. This broken region ensures that default endpoint resolution will fail. We then applied an EndpointResolver to the S3 input to work around these issues, but that EndpointResolver applied to all calls, which works around all of the above issues for customizing the s3 endpoint, but has the side effect of forcing everything to a single URL and then our SQS call ends up going to the S3 endpoint.Option 1 - Fix overwritting region with a bad value, support default_region, and fix the Endpoint Resolver,
So to fix custom endpoint usage with SQS we need to scope the endpointresolver to only return an sdkendpoint when we're resolving an endpoint for the S3 service and when the endpoint field in the config is not empty. When a non-S3 service wants an endpoint we need the EndpointResolver to return
return awssdk.Endpoint{}, &awssdk.EndpointNotFoundError{}
so that the default endpoint resolution can occur (which is what will actually swap the S3 with SQS when called by the SQS service.Required:
Optional:
Option 2 - Fix parsing custom endpoints and remove the Endpoint Resolver
We remove the Endpoint Resolver and fix the other bugs that caused it to be necessary. This is a more invasive change.
Required work on Main:
Option:
Keeping the current behavior:
This means we cannot use custom domains on AWS with the AWS SDK. The functionality we implemented is basically BaseEndpoint instead of Endpoint. The right way to implement this behavior would probably be something like:
The text was updated successfully, but these errors were encountered: