Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add abitility to set QPS and Burst limits for api client #2667

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

Demch1k
Copy link

@Demch1k Demch1k commented Jun 18, 2024

We have a problem in our big kubernetes clusters with 300+ postgres clusters.
When operator starts reconcile it can stack creating new clusters for 10-15 minutes and We can see in the logs messages like that:

I0618 03:13:56.195420 1 request.go:697] Waited for 17.19394453s due to client-side throttling, not priority and fairness, request: POST:https://10.141.64.1:443/apis/apps/v1/namespaces/postgres-ns/statefulsets

Especially in working hours when our developers create new environments.
This PR adds ability to set kubernetes qps and burst limits for client.

@Demch1k
Copy link
Author

Demch1k commented Jun 19, 2024

Any updates ?

cmd/main.go Outdated Show resolved Hide resolved
@FxKu
Copy link
Member

FxKu commented Jun 26, 2024

Can you not make these variables also configurable similar to NoDatabaseAccess and NoTeamsAPI. We can use 5 and 10 as default values.

@FxKu FxKu added this to the 1.14.0 milestone Jun 26, 2024
@Demch1k Demch1k requested a review from FxKu July 8, 2024 15:42
@Demch1k
Copy link
Author

Demch1k commented Jul 9, 2024

@FxKu I have reverted my previous changes because we can't change qps and burst limits for kube client after initialization.
Without kube client we can't obtain crd with limits. Only one way to pass it is used cli args.

@FxKu
Copy link
Member

FxKu commented Jul 22, 2024

Config changes always require a replacement of the operator pod. Doesn't this initialize the client as well each time?

@Demch1k
Copy link
Author

Demch1k commented Jul 26, 2024

@FxKu I mean that you can't change kubernetes client parameters after client has been initilized.
To be able to read it from operator configs you need initilize kubernetes client first and read custom resource. After that you must reinitilize kubernetes client to apply new values like QPS or Burst but I don't think that is possible. If you know how to do it, help me please with it.

@Demch1k
Copy link
Author

Demch1k commented Aug 6, 2024

@FxKu We have been testing this fix in our k8s cluster with 200+ postgres clusters for 2 weeks now and it really helps.

@FxKu FxKu added the minor label Aug 23, 2024
@Demch1k
Copy link
Author

Demch1k commented Sep 9, 2024

@FxKu any updates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Open Questions
Development

Successfully merging this pull request may close these issues.

2 participants