Memory spikes when (re)loading nginx configuration #6428
dbaumgarten
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
whenever nginx reload's it's configuration we see quite a large spike in memory-consumption for the pod.
The more reloads happen in a short timeframe the larger that spike is.
I understand that is because of the way such a reload works (new worker processes, draining of existing processes etc).
However I am a little surprised by the dimensions of that increase.
Here is a screenshot of the memory-consumption of nginx pods when they are beeing replaced by other pods via a rolling update.
(The behavior is very similar when just a config-reload is peformed)
As you can see the memory-usage of a pods spkies from <2GB to around ~10GB. A 5-fold increase.
Is such a large increase really normal? Is there something going wrong?
Given that reloads might happen when nginx is under high load (and therefore autoscaling increases the number of nginx and backend pods) that can become an issue.
High Load -> High Resource Usage -> Autoscaling triggers -> New Pods are created -> Config reload because of new pods -> Even higher Resource Usage because of the reloads.
Currently we have solved the issue by simply setting very high memory requests and limits (12GB).
But setting a 12 GB request for a pod that usually uses ~3GB of memory just feels wrong.
Beta Was this translation helpful? Give feedback.
All reactions