KEP-4832: Asynchronous preemption in the scheduler

[sanposhiho]

• One-line PR description: Add KEP-4832.

• Issue link: Asynchronous preemption in the scheduler #4832

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sanposhiho
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sanposhiho]

/cc @alculquicondor @Huang-Wei @mengqiy

[sanposhiho]

/cc @hakuna-matatah

[alculquicondor]

cc @macsko @dom4ha

[sanposhiho]

cc @wojtek-t

🙏

[dom4ha]

Moving the discussion from kubernetes/kubernetes#125491 (comment). I'm mainly concerned that making preemption API calls async will be replaced with setting nominatedNodeName, which IIUC is also an API call (the reservation has to be visible for autoscaler, so cannot be just within the scheduler cache, which I think is assumed in this KEP).

Is it a valid concern? If yes, then calling AddNominatedPod (setting nominatedNodeName) async as well maybe could help. I'm not suggesting adding new extension point here though.

[sanposhiho]

making preemption API calls async will be replaced with setting nominatedNodeName, which IIUC is also an API call

Setting nominatedNodeName requires API call, but we won't do that at PostFilter. We will only do AddNominatedPod, which doesn't make any API calls since it just modifies the cache. And any actual API calls for the pod deletion and setting nominatedNodeName will be done asynchronously. That's what this KEP proposes.

the reservation has to be visible for autoscaler, so cannot be just within the scheduler cache

The reservation has to be visible once the preemption (= actual pod deletion) is done because otherwise CA would make incorrect decision with the preemption target nodes, which is likely low-utilized just after the preemption.
In other words, not need until then, I believe.