Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: adding "since" annotation to k8s nodes #7883

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

MenD32
Copy link

@MenD32 MenD32 commented Mar 1, 2025

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixes bug where upon deployment restart, cluster-autoscaler resets the the node scaledown timer, which can cause nodes to never scale in extreme cases.

Which issue(s) this PR fixes:

Fixes #7873

Special notes for your reviewer:

this issue has already affected me, and I'm highly motivated to fix this ASAP. so I should be available during the next few days, in order to resolve this quickly. if you have a better way of fixing this HA issue, please suggest one 😄

Does this PR introduce a user-facing change?

Fixed: Cluster Autoscaler now persists scale-down cooldowns across pods. cooldown timestamps were stored in memory, causing inconsistencies when the original cluster autoscaler pod becomes unavailable. Now, cooldown state is stored as node annotations:

"cluster-autoscaler.kubernetes.io/unneeded" ("true"/"false")
"cluster-autoscaler.kubernetes.io/unneeded-since" (RFC3339 timestamp)

This ensures consistent scale-down behavior even if the original pod becomes unavailable.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


Signed-off-by: MenD32 <amit.mendelevitch@gmail.com>
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 1, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @MenD32. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. area/cluster-autoscaler labels Mar 1, 2025
@k8s-ci-robot k8s-ci-robot requested a review from feiskyer March 1, 2025 22:20
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: MenD32
Once this PR has been reviewed and has the lgtm label, please assign feiskyer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested a review from vadasambar March 1, 2025 22:20
MenD32 added 4 commits March 2, 2025 15:28
Signed-off-by: MenD32 <amit.mendelevitch@gmail.com>
Signed-off-by: MenD32 <amit.mendelevitch@gmail.com>
Signed-off-by: MenD32 <amit.mendelevitch@gmail.com>
…32/autoscaler into feat/node-cooldown-timer-annotation
@k8s-ci-robot k8s-ci-robot added area/provider/aws Issues or PRs related to aws provider size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 2, 2025
Signed-off-by: MenD32 <amit.mendelevitch@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cluster-autoscaler area/provider/aws Issues or PRs related to aws provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cluster Autoscaler gets OOMKilled instead of scaling down
2 participants