server: stop NoTransitionException warning spam for fenced hosts in HA manager#13588
Open
nikolauseppinger wants to merge 1 commit into
Open
Conversation
…A manager After the Host HA framework successfully fences a host, FenceTask puts it into maintenance mode, which makes the host ineligible for the HA provider. The background poll task then attempted an Ineligible transition on every poll round, but the HA state machine intentionally has no Fenced -> Ineligible transition (a fenced host must remain Fenced until a health check passes). The result was an endless stream of warnings: WARN [o.a.c.h.HAManagerImpl] Unable to find next HA state for current HA state=[Fenced] for event=[Ineligible] ... com.cloud.utils.fsm.NoTransitionException Skip the Ineligible transition attempt for hosts in Fenced state, since their ineligibility is expected and the intended exit from Fenced is Fenced -> HealthCheckPassed -> Ineligible. No behavioural change otherwise: the host is still skipped by the poll task while fenced and ineligible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
After the Host HA framework successfully fences a host,
FenceTaskputs itinto maintenance mode, which makes the host ineligible for the HA provider
(
KVMHAProvider.isEligible()returns false for hosts in maintenance).The
HAManagerBgPollTaskthen attempts anIneligibletransition on everypoll round via
validateAndFindHAProvider(), but the HA state machineintentionally has no
Fenced -> Ineligibletransition — a fenced host mustremain
Fenceduntil a health check passes(
Fenced -> HealthCheckPassed -> Ineligible).The result is an endless stream of warnings, every poll round, for every
fenced host, until the operator resolves the host:
This PR skips the
Ineligibletransition attempt for hosts inFencedstate, since their ineligibility is expected (fencing puts them in
maintenance) and the intended exit from
Fencedis via a passing healthcheck. There is no behavioural change otherwise: the host is still skipped
by the poll task while fenced and ineligible.
Deliberately NOT added: an FSM transition
Fenced -> IneligibleviaEvent.Ineligible. That would make theFencedstate effectively last onlyuntil the next poll round (fencing always puts the host in maintenance →
always ineligible), breaking the sticky-Fenced semantics that
isVMAliveOnHost()andgetHostStatusFromHAConfig()rely on while theVM HA restart work items for the fenced host are being processed.
Covered by new unit tests (fenced case plus regression cases for the
existing eligibility logic).
Observed on 4.22.1.0 with KVM Host HA + IPMI OOBM; code path unchanged on main.
Types of changes
Bug Severity
How Has This Been Tested?
HAManagerImplTest(4/4 pass)mvn -pl server -am testbuilds green including checkstyle