Skip to content

server: stop NoTransitionException warning spam for fenced hosts in HA manager#13588

Open
nikolauseppinger wants to merge 1 commit into
apache:mainfrom
nikolauseppinger:fix/host-ha-fenced-ineligible-notransition
Open

server: stop NoTransitionException warning spam for fenced hosts in HA manager#13588
nikolauseppinger wants to merge 1 commit into
apache:mainfrom
nikolauseppinger:fix/host-ha-fenced-ineligible-notransition

Conversation

@nikolauseppinger

Copy link
Copy Markdown

Description

After the Host HA framework successfully fences a host, FenceTask puts it
into maintenance mode, which makes the host ineligible for the HA provider
(KVMHAProvider.isEligible() returns false for hosts in maintenance).

The HAManagerBgPollTask then attempts an Ineligible transition on every
poll round via validateAndFindHAProvider(), but the HA state machine
intentionally has no Fenced -> Ineligible transition — a fenced host must
remain Fenced until a health check passes
(Fenced -> HealthCheckPassed -> Ineligible).

The result is an endless stream of warnings, every poll round, for every
fenced host, until the operator resolves the host:

WARN [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:[ctx-...]) Unable to find
next HA state for current HA state=[Fenced] for event=[Ineligible] for host
Host {"id":10,...} with id 10. com.cloud.utils.fsm.NoTransitionException:
Unable to transition to a new state from Fenced via Ineligible

This PR skips the Ineligible transition attempt for hosts in Fenced
state, since their ineligibility is expected (fencing puts them in
maintenance) and the intended exit from Fenced is via a passing health
check. There is no behavioural change otherwise: the host is still skipped
by the poll task while fenced and ineligible.

Deliberately NOT added: an FSM transition Fenced -> Ineligible via
Event.Ineligible. That would make the Fenced state effectively last only
until the next poll round (fencing always puts the host in maintenance →
always ineligible), breaking the sticky-Fenced semantics that
isVMAliveOnHost() and getHostStatusFromHAConfig() rely on while the
VM HA restart work items for the fenced host are being processed.

Covered by new unit tests (fenced case plus regression cases for the
existing eligibility logic).

Observed on 4.22.1.0 with KVM Host HA + IPMI OOBM; code path unchanged on main.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Bug Severity

  • Minor

How Has This Been Tested?

  • New unit tests in HAManagerImplTest (4/4 pass)
  • mvn -pl server -am test builds green including checkstyle

…A manager

After the Host HA framework successfully fences a host, FenceTask puts it
into maintenance mode, which makes the host ineligible for the HA provider.
The background poll task then attempted an Ineligible transition on every
poll round, but the HA state machine intentionally has no Fenced ->
Ineligible transition (a fenced host must remain Fenced until a health
check passes). The result was an endless stream of warnings:

  WARN [o.a.c.h.HAManagerImpl] Unable to find next HA state for current
  HA state=[Fenced] for event=[Ineligible] ...
  com.cloud.utils.fsm.NoTransitionException

Skip the Ineligible transition attempt for hosts in Fenced state, since
their ineligibility is expected and the intended exit from Fenced is
Fenced -> HealthCheckPassed -> Ineligible. No behavioural change
otherwise: the host is still skipped by the poll task while fenced and
ineligible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant