Skip to content
nautilus nautilus nautilus

Verified: Redundancy and retained state

Each claim the Redundancy and retained state page makes, with the tests that prove it in the last green CI run: 12 of 27 verified, 4 with no test yet. The claims live in docs/claims/redundancy.yaml.

Last green CI run 16 Oct 2026, 14:48 UTC on c7618d3 · all runtime claims

RED-001 partly verified

Retained values are applied before the first scan and win over the manifest's init seeds.

where the docs say so · Win-over-init is asserted. "Before the first scan" is not: the test reads SP after the first scan with a program that never reads SP, so a load that landed after the first scan's logic would pass too.

  • pass TestRetainedValueLoadsBeforeFirstScan in runtime

RED-002 verified

Every `role: setpoint` tag is saved to the retain store, whatever its shape: a scalar, a struct, an array of scalars, an array of structs, or a multi-dimensional array.

where the docs say so · The restart and failover tests save and reload one setpoint of every kind the store takes: BOOL, REAL, TIME, STRING and enumeration scalars, a struct, and arrays of REAL, DINT, BOOL, TIME, STRING, a struct with an enumerated member, and a two-dimensional INT array, through the file store (restart) and the ConfigMap store (failover). Each element comes back at its declared kind. The skip test asserts both manifest routes to an ARRAY OF TON setpoint are refused, then (writing the values into the store directly, as no project can) that an array of FB instances and a struct with an FB member are left out, the rest of the state still saves, and each is logged once across two saver ticks: defense in depth.

  • pass TestRetainStateContents in runtime
  • pass TestRetainStatePersistsStructTag in runtime
  • pass TestRetainedArraySetpointsSurviveRestart in runtime
  • pass TestRetainedArraySetpointsSurviveFailover in runtime
  • pass TestRetainSkipsArrayOfFunctionBlocks in runtime

RED-003 verified

An integer-typed setpoint tag is restored with its retained value.

where the docs say so

  • pass TestLoadRestoresIntegerKind in runtime

RED-004 verified

A struct-typed setpoint tag is saved and restored with its retained member values.

where the docs say so

  • pass TestRetainStatePersistsStructTag in runtime
  • pass TestRetainedStructTagRestoresBeforeFirstScan in runtime

RED-005 verified

The source of an online-edited program is retained and run after a restart, still shows as modified so naut pull sees it, and a never-edited program is not persisted.

where the docs say so

  • pass TestRetainedProgramApplies in runtime
  • pass TestRetainStateContents in runtime

RED-029 verified

A deploy wins: an online edit is reloaded only by the same build it was made on; a build whose program for that task differs replaces the edit and names it in retainDiscardedEdits, keeping the last replaced source per task under discardedPrograms (never applied), and a build that already contains the edit clears it from the store.

where the docs say so

  • pass TestOnlineEditSurvivesARestartOfTheSameBuild in runtime
  • pass TestNewDeployReplacesARetainedOnlineEdit in runtime
  • pass TestBuildContainingTheEditClearsTheRetainedEntry in runtime
  • pass TestTakeoverAppliesOnlyASameBuildEdit in runtime

RED-006 partly verified

Inputs, state and outputs are not retained, and the PID integral is not persisted (it warm-starts from live field values).

where the docs say so · A state tag is excluded from what is saved and ignored on load even if the store names it. Inputs, outputs and the PID warm start are not asserted.

  • pass TestRetainStateContents in runtime
  • pass TestLoadIgnoresTagsOutsideTheRetainedSet in runtime

RED-007 partly verified

State is saved only when something changed, only by the leader, and a failed save is retried with the same content on the next tick; a standby never saves.

where the docs say so · The 2-second save interval and the 2-second worst-case loss bound are not asserted.

  • pass TestSaverWritesOnChangeAndRetriesOnFailure in runtime
  • pass TestStandbyNeverSaves in runtime

RED-008 partly verified

Store failures surface in diagnostics as retainErrors and lastRetainError.

where the docs say so · A failed save is asserted to bump retainErrors; lastRetainError after a store failure is not asserted, nor is a failed load. (The broken-program test, which does check lastRetainError, is not a store failure and was removed.)

  • pass TestSaverWritesOnChangeAndRetriesOnFailure in runtime

RED-010 partly verified

Outside a cluster, retained state is saved to the file named by retain.file and loads back from it.

where the docs say so · The round trip is tested on a fileStore built directly; retain.New choosing the file store outside a cluster, and the retain.file wiring, are not exercised.

  • pass TestFileStoreSaveLoadRoundTrip in retain

RED-011 partly verified

In a cluster, retained state is saved to the named ConfigMap (created if absent) and loads back from it.

where the docs say so · Create-if-absent and the round trip are asserted on a ConfigMap store built directly; retain.New choosing the ConfigMap store in a cluster, and the retain.configmap wiring, are not exercised.

  • pass TestConfigMapStoreSaveCreatesWhenAbsent in retain
  • pass TestConfigMapStoreSaveUpdatesWhenPresent in retain
  • pass TestConfigMapStoreRoundTrip in retain

RED-012 no test yet

The defaults hold: retain file retain.json outside a cluster and configmap named after the project with -retain suffix in a cluster; and redundancy lease defaults to the project name.

where the docs say so · No test asserts the default names.

RED-013 partly verified

The leader scans; a standby does nothing at all: no field I/O, no logic, no output writes.

where the docs say so · Asserted: a standby runs no scans (stats count 0) and no logic, and a leader scans. No field I/O and no output writes to a driver are not asserted (the test project has no drivers).

  • pass TestStandbyDoesNotScan in runtime
  • pass TestTakeoverReloadsAndResetsFrames in runtime

RED-014 verified

Replicas elect exactly one leader through a Lease: the first replica acquires it, keeps it by renewing, and a standby takes it only once it expires; two replicas racing for an expired lease cannot both win.

where the docs say so · RED-015 (the racing case) merged here.

  • pass TestFreshClusterAcquire in leader
  • pass TestRenew in leader
  • pass TestFollowerThenSteal in leader
  • pass TestCASRace in leader

RED-016 verified

Failover is bounded by the lease: a leader that cannot renew steps down once the lease duration elapses, not before.

where the docs say so

RED-017 no test yet

Failover takes at most 4 s worst case, and a standby ticks every 1 s.

where the docs say so · Leader tests use a shortened lease duration with a synthetic clock; the 4 s bound and 1 s tick defaults are not asserted.

RED-018 partly verified

A clean shutdown (SIGTERM) hands the lease over so a standby takes it on its next tick rather than waiting out the lease.

where the docs say so · Release is asserted to drop leadership and let the follower acquire on its next tick; that a SIGTERM shutdown calls Release (cmd/naut defer) is not tested.

RED-019 verified

On takeover the new leader re-reads the retain store (picking up retunes the old leader accepted) and resets program frames so warm-start logic runs against live values.

where the docs say so

  • pass TestTakeoverReloadsAndResetsFrames in runtime

RED-020 no test yet

On takeover the scan clock is zeroed so the first dt is one scan, not the time spent standing by.

where the docs say so · No test asserts the first dt after a takeover.

RED-021 partly verified

Outside a cluster redundancy degrades gracefully: the controller runs standalone and is always the leader.

where the docs say so · The standalone elector is tested; nothing shows the runtime scanning with a redundancy: section outside a cluster.

  • pass TestStandalone in leader

RED-022 partly verified

A standby reverse-proxies API traffic to the leader, whose address rides inside the Lease.

where the docs say so · Proxying is asserted for GET /api/state and POST /api/forces with a fake cluster status that hands the server the leader address; that a standby learns that address from the Lease is not asserted.

  • pass TestStandbyProxiesToLeader in server
  • pass TestForceProxiedFromStandby in server

RED-023 verified

Every replica serves the dashboard itself, and a standby answers /api/cluster with its own view rather than the leader's.

where the docs say so · TestStandbyWithNoLeaderServesLocally removed: serving locally with no known leader is not a promise of the page.

  • pass TestStandbyAnswersClusterAndIndexLocally in server

RED-024 verified

GET /api/cluster reports each replica's own view: mode, leader and leader transitions.

where the docs say so

  • pass TestClusterEndpointReportsStatus in server
  • pass TestFreshClusterAcquire in leader

RED-025 no test yet

Identity in /api/cluster comes from POD_NAME and the proxy target from POD_IP.

where the docs say so · No test sets POD_NAME or POD_IP.

RED-026 partly verified

The Lease and ConfigMap are spoken to directly with only get, create and update verbs (no delete); a releasing leader expires its lease by writing an epoch renew time.

where the docs say so · Release makes the lease immediately takeable; neither the epoch renew time nor the verbs used (no delete) are asserted against a recorded request log.

RED-027 verified

A different election scheme plugs in through runtime.Coordinator, which is the single method IsLeader.

where the docs say so · Both tests drive the runtime with a hand-written flag Coordinator, which exercises the seam.

  • pass TestStandbyDoesNotScan in runtime
  • pass TestTakeoverReloadsAndResetsFrames in runtime

RED-028 verified

A retained struct or array is restored onto the current tag: an array the project has since grown keeps its seeds for the new elements, and one it has shrunk keeps its seeds and reports the mismatch in lastRetainError.

where the docs say so

  • pass TestRetainedArrayLengthChange in runtime
  • pass TestRetainedStructTagRestoresBeforeFirstScan in runtime