Skip to content
nautilus nautilus nautilus

Verified: Testing (naut test)

Each claim the Testing (naut test) page makes, with the tests that prove it in the last green CI run: 8 of 32 verified, 4 with no test yet. The claims live in docs/claims/testing.yaml.

Last green CI run 16 Oct 2026, 14:48 UTC on c7618d3 · all runtime claims

TEST-001 partly verified

naut test runs the resource on a virtual clock that both the measured scan-to-scan dt (bound to dt-tag) and the TON/TOF/TP millisecond base follow, so advance: 10s runs ten seconds of process time exactly.

where the docs say so · TOF and TP on the virtual base are covered by the conformance corpus, not here.

TEST-002 verified

Runs are deterministic: the scheduler lands the clock exactly on each tick, so dt is exactly the configured period and a loop's trajectory is identical on every run.

where the docs say so

TEST-003 partly verified

The fixture is nautilus.yaml: tasks, scan rates, tag roles, seeds, units and descriptions come from the manifest, so a test never restates them.

where the docs say so · The suite runs against the manifest; a retuned manifest value flowing into the tests is not asserted.

TEST-004 verified

A test file that breaks the format's rules is refused: a step takes at most one of scans, advance or until, hold applies only with until, a test needs a name, and a test has either steps: or inline step keys, not both.

where the docs say so

  • pass TestSpecRejectsMistakes in acceptance (13 of 13 passed)

TEST-005 verified

A test with no steps: is one step, so that step's keys may be written inline on the test.

where the docs say so

TEST-006 partly verified

scans: N runs until the main task has completed N more scans.

where the docs say so · The Go scheduler's Scans(n) is exercised; the YAML `scans:` count against the main task specifically is not isolated.

TEST-007 partly verified

advance: d runs every due tick up to now+d and lands exactly there; a step with none of scans/advance/until asserts on the seeded state before anything runs.

where the docs say so · The omit-all-three-keys seeded-state assertion has no test, and no test reads the clock after an advance to show it lands exactly on now+d (the TON test only brackets the trip between 9.5 s and 10.5 s).

TEST-008 partly verified

until: runs up to the bound and stops as soon as expect holds; with hold: the expectation must stay true for that long, so a loop that merely passes through its target does not count as settled.

where the docs say so · No test shows an overshooting loop failing a hold: that a pass-through is rejected is untested.

TEST-009 verified

always: is checked after every scan in the step (an invariant, not an endpoint).

where the docs say so · Spike is TRUE on the third scan only: always: fails at 300 ms (that scan) while an end-of-step expect: passes, with scans: and with until:.

  • pass TestAlwaysCatchesAViolationThatRecovers in acceptance

TEST-010 partly verified

given: writes are routed by the role the manifest declared: an input goes to the driver's input image and arrives at the top of the next scan, everything else is written straight to the store, applied before the step's first scan.

where the docs say so · A non-input given visible with scans: 0 (written straight to the store, before the first scan) is asserted. No test shows an input given landing in the driver's input image and arriving only at the top of the next scan. (TestPumpSealInHysteresis was removed: it sets tags directly in Go and never goes through given:.)

  • pass TestGivenComposesAfterZeroScans in acceptance

TEST-011 no test yet

Writing an undeclared tag in given: is an error, not a new tag.

where the docs say so · Gap. Only force: of an unknown tag is asserted (undefined tag), not given:.

TEST-012 verified

A dotted name in given: addresses one field of a UDT tag and zero-fills the rest of that struct the first time a test touches it, and every dotted field of the same tag in one given map composes onto a single write, independent of key order.

where the docs say so

  • pass TestGivenSingleFieldStillZeroFillsTheRest in acceptance
  • pass TestGivenComposesTwoFieldsOfSameStructInOneMap in acceptance
  • pass TestGivenNestedFieldAndSiblingFieldCompose in acceptance
  • pass TestGivenScalarAndStructFieldInSameStep in acceptance

TEST-013 verified

An array element is [n] from the declared lower bound in given: and expect:, an enumeration member takes its name (or integer) at any depth, and a name that is no member fails the test with the members listed.

where the docs say so

  • pass TestGivenEnumMemberByPath in acceptance
  • pass TestGivenEnumNonMemberNamesMembers in acceptance

TEST-014 no test yet

Tests never open a socket: whatever driver: says (including eip), the harness substitutes a stub, so a project bound to hardware is testable offline.

where the docs say so · Gap. No test loads a project with a network driver and checks that no connection is attempted.

TEST-015 partly verified

Expectations take an exact value (BOOL, STRING or number) or a matcher: near with tol (defaulting to the file's tolerance), gt, ge, lt, le, between (inclusive) and eq.

where the docs say so · The matchers parse and the near/between forms run in the suite; the semantics of each comparison (ge vs gt boundaries, inclusive between, the tolerance default) are not asserted by a dedicated test.

TEST-016 partly verified

An ST expression string (or a list mixing mappings and strings) is compiled by the same compiler as the logic, can use tags and library FUNCTIONs, and a mapping is tag matchers while a string is an expression.

where the docs say so · Mapping vs string parsing, a mixed list, an expression over tags, and a false expression failing are asserted. No test calls a library FUNCTION from an expectation (TestTestLibDir was removed: the libdir fixture's suite has no expression expectations at all).

TEST-017 partly verified

Program locals are addressable as task.local in the matcher form (exact or with a matcher), resolution is tag-first then <tag>.<field> then <task>.<local> then an error naming both namespaces, and task.local is not a name an ST expression string can use.

where the docs say so · Only exact-value task-local matchers are asserted, never a comparison matcher (main.integral: { gt: 0.0 }). The resolution order is not asserted: every name in the tests resolves one way only; no case has a name that is both a tag (or tag field) and a task local.

  • pass TestTaskLocalStillResolves in acceptance
  • pass TestTaskLocalInListForm in acceptance
  • pass TestUnknownHeadNamesBothNamespaces in acceptance
  • pass TestTaskLocalNotResolvedInExpression in acceptance

TEST-018 no test yet

A scan that faults fails the test that ran it, with the error and the virtual timestamp, without the author writing an assertion.

where the docs say so · Gap. A failure with step 0 / line 0 for a faulted scan is described in the docs; no acceptance test provokes a faulting scan.

TEST-019 partly verified

alarms: asserts host alarm state (active, unacked, shelved, state, priority, journal) at the end of a step, and the ack, shelve and unshelve verbs in given: make the same calls as POST /api/alarms/*.

where the docs say so · Only active: is shown failing on a mismatch (TestAlarmFailureNamesWhatItSaw); unacked, shelved, state, priority and journal appear only in passing tests, so a check that ignored them would still pass. That the verbs make the same calls as POST /api/alarms/* is not asserted.

  • pass TestAlarmAckVerb in acceptance
  • pass TestAlarmShelveAndUnshelveVerbs in acceptance
  • pass TestAlarmOnDelayWalksInVirtualTime in acceptance
  • pass TestAlarmFailureNamesWhatItSaw in acceptance (7 of 7 passed)

TEST-020 verified

The alarm engine reads the runtime's virtual clock, so a five-minute on-delay is walked exactly and a fifteen-minute shelf expires under advance: 16m.

where the docs say so

TEST-021 verified

Using alarms: keys without a manifest alarms: section is an error rather than a quiet pass, and alarms: does not combine with until:.

where the docs say so

  • pass TestAlarmKeysWithoutAnEngineAreAnError in acceptance
  • pass TestAlarmsAndUntilAreRefused in acceptance

TEST-022 partly verified

Alarm state does not leak between tests, and WithAlarms builds the manifest's own engine over each test's runtime with an in-memory journal and no notifiers.

where the docs say so · The in-memory journal and absence of notifiers are not asserted.

  • pass TestAlarmStateDoesNotLeakBetweenTests in acceptance

TEST-023 partly verified

suspend: [task] stops the named tasks from scanning for the duration of a test (file default overridable per test), turning a closed loop into an open-loop test.

where the docs say so · The file-level suspend default and the per-test override are not asserted.

TEST-024 partly verified

force: puts tags in the controller's force table and they stay held against the driver's input image, the plant model and the logic until unforce: names them (or all); dotted member paths work as in given:; forces end with the test.

where the docs say so · Holding against the plant model and the logic, unforce by name and unforce: all are asserted. Holding against a driver's input image, dotted member paths in a force: step and forces ending with the test are not. (TestForceStepMistakes was removed with the claim's mistakes-are-errors clause, which the page does not make.)

  • pass TestForceHoldsAgainstPlantAndLogic in acceptance

TEST-025 partly verified

A failure reports the assertion that broke (file:line of the tag key, expression or alarms: key), the step and its first line, the virtual time, and a trajectory of the process values with their units and descriptions.

where the docs say so · Line and step line are asserted; the trajectory trace with unit and description is not.

  • pass TestFailureLinePointsAtTheAssertion in acceptance
  • pass TestFailureReportsBothLines in acceptance

TEST-026 partly verified

The -json event carries the test's line, passed, scans, elapsedMs and for failures step, line, stepLine, atMs, reason, detail and trace.

where the docs say so · Only the failure line/stepLine/detail fields are decoded; the other fields and the trace are not.

TEST-027 no test yet

naut test exits non-zero on any failure, and -run filters tests by name pattern.

where the docs say so · Gap. cmd/naut has no test for the test subcommand's exit code or -run flag.

TEST-028 partly verified

*_test.yaml files are discovered in the project, and never reach a deployed controller: naut build excludes them from the embedded project.

where the docs say so · Discovery is asserted; the exclusion from the built binary's archive is not (TestDiscovery's comment promises it but only checks discovery).

TEST-029 partly verified

An SDK (Go) project runs the same suites through the same harness (acceptance.Run with WithAlarms), and hand-written Go tests get the virtual clock through acceptance.NewRuntime and its scheduler's Advance, Scans, AdvanceUntil and Suspend.

where the docs say so · The YAML-from-Go acceptance.Run entry point is exercised through RunDir (TestHeatedTankSuite), not Run.

TEST-030 partly verified

The harness replays one deterministic interleaving (next task due, ties broken by declaration order).

where the docs say so · Reproducibility is asserted; the declaration-order tie break is not isolated.

TEST-031 partly verified

naut test --target logix drives the same scenarios against a Logix controller: given writes go to the controller and wait to read back, advance: d is d of wall time plus a poll, scans: n is n task periods plus a poll (a continuous task counting one poll each), until/hold/always are evaluated per poll, suspend is a no-op, and alarm verbs are unavailable.

where the docs say so · Run against an in-process fake controller; the real-controller run is verified on hardware only. Given writes going to the controller, a write that never reads back being refused, and always: catching a violation that recovers before the step ends (per poll) are asserted; the advance/scans timing, suspend being a no-op and the alarm-verb refusal are not.

TEST-032 partly verified

Program-scope tags on a Logix controller are served as <Program>_<Tag>, and a scenario may use the bare name.

where the docs say so · Bare-name resolution is asserted against a fake controller that serves MainProgram_-prefixed names (the scenario passes, and a failure names MainProgram_RunCmd); a real controller's naming is verified on hardware only.