Introduction

NXD is a declarative infrastructure reconciliation engine for Nix-authored desired state. It evaluates consumer Nix modules into canonical JSON, plans structural and host changes as a dependency-ordered graph, applies persisted plans under digest approval, and verifies the result through typed Rust providers.

This manual documents behavior that exists in the tree. NXD is pre-release: where a capability is planned rather than shipped, the page says so.

Where to start

The pipeline

Every NXD invocation moves through the same stages. The boundaries between them are the load-bearing part of the design: each stage may only consume what the previous one produced.

Nix flake modules
      │
      ▼  evalConfiguration — offline, no network, no decryption
Canonical JSON specification
      │
      ▼  nxd plan — dependency-ordered action graph + risk classification
Persisted plan  (sha256:…)
      │
      ▼  approval of that exact digest
Provider mutation
      │
      ▼  nxd verify — live state re-read
Post-apply verification and idempotency check

Key concepts

Nix-authored canonical configuration

Site topology, host inventory, hypervisor settings, and service attachments are authored in Nix modules under nxd.site, nxd.providers, nxd.secrets, and nxd.operations. nxd.lib.evalConfiguration exports that desired state as canonical JSON.

Evaluation is strictly offline. It contacts no endpoint and decrypts no secret, so planning cannot be influenced by the state of the infrastructure being planned against.

Canonical JSON specification

One reproducible document describing the full desired state:

• deploymentTarget — host declarations, build settings, SSH identities, and plane attachments.
• providerInstances — provider configuration (Nix, PVE, PBS, VMware, Headscale, Identity, DigitalOcean, WSL).
• secretBindings — references to SOPS/age documents. References only; never values.
• resources — provider-owned managed resources such as pveVm, pbsDatastore, and headscaleNode.

Dependency-ordered action graph

The scheduler computes an acyclic graph ordering actions across every provider, rather than running per-provider steps in sequence.

Prerequisites are edges, not conventions. Minting a Headscale preauth key or resolving an SSH host key is ordered before the target activation that consumes it, because the dependency is recorded in the graph.

Every action carries a risk classification, ordered ReadOnly < Reversible < ServiceImpacting < Destructive < IdentityCritical. The classification is visible at approval time, so the class of change is known before it is authorized, and evidence records the highest risk in the plan. See the safety model for what each level means.

Digest-bound approval

A plan is persisted to disk and hashed. Applying requires approval of that exact digest. If configuration changes between planning and applying, the digest no longer matches and the apply refuses rather than executing a plan nobody reviewed.

See the safety model for the full refusal set.

Secret mediation

Secrets stay out of Nix store paths and out of canonical JSON. Configuration carries binding references such as secret/bar/hosts/medo/ssh-host-ed25519-private-key, and nothing else.

At apply time, SOPS/age resolves those references directly into process memory, or into staging files with 0600 permissions when a subprocess requires a path. The Nix store is world-readable, which is precisely why nothing sensitive is placed in it.

Idempotent verification

Success is a re-read of live state, not a zero exit code. Every lifecycle phase verifies the target against the desired closure, and a converged system replans to zero actions with desiredSystemPath == activeSystemPath and systemPathMatches: true.

Quick start with disposable fixtures

This tutorial uses checked-in synthetic fixtures only. It does not contact a real provider.

nix develop
cargo run -- validate --config-json tests/fixtures/canonical/minimal-site.json --format json
cargo run -- plan synthetic/example --config-json tests/fixtures/canonical/minimal-site.json --format json

For the Gate C backup-job fixture:

cargo run -- validate --config-json tests/fixtures/infrastructure/backup-job-site.json --format json

For Nix-authored sites, use the shared model and let NXD project canonical resources:

{
  nxd = {
    stack.name = "example";
    site.guests.web = {
      cluster = "lab";
      node = "pve-a";
      vmid = 120;
      tags = [ "production" ];
    };
    providers.pve.lab = {
      endpoint = "https://pve.example.invalid:8006";
      credentialBinding = "secret/pve-api";
    };
    secrets.bindings.pve-api = {
      resolver = "sops-age";
      document = "sites/example/pve.yaml";
      key = "token";
    };
    operations.production.includeTags = [ "production" ];
  };
}

Export it through nxd.lib.evalConfiguration as nxdConfigurations.<site>, then run nxd validate --source .#nxdConfigurations.<site>. Validation does not contact provider endpoints or resolve secret values.

Do not substitute production endpoints into fixture files. Real endpoint runs require an accepted runbook and explicit owner authorization.

Validate configuration

Use validate to parse and validate canonical JSON without contacting a provider:

nxd validate --config-json tests/fixtures/infrastructure/backup-job-site.json --format json

Validation proves local shape and reference constraints. It does not prove live provider compatibility, credentials, trust material, or target identity.

Choose where a system is built

Building a NixOS system needs memory and CPU. The machine that will run the system is often not the best machine to build it — and sometimes cannot build it at all.

deployment.buildOn decides this per host.

The options

buildOn = "builder" requires deployment.builder = "user@host".

The low-memory case

Evaluation — working out what to build — can require more memory than a small machine has, before a single package is compiled. A 1 GB VM will often be OOM-killed during evaluation.

Set lowMem:

deployment = {
  lowMem = "yes";
};

With lowMem = "yes", a target build is automatically upgraded to instantiated: the orchestrator performs the memory-hungry evaluation, and the target realizes the reviewed derivation into its own store. The small machine never holds the build plan, but the finished system is still assembled where it runs, so no large closure transfer is needed.

NXD validates that the derivation produces the reviewed output, so this is not a weaker guarantee than building anywhere else.

Declare builders explicitly

Use builderBySystem to declare which builder serves which system type:

deployment = {
  buildOn = "builder";
  builderBySystem = {
    "x86_64-linux" = "deploy@utils";
  };
};

NXD deliberately does not inherit ambient builder configuration from /etc/nix/machines. A build that silently used whatever the workstation happened to have configured would make placement invisible and unreproducible between operators. Declaring builders in the repository keeps placement part of reviewed configuration.

Confirming what happened

Provider progress reports both the configured and the resolved placement:

✓ [medo] build placement configured=auto builder=deploy@utils

If auto resolved somewhere unexpected, this line says so.

A note on evaluating locally

Evaluating on your workstation instead of a Linux builder is a mitigation for memory pressure, not a speedup. Local evaluation on macOS in particular is slower than evaluating on a Linux builder. Choose it when the alternative is an OOM kill, not to make deployments faster.

Architecture overview

NXD keeps three concerns apart: the consumer repository owns what the infrastructure should be, the core engine owns what must change and in what order, and providers own how a specific system is actually mutated.

The boundaries are what make the engine testable and the providers replaceable. Each layer may only consume what the previous one produced.

Ownership boundaries

┌──────────────────────────────────────────────────────────────┐
│ Consumer repository                                          │
│ Topology, inventory, addresses, storage, site ACLs           │
│ SOPS/age encrypted secrets (references only, never values)   │
└───────────────────────────┬──────────────────────────────────┘
                            │ evalConfiguration  (offline)
                            ▼
┌──────────────────────────────────────────────────────────────┐
│ Core engine — nxd-core                                       │
│ Canonical JSON parsing and validation                        │
│ Action graph scheduling and risk classification              │
│ Plan persistence, approval evidence, journaling              │
└───────────────────────────┬──────────────────────────────────┘
                            │ provider port protocol
                            ▼
┌──────────────────────────────────────────────────────────────┐
│ Providers — crates/providers/                                │
│ nix         NixOS and Darwin build, transfer, activation     │
│ pve         Proxmox VE API, PXE assets, QDevice witness      │
│ pbs         Proxmox Backup Server API                        │
│ headscale   Tailnet control plane: users, keys, nodes, ACLs  │
│ identity    SOPS/age key sink and host trust                 │
│ vmware      VMware Fusion and ESXi VMX substrate             │
│ digitalocean, wsl                                            │
└──────────────────────────────────────────────────────────────┘

The core never speaks a provider’s protocol, and a provider never decides ordering. A provider that learned to schedule its own work would break the one property the whole design exists to provide: that ordering across provider boundaries is decided in one place.

Evaluation and canonical JSON

nxd.lib.evalConfiguration (or evalConfigurationUnstable for in-development schemas) evaluates consumer modules into a canonical JSON specification.

• Offline. Evaluation performs no network I/O, contacts no API, and decrypts no secret.
• Target-scoped. lib.selectTargetInventory extracts the resource subgraph for the selected targets, so planning one host does not evaluate every host in the site.

Offline evaluation is a correctness property, not a performance one: if evaluation could read live state, the plan would depend on the thing it is planning against, and two runs against a drifting endpoint could disagree.

Transport — nxd-transport

One shared SSH layer serves every provider.

• Multiplexed by default. ControlMaster=yes with ControlPersist=10m reuses a single connection instead of paying a TCP and SSH handshake per remote command.
• Host keys pinned. Every connection verifies the target’s key against the managed identity record. There is no prompt and no fallback.
• Jump hosts. Targets behind a bastion route through SSH proxy-jump without ambient credentials.

These are one decision, not two. Before the transport was shared, twelve non-test modules across four providers invoked SSH and exactly one multiplexed — which meant host key handling was also inconsistent. Consolidating fixed both.

Provider port

Providers implement four handlers, mirroring the pipeline:

Only apply() mutates. plan() emitting an action is a proposal, not a commitment — nothing runs until a plan carrying that action is persisted, hashed, and approved.

Where to go next

• Safety model — the refusal set, approval binding, and risk levels.
• CLI reference — commands, intents, and flags.
• Resource schema reference — provider resource fields.

Safety model

NXD is built so that the destructive step is the one that is hardest to reach by accident. Every safety property below is a refusal in code, not a convention.

The core separation

apply never re-plans. It executes one plan that already exists on disk, which is what makes the reviewed artifact and the executed artifact the same object.

Why approval binds to a digest

A plan is written to disk and hashed. Approval is recorded as evidence bound to that hash, so authorization cannot drift from what was reviewed.

apply refuses when any of these hold:

• The plan expired. Plans carry expiresAtUnix. Applying past it fails with the expiry and current time, and the fix is a new plan — expiry is not overridable.
• The digest does not match. Evidence names a specific plan digest. Evidence for a different plan is rejected.
• The approval summary does not match the actions. approvalRequirements is recomputed from the plan’s own actions and install mode, and compared. A summary that was edited, stripped, or forged after planning is refused with “plan approvalRequirements do not match its actions; recreate and review the plan”. The summary is convenience data, never an independent authority.
• Evidence is missing on a destructive plan with non-interactive stdin. CI cannot silently inherit a human’s approval. The refusal prints the digest, the expiry, every requirement with its risk and resources, and the two commands needed to proceed.
• Evidence is supplied for a plan that needs none. Mismatched expectations are an error in both directions.

Approval evidence has its own constraints: the principal must be a non-empty operator identity under 128 characters with no control characters, expiry must be in the future, and evidence may not outlive the plan it approves. Evidence defaults to a one-hour TTL, clamped to the plan’s own expiry.

Risk classification

Each action carries one of five risk levels, ordered lowest to highest:

Evidence records the highest risk present in the plan, so approving a low-risk plan never authorizes a high-risk one.

Default plan output groups work by target and lists destructive and identity-critical actions first, naming the exact resources touched. --verbose renders the full graph with identifiers and dependency edges.

Secrets

Secrets appear in configuration only as bindings and are resolved at runtime.

A secret value must never appear in Nix expressions, canonical JSON, plans, logs, events, argv, URLs, or committed documentation. argv matters specifically because process arguments are readable by other users on the same host — secrets reach subprocesses through the environment or 0600 files, never as command-line arguments.

Enrollment continuation

Interactive durable enrollment (deploy, switch --reenroll, and install or convert intents) asks once for the enrollment plan plus a displayed, bounded same-session continuation across the selected targets.

The continuation is not pre-authorized work. It is freshly planned, persisted, digested, and validated against the authorized scope before any approval evidence is written. Any change to source, target, action, provider, secret, dependency, or risk stops for explicit review.

The split exists because minting is decided by reading the current persisted binding and matching it against live inventory, and minting mutates an input that the installing plan’s own planning reads. One plan cannot honestly contain both. Non-interactive and separate plan/apply workflows still require evidence per digest.

Plan retention

nxd clean is a dry run unless --apply is given. Even then it only archives expired plans that carry no approval, journal, or pin evidence — it never deletes reviewed evidence.

• --pin <plan> protects a plan through a long review.
• --unpin <plan> releases it.
• --recover <archived-plan> restores an archived plan.

Recovery restores the artifact, not its authority: expiry, compatibility, and approval checks all still apply.

What this does not protect against

Stated plainly, because a safety model that implies more than it delivers is worse than none:

• An operator who approves a plan without reading it. Approval proves what was authorized, not that anyone understood it.
• Anything outside NXD’s model — a change made directly on a host, or through a provider’s own UI, is drift NXD can only detect at the next plan.
• Correctness of your desired state. NXD converges infrastructure to what you declared; it has no opinion on whether that is what you wanted.

Installation and recovery

Installing a machine is not a special mode. It is a lifecycle composed into the same reviewed action graph as everything else, with the same digest-bound approval and the same risk classification.

The division of ownership is deliberate: your topology selects the target and its provisioning, the infrastructure provider (Proxmox, VMware, DigitalOcean) establishes the guest, and the Nix provider owns realization, transport, Disko, kexec, first boot, and convergence.

Three install modes

--install-mode selects the policy for an existing guest:

These have distinct approval requirements, because they have distinct consequences. Before any destructive transition, NXD validates required secrets, disks, storage, network, cache, builder, source and target identity, and replacement preflight. A destructive install that cannot satisfy its preconditions refuses before it touches anything.

Conversion

--intent convert takes over an existing machine via kexec, managing the declared source transition only.

If a destination guest is involved, it is reconciled by its owning provider as normal work — a conversion never silently creates or deletes a guest as a side effect.

Bootstrap trust is temporary

Installing a machine requires trust that the finished machine must not keep.

Bootstrap services are bounded, scoped to one operation, authenticated, and removed or disabled at handback. Operation-private trust never becomes ambient steady-state trust. A key that existed to install a machine does not survive into its running configuration.

A bootstrap server serves only the reviewed operation and its allow-listed artifacts, on its configured interface and port, with connection, request, and size limits. It re-attests file descriptors before serving and cleans up on success, failure, cancellation, and timeout.

SSH-agent access during an operation is a typed binding. Its socket is never copied into desired state or inherited indiscriminately.

Takeover, reconnect, DHCP re-observation, first boot, and stable-trust handback each have explicit bounded timeouts and postconditions, so a stalled install fails cleanly instead of hanging.

Installing on low-memory machines

Low-memory installation uses the reviewed takeover and build strategy rather than a separate code path. Disko and bootstrap work may use a bounded tmpfs, while large system realization uses the configured cache or a compatible builder — so the constrained machine never has to hold work it cannot finish.

See build placement for choosing where a build happens.

Artifacts are immutable and attested

Installer, PXE, WSL, and recovery artifacts are immutable outputs carrying a manifest, checksum, provenance, a non-empty SBOM, and a source revision.

Providers consume reviewed artifacts. Consumers declare source identity, answer and first-boot content, output identity, and site values — they do not maintain derived store paths by hand.

Two rules protect secrets in installer media:

1. Any answer file or media containing a secret is materialized only inside the reviewed action, from an action-scoped confidential input. It is excluded from the Nix store and from the public manifest, and removed on success, failure, cancellation, or timeout.
2. Public handoff evidence binds artifact role, product, version, source digest, prepared digest, storage identity, and run-created ownership. It never contains artifact bytes, secret bytes, or rendered secret fields.

Capture and recovery

Capture and recovery operate on the selected producer and bounded artifacts only — never the whole site.

Capture refuses to write through a symlink, refuses an existing destination, and refuses an unsafe directory, so a capture cannot be redirected into overwriting something it should not.

For full details, see docs/architecture-design/installation-and-recovery.md in the repository.

Transport and connections

Every provider that talks to a remote machine goes through one shared layer, nxd-transport. It is core infrastructure, not a provider: nothing in NXD opens its own SSH connection or spawns its own unbounded subprocess.

That consolidation is the point. Host key verification, connection reuse, timeouts, and output sanitization are decided once and enforced everywhere, rather than reimplemented per provider with slightly different rules.

Host keys are pinned, always

Every connection verifies the target’s Ed25519 host key against the identity record NXD manages for that host.

There is no interactive prompt, no trust-on-first-use, and no fallback to an unverified shell. A host whose key does not match the reviewed identity is a refusal, not a warning — the same key material that provisioned the machine is the key material used to reach it.

Connections are reused

SSH connections are multiplexed by default: ControlMaster=yes with ControlPersist=10m. The first connection to a host establishes a master, and every subsequent command in that operation reuses it.

Without this, each remote command pays a full TCP handshake plus an SSH handshake. A deployment that runs twenty commands against a host pays that cost twenty times, which on a distant or slow link dominates the actual work.

Reuse and verification are the same decision. Before the transport was shared, twelve non-test modules across four providers invoked SSH and exactly one multiplexed — which also meant host key handling varied between them.

Jump hosts

Targets behind a bastion are reached with SSH proxy-jump (-J), configured per target rather than assumed from ambient SSH config.

Reachability probing is proxy-aware. A target on an isolated subnet that is only routable through its jump host is probed through that jump host, so an unreachable-looking target is genuinely unreachable rather than an artifact of probing from the wrong network position.

Subprocesses are bounded

Local and remote command execution runs through a common process runner that provides:

• Streaming output, so long operations report progress instead of blocking until completion.
• Line-level sanitization, so secret values cannot reach logs or the terminal.
• Timeouts and cancellation, so a hung remote command fails the action rather than the whole run.

What this means in practice

You do not configure the transport directly. It is worth knowing about for three reasons:

1. A host key mismatch is a hard failure. If you rebuild a machine outside NXD and its host key changes, the reviewed identity must be updated — the transport will not silently accept the new key.
2. Jump hosts belong in your configuration, not in ~/.ssh/config. NXD does not inherit ambient SSH configuration, so routing is visible in the repository.
3. Slow first connection, fast subsequent ones is expected. The master connection is established once per operation.

For the design rationale and the measurements behind the consolidation, see docs/architecture-design/nxd-transport.md in the repository.

Performance model

NXD’s performance rules are mostly about not doing work. Two ideas carry almost all of it: evaluate only what was selected, and let independent work proceed independently.

Select first, then evaluate

Nix evaluation is the expensive part of any deployment tool. The naive approach evaluates the whole site and filters afterwards, so planning gets slower with every host added, whether or not you touched them.

NXD selects before it works. Planning one target evaluates that target’s systems, its providers, and its secret bindings. It does not:

• evaluate unrelated hosts to discover they were not selected;
• observe endpoints belonging to resources outside the selection;
• recursively hash or enumerate closures to prove equality; or
• resolve secrets for hosts that are not part of the operation.

The practical consequence is that planning cost tracks what you asked for rather than how large the site has become.

Related invariants:

• The selected canonical site is evaluated once per command, and its JSON is parsed once.
• Exact-target projections are used only when they are semantically equal to the selected slice of the full site. A broad selector falls back to the full site and then deterministic selection, rather than an unsafe partial projection.
• Selected output evaluation is batched where practical.

Independent work runs concurrently

The scheduler starts an action as soon as its own dependencies have succeeded and no conflicting resource is locked. It does not wait for artificial batch waves, so fast work never idles behind slow, unrelated work.

This matters most across mixed fleets. A Proxmox guest, a cloud droplet, and a Darwin host share no dependencies and no ownership locks, so they progress in parallel while the ordering that genuinely matters is still enforced.

Concurrency is bounded by --parallel (default 5).

Two rules keep this safe:

• Only actions whose complete dependency set succeeded may start. If a prerequisite fails, its dependants remain unexecuted rather than being reported as provider failures.
• The scheduler acquires ownership locks for conflicting resources, so concurrency never lets two actions mutate the same thing.

Connections are reused, observations are not

Authenticated transport connections are reused within an operation, which removes a per-command handshake. See transport.

Observations are deliberately not reused. NXD never carries an observation across observe, apply, verify, endpoint identity, credential, reviewed configuration, or process boundaries. Caching live state would make a plan depend on stale information, which is exactly the failure mode digest-bound approval exists to prevent.

For the same reason, NXD prefers Nix’s own store and substitution behavior over maintaining custom caches or artifact registries.

Delivery takes one route

Each reviewed output is bound to a single delivery route. Builder-direct install and conversion batch system, Disko, and kexec inspection over one connection.

When substituteOnDestination is enabled (the default), the destination fetches what it needs from its own configured signed cache, and the reviewed builder is a lower-priority trusted source for exact reviewed paths missing from that cache. The closure is never routed through the orchestrator as an intermediary.

Measuring it

--profile reports per-phase timings. See output and logs.

For the complete invariant list and route composition table, see docs/architecture-design/performance-and-profiling.md in the repository.

Output and logs

NXD writes to two places at once, with different jobs.

Every lifecycle operation uses both, whether it targets one host or twenty. The terminal never becomes the only record, and the log file never becomes the thing you have to read to follow a normal run.

Progress lines name what they did

A progress message states the specific resource, the action, and the parameters that matter. Vague provider phases such as “VMware action completed” are deliberately not allowed — a message that does not say which resource it touched is not useful in a postmortem.

Step formatting depends on duration, to avoid noise:

• Fast steps emit a single completed line with exact parameters:

  [medo-test] Configured DNS resolvers (1.1.1.1, 8.8.8.8)
  

• Slow steps emit an active line when they start and a completion line annotated with how long they took:

  [medo-test] Partitioning & formatting disk /dev/vda with Disko...
  [medo-test] Partitioned & formatted disk /dev/vda (ext4) (18s)
  

A fast step never emits both a started and a completed line for the same work.

Secrets are redacted in both channels

Secret values, private keys, tokens, and confidential parameters are redacted from the terminal and from log files. A log file is a durable artifact that often gets pasted into an issue or a chat, so it is held to the same standard as anything else that leaves the machine.

Profiling is separate

--profile reports internal phase timings for diagnosing where a slow run spent its time:

PROFILE phase=inventory-evaluation duration_ms=4120

It covers inventory evaluation, runtime composition, selected-output evaluation, endpoint attestation, secret resolution, observation, planning, apply, and total time.

Profiling output goes to stderr (or structured JSON) so it never disturbs the normal progress stream or any JSON on stdout. Progress reports human durations for milestones you care about; profiling reports machine-readable latency for phases you are investigating. They share the same underlying timers and do not duplicate each other’s lines.

Phase names are stable operator diagnostics. They are not a second lifecycle model, and commands do not invent provider phases by parsing action names.

Build placement is reported

Provider progress reports both the configured and the resolved build placement, so an auto setting never leaves you guessing where the work actually happened:

✓ [medo] build placement configured=auto builder=deploy@utils

For the full invariants, see docs/architecture-design/logging-and-progress.md in the repository.

CLI reference

Generated from the released Clap command model. Do not edit by hand.

Provider resource reference

Generated from the released linked-provider metadata. Do not edit by hand.

Shared Nix option reference

Generated from nix/modules/authoring/. Do not edit by hand.

NXD derives canonical IDs, provider resources, associations, operation sets, and allowlisted PVE host-state files. It refuses duplicate IDs, unknown or unused reserved operation tags, unsupported host-state paths, and empty operation policy.

Compatibility matrix

See docs/compatibility.md in the repository source. Released manuals will include a generated compatibility matrix from release metadata.

Incident response

See docs/runbooks/incident-response.md in the repository source.

Do not include secrets, private endpoints, raw production journals, or unredacted infrastructure captures in public reports.

Release readiness

See docs/release/release-policy.md and docs/release/supply-chain.md.

NXD is pre-release. Release signing, SBOM, and provenance automation are planned pre-alpha work.

Security policy

See SECURITY.md in the repository root for current reporting instructions.

Common failures

Validation fails

Check for unknown fields, duplicate resource IDs, unresolved references, and invalid provider configuration.

Apply refuses a plan

Apply may refuse expired, stale, wrong-provider, incompatible, or changed-input plans. Re-run plan after investigating the refusal.

Provider operation is ambiguous

Mutation disconnects and timeouts are ambiguous. Do not blindly retry mutation. Observe current state and replan.

Contributing

Read CONTRIBUTING.md, AGENTS.md, and docs/README.md before proposing a change.

Non-trivial work requires accepted artifacts, tests, evidence, and review.