Run and check an AeroNyx node

This runbook covers the day-to-day commands for a registered AeroNyx node: status, health, logs, relay and mailbox proofs, safe upgrades, rollback, and what to check after a restart. Every command is in the node repository (deploy/node/aeronyx-node.sh and the aeronyx-server binary) and is shown with the options the code accepts.

If you have not installed a node yet, start with Install and register an AeroNyx node.

Before you start

Run everything from a root shell (sudo -i). The configuration directory /etc/aeronyx and the state directory /var/lib/aeronyx are mode 700, so status, health, validation and the relay commands cannot read them as another user.

Change into the repository the service actually runs from, and pass it with --repo-dir on every script command. Without --repo-dir, the script assumes /opt/aeronyx/AeroNyx.

cd "$(systemctl show aeronyx-server -p WorkingDirectory --value)"
pwd

On nodes installed with the Nodeboard command, this prints /root/open/AeroNyx.

Commands at a glance

Status

./deploy/node/aeronyx-node.sh status --repo-dir "$PWD"

status prints, in order:

1. systemctl status aeronyx-server (last 20 lines)
2. a summary of the local health API (http://127.0.0.1:8421/api/vpn/health) and operator status API (http://127.0.0.1:8421/api/operator/status)
3. local discovery readiness from http://127.0.0.1:8422/api/discovery/status: ChatRelay advertisement and peer quorum
4. the last upgrade workflow state from /var/lib/aeronyx/upgrade-status.json
5. an operator recommendation: operator_status, operator_title, operator_detail and operator_next_step

The recommendation comes from the health check. Its status is one of:

Health

./deploy/node/aeronyx-node.sh health --repo-dir "$PWD" --json

The health check reads files, systemd and the local endpoints; it never changes the host. Each check prints [PASS], [WARN] or [FAIL], then a [SUMMARY] pass=… warn=… fail=… line, then one line of JSON. Use --json-only to print only the JSON. The command exits with a non-zero status if any check fails.

What it checks: the repository, the tracked Cargo.lock and the release binary; configuration validation with aeronyx-server validate; registration and key files; the systemd unit, restart policy and hardening; memory, disk and ports 51820 and 8421; IP forwarding and NAT; the local health endpoint; discovery; and release backups.

Main JSON fields:

You can read the raw local endpoints directly:

curl -s http://127.0.0.1:8421/api/vpn/health
curl -s http://127.0.0.1:8422/api/discovery/status

Logs

./deploy/node/aeronyx-node.sh logs --repo-dir "$PWD" --lines 200
./deploy/node/aeronyx-node.sh logs --repo-dir "$PWD" --follow

These wrap journalctl -u aeronyx-server -n <lines> --no-pager (default 160 lines) and journalctl -u aeronyx-server -f. Use journalctl directly for time ranges, for example journalctl -u aeronyx-server --since "1 hour ago".

Relay and mailbox proofs

These commands answer different questions. Do not treat a passing synthetic probe as proof of real delivery.

relay-probe (synthetic)

./deploy/node/aeronyx-node.sh relay-probe --repo-dir "$PWD" --json
./deploy/node/aeronyx-node.sh relay-probe --repo-dir "$PWD" --two-hop --json

Sends one synthetic, random opaque BlindRelay envelope from this node to a discovered ChatRelay peer and reports only counter changes. --two-hop sends it through a middle hop to a terminal node. The result status is ok, attention, blocked or error, and two_hop_readiness explains what is missing when a two-hop path is not possible. It reads the peer cache and the node key, so it needs root. It is not user traffic and does not prove mailbox delivery. fleet-smoke runs a similar synthetic check across several public nodes you list with --endpoints.

relay-smoke (authenticated, end to end)

./target/release/aeronyx-server relay-smoke --confirm-live-relay-smoke --json

relay-smoke acts as a real client against the node on this host. It opens two short-lived sessions with throwaway identities and completes the production handshake, pinned to this node's key. It sends one 48-byte random message, end-to-end encrypted, through the authenticated multi-hop relay path. It then verifies the signed terminal receipt for that exact request, pulls the message from this node's mailbox, decrypts it, acknowledges it, and checks that the mailbox is empty and the sessions are closed.

It refuses to start unless every gate passes:

Notes:

• It must run as root on the node itself. It reads /etc/aeronyx/server.toml and the node key, and accepts only loopback addresses (--server, default 127.0.0.1:51820; --health-url, default http://127.0.0.1:8421/api/vpn/health).
• --timeout-seconds sets the proof window: 5 to 120 seconds, default 30.
• Zero active sessions means a node that serves VPN users must be drained first. Turn on Maintenance Mode in Nodeboard and wait for sessions to reach zero.
• The two test sessions are real protocol sessions, so they can appear briefly in Nodeboard Sessions.
• A pass has "status": "passed", terminal_receipt_observed, entry_mailbox_round_trip_verified, entry_mailbox_ack_verified and e2e_ciphertext_verified all true, and session_cleanup is explicit_authenticated_close. terminal_replica_cleanup is node_ttl_managed: acknowledging on the entry node does not delete the copy held by the terminal node, which expires on its own.
• Any failed gate, error or missing field is a failed proof, not a partial success.

mailbox-probe (client-sourced)

./target/release/aeronyx-server mailbox-probe \
  --seed http://SEED_PUBLIC_IP:8422 \
  --target <64-hex node ID of the mailbox node> \
  --entry <64-hex node ID of a different entry node> \
  --confirm-live-mailbox-probe --json

mailbox-probe behaves like an app: it uses throwaway keys, builds its own onion route from signed descriptors collected from the --seed nodes, and sends plain HTTP to the entry node's public peer API, with no VPN session. It runs the full anonymous mailbox lifecycle on the target node: ticket_issue, lease_create, put, pull_one, ack and a final pull_one. It checks that the pulled bytes match and that the mailbox is empty afterwards. A pass reports "status": "verified" and the number of hops.

• --seed can be repeated. Without --entry the route has one hop.
• A node ID is the node's 64-character hex public key. On a node you operate, ./target/release/aeronyx-server pubkey -c /etc/aeronyx/server.toml prints it.
• The target must have the anonymous mailbox enabled ([memchain.chat_relay.anonymous_mailbox] enabled = true); it is off by default.
• --work-bits (default 12) must be at least the target's ticket work setting. --timeout-seconds is per request (default 20).
• It does not read local node files, so it can run from any machine with the binary. It proves custody of an opaque item, not that a recipient can decrypt it.

Upgrade safely

Use the upgrade workflow, not git pull followed by systemctl restart:

./deploy/node/aeronyx-node.sh upgrade --repo-dir "$PWD" --build-priority live --build-jobs auto

What it does:

1. Takes a deployment lock and refuses to run if tracked files in the checkout are modified (--allow-dirty overrides).
2. Pulls main (or builds an exact commit in an isolated checkout with --commit <40-hex SHA>).
3. Builds with the repository-pinned Rust toolchain. live priority (the default) and --build-jobs auto use about half the CPUs so the running node keeps serving.
4. Validates /etc/aeronyx/server.toml with the new binary. The configuration file is never overwritten.
5. Backs up the binary that is actually running into /var/lib/aeronyx/releases/.
6. Checks active sessions after the build, and again just before the restart. If there are any, or if the count cannot be read while the service runs, it stops without switching versions.
7. Updates the systemd units, swaps in the new binary, and restarts the service.
8. Waits up to 180 seconds (90 checks, 2 seconds apart) for local health to report ok from the commit it just built. If the restart, the service state or the health check fails, it restores the previous binary and units automatically.
9. Keeps the 10 most recent backups of each type and records the result in /var/lib/aeronyx/upgrade-status.json.

If the node has active sessions, the build still runs and the upgrade then stops with Active sessions appeared during compilation. Enter maintenance mode, drain sessions, and run the upgrade again. Turn on Maintenance Mode in Nodeboard Settings, wait for active sessions to reach zero, and run the same command again. The rebuild reuses the cache.

Stage now, restart later

./deploy/node/aeronyx-node.sh upgrade --repo-dir "$PWD" --no-restart

--no-restart builds, validates and backs up, then installs the new binary on disk without restarting. status then reports upgrade_staged. The running process keeps the old version. Any later restart, including a reboot or an automatic restart after a crash, starts the new binary.

To cut over in a maintenance window, drain the node and run the upgrade again without pulling, so it switches to exactly what you staged with the session check, health wait and rollback:

./deploy/node/aeronyx-node.sh upgrade --repo-dir "$PWD" --skip-pull

Do not use --force. It restarts even when sessions are active or the session count is unknown, and disconnects users.

Roll back manually

The upgrade rolls back automatically when the new release fails to start or to report healthy. To return to an earlier release later, promote one of the saved binaries:

ls -lt /var/lib/aeronyx/releases/
./deploy/node/aeronyx-node.sh promote-binary --repo-dir "$PWD" \
  --binary /var/lib/aeronyx/releases/<aeronyx-server.binary.YYYYMMDD_HHMMSS.xxxxxx>

promote-binary validates the configuration with the chosen binary and refuses to run unless active sessions are exactly zero. It asks you to type PROMOTE BINARY, backs up the current binary next to it, swaps the binary and restarts. If the service is not healthy within 180 seconds, it restores the previous binary. It replaces only the binary, not the systemd units.

After any restart

Check these in order:

1. systemctl is-active aeronyx-server returns active. The unit is Type=notify, so active means the process reported ready; startup may take up to 300 seconds before systemd gives up.
2. ./deploy/node/aeronyx-node.sh status --repo-dir "$PWD" shows operator_status=ok, or a warning you understand.
3. ./deploy/node/aeronyx-node.sh health --repo-dir "$PWD" --json ends with "success":true and local_vpn_health.status is ok.
4. The running commit is the one you intended: curl -s http://127.0.0.1:8421/api/vpn/health reports it as runtime.git_commit. (The runtime.git_commit in health --json is the checkout's commit.)
5. discovery_readiness.peer_quorum.quorum_ready is true after the first gossip rounds.
6. For relay nodes: relay-probe --two-hop --json reports ok.
7. In Nodeboard, the heartbeat is fresh, Rust Runtime shows the new version, and the restart appears under Alerts / Events. Turn off Maintenance Mode when you are done.
8. journalctl -u aeronyx-server --since "15 min ago" has no repeating errors.

How this appears in Nodeboard

Nodeboard receives the node's signed heartbeat and session reports. It does not run commands on the host by itself; actions you choose there are queued and the node runs them on its next heartbeat.

Node detail also has an Operator Runbook card with copyable health, status, logs and staged-upgrade commands for the detected repository path.

What the outputs contain

status, health, relay-probe, relay-smoke and mailbox-probe print aggregate values only: counts, states, timings and reason codes. They do not print private keys, registration codes, client public IPs, destinations, DNS contents, payloads or message plaintext. Session records with client keys are visible in Nodeboard, not in these outputs; see the Nodeboard operator console guide. When you share output for support, share the JSON and its timestamp, not host addresses or credentials.

Frequently asked questions

Which command should I run first when something looks wrong?

Run ./deploy/node/aeronyx-node.sh status --repo-dir "$PWD" as root. It shows the service state, local health, discovery and the recommended next step, and it does not change the host.

Why does health fail with "config missing" or Git warnings?

You are probably not root, or you are pointing at the wrong repository. /etc/aeronyx and /var/lib/aeronyx are root-only, and without --repo-dir the script checks /opt/aeronyx/AeroNyx instead of the checkout the service runs from.

What is the difference between relay-probe and relay-smoke?

relay-probe sends one synthetic opaque envelope and only shows that a relay path accepts traffic. relay-smoke acts as a real client: it verifies a signed terminal receipt and a full mailbox pull and acknowledgement for one encrypted test message.

Why does relay-smoke say it needs zero active sessions?

It attributes counter changes to its own test sessions, so it only runs on an idle node. Turn on Maintenance Mode in Nodeboard, wait for active sessions to reach zero, then run it.

Can I upgrade without disconnecting users?

You can build and stage an upgrade with --no-restart while users are connected. Switching to the new version needs a restart, so drain the node with Maintenance Mode and then run upgrade --skip-pull.

How do I go back to the previous version?

A failed upgrade rolls back automatically. To roll back later, use promote-binary with one of the backups in /var/lib/aeronyx/releases/; it requires zero active sessions and restores the previous binary if the promoted one is unhealthy.