Decentralized Privacy Node Operations and Health Checks

This guide explains how operators and engineers should think about AeroNyx decentralized privacy node health. The goal is to make node operation reliable without exposing user content or private routing data.

Core health layers

Recommended operator workflow

Open Nodeboard at https://app.aeronyx.network, choose a node, and check:

1. Protocol health - service, heartbeat freshness, transport state, and aggregate counters.
2. Discovery - peer counts, valid peers, gossip freshness, restart recovery, and rejected descriptor counters.
3. Relay foundation - blind relay status, routeability, abuse guard, and proof readiness.
4. Capacity - IP pool capacity, session limits, file descriptors, conntrack pressure, packet drops, pps, and bps.
5. Incidents - recent events, command audit state, and closure evidence after restart or upgrade.

Two-hop path proof checks

The decentralized node now exposes display-ready two-hop proof quality fields:

{
  "status": "ready",
  "proof_ready": true,
  "recent_success_ready": true,
  "failure_streak_active": false,
  "stale_after_seconds": 1800,
  "next_action": "continue monitoring repeated two-hop path proof freshness"
}

Operators should interpret these fields as follows:

Public network stats

The public aggregate endpoint exposes fleet-level privacy-safe status:

GET https://api.aeronyx.network/api/privacy_network/vpn/public/network-stats/

Relevant object:

data.protocol_status.protocol_foundation.two_hop_path_proof_history

Example healthy aggregate:

{
  "reported_nodes": 3,
  "proof_ready_nodes": 3,
  "recent_success_ready_nodes": 3,
  "failure_streak_nodes": 0,
  "status_counts": {
    "ready": 3
  }
}

The legacy vpn path remains for API compatibility. Public copy should describe the product as the AeroNyx Privacy Network or AeroNyx Privacy Protocol.

Restart and upgrade checks

After restart or upgrade:

1. Confirm the service becomes active.
2. Confirm heartbeat freshness in Nodeboard.
3. Confirm peer cache recovery.
4. Confirm discovery stage returns to a ready state.
5. Confirm two-hop proof status returns to ready.
6. Confirm no incident remains open without closure evidence.

If active user sessions exist, operators should prefer maintenance mode or a planned window before restart or upgrade.

Privacy rules for operators

Health checks may expose operational counters, but they must not expose:

• plaintext messages
• encrypted payload bodies
• raw route IDs
• client public IP activity
• DNS contents
• browsing domains or URLs
• MemChain plaintext
• voucher secrets
• wallet-level traffic
• social graph edges

The operator experience should feel rich and commercial while preserving the blind-node invariant.

Pinned witness operations

Pinned witnesses are an explicit operator trust policy for a MemChain commitment coordinator. Configure only independently operated, audited followers that are expected to preserve this coordinator's commitment history.

[memchain]
commitment_coordinator_enabled = true
commitment_witness_node_ids = [
  "<64-hex-ed25519-node-id-1>",
  "<64-hex-ed25519-node-id-2>",
]
commitment_witness_startup_required = false
commitment_witness_min_verified = 2

Replace every placeholder before validation. The node accepts at most three unique, nonzero 32-byte Ed25519 identities. These settings are valid only on the commitment coordinator.

Safe rollout

1. Upgrade at least two audited follower nodes so their signed peer-checkpoint endpoint is compatible.
2. Add their exact node identities with commitment_witness_startup_required = false.
3. Run aeronyx-server validate -c /etc/aeronyx/server.toml.
4. Restart once and verify that the witness scope is operator_pinned and signed evidence is accepted.
5. Repeat the restart test and confirm no remote-ahead or divergence result.
6. Enable commitment_witness_startup_required = true only after evidence is consistently available.

With strict enforcement disabled, missing or unreachable witnesses produce a visible degraded warning but preserve availability. A verified remote-ahead or divergent result always blocks startup. With strict enforcement enabled, the absence of any verified pinned evidence also blocks startup.

The peer checkpoint endpoint is a signed protocol endpoint. An anonymous browser GET is not a valid health check. Use node startup logs, Nodeboard's privacy-safe status, and the signed reconciliation result instead.

Never pin arbitrary permissionless peers or multiple nodes controlled by the same operator and call that independent evidence. Pinned witnesses improve rollback detection; they do not create consensus, quorum, or finality by themselves.

Current verified state: Verified production rollout — July 14, 2026

One production coordinator has three operator-pinned witness identities. Two independently operated, audited follower nodes now run compatible checkpoint services and each verified and stored the same 33-block, 8,339-commitment history. The coordinator accepted distinct signed converged responses from both followers, then enabled commitment_witness_startup_required = true.

On the verified restart, the startup guard reported configured=3, eligible=3, attempted=3, verified=2, and converged=2; UDP and TUN listeners opened only after this check passed. This is operator-pinned rollback evidence, not consensus, quorum, public-chain finality, or token execution.

Enforced 2-of-3 startup threshold — July 14, 2026

The production coordinator now sets commitment_witness_min_verified = 2 together with commitment_witness_startup_required = true. Startup stops when fewer than two distinct pinned witnesses provide valid signed evidence: zero valid responses produce signed_checkpoint_unavailable, while one valid response produces signed_checkpoint_threshold_unmet. The latest production restart verified two of three configured witnesses before UDP, TUN, or the public API listener opened.

This is an operator-defined startup threshold, not network consensus, quorum, finality, leader election, or fork choice. Rust heartbeat status reports only aggregate policy and outcome fields, including startup_minimum_verified; it does not expose witness identities, endpoints, checkpoint hashes, or signatures.

Related protocol specification

Signed Commitment Ledger and Witness Protection

MemChain AOF integrity check

Add this read-only gate to every planned upgrade and restart. It is safe for operator automation because its output is aggregate-only.

sudo aeronyx-server memchain verify-aof \
  --config /etc/aeronyx/server.toml

sudo aeronyx-server memchain verify-aof \
  --path /var/lib/aeronyx/.memchain

Privacy: Never upload, paste, or publish the AOF itself. Share only the aggregate verifier output.

Complete integrity and proof boundary: Signed Commitment Ledger and Witness Protection.

Guarded Cargo build-cache maintenance

Repeated source builds can leave tens of gigabytes of regenerable Cargo objects on a node. Use the unified operator command to inspect and reclaim that space without deleting protocol state or restarting the service.

1. Inspect without changing the host

Run the read-only inventory first. It reports the legacy repository target, the isolated build root, the pinned Rust version, the protected binary SHA-256, and aggregate filesystem capacity.

cd /root/open/AeroNyx
./deploy/node/aeronyx-node.sh build-cache --repo-dir "$PWD"

2. Preview every removable entry

Review the dry-run line by line. The destructive path requires root plus an explicit --yes; no implicit cleanup runs during status or upgrade.

sudo ./deploy/node/aeronyx-node.sh prune-build-cache \
  --repo-dir "$PWD" \
  --dry-run

3. Run the confirmed prune

sudo ./deploy/node/aeronyx-node.sh prune-build-cache \
  --repo-dir "$PWD" \
  --yes

For a custom build root, export AERONYX_BUILD_TARGET_ROOT for all three commands so inventory and pruning resolve the same isolated target layout.

Protected and removable scope

• Protected: The command preserves the active target/release/aeronyx-server, the current pinned-toolchain/service cache, other release executables and rollback artifacts, /var/lib/aeronyx protocol data, /etc/aeronyx identity/configuration, and caches owned by other services.
• Regenerable: It removes only known regenerable Cargo outputs such as legacy debug or cross-target directories, deps, build, .fingerprint, incremental, .rlib, .rmeta, .d, and older toolchain targets for the same service.
• Execution guard: Before deletion, the command takes the install/upgrade deployment lock and verifies that the running systemd MainPID maps to the protected binary. It hashes that binary before and after pruning and fails if the hash changes.

Post-maintenance verification

Cache pruning does not restart the node and does not require draining active sessions. Afterward, confirm service health, startup readiness, discovery quorum, and two-hop proof freshness.

./deploy/node/aeronyx-node.sh status --repo-dir "$PWD"
./deploy/node/aeronyx-node.sh health --repo-dir "$PWD" --json

Production validation evidence — July 26, 2026

A controlled production run reclaimed 50,649,768 KiB; filesystem usage fell from 90% to 65%. The protected binary SHA-256, MainPID, and restart count remained unchanged. Startup checks stayed ready with zero failures, and two-hop proof status stayed ready and stable.

Privacy boundary: The inventory exposes local paths, sizes, toolchain version, binary hash, and aggregate capacity only. It never reads or prints encrypted payloads, MemChain records, keys, peer identities, routes, client addresses, DNS contents, destinations, or social-graph data.

Optional cognitive-worker provider operations

The SuperNode cognitive worker is optional and disabled by default. Enable it only on a processing node whose trust role is understood.

Minimal local-provider configuration

[memchain.supernode]
enabled = true

[[memchain.supernode.providers]]
name = "local-ollama"
type = "openai_compatible"
api_base = "http://127.0.0.1:11434/v1"
api_key = ""
model = "llama3.2"
max_tokens = 1000
temperature = 0.3

[memchain.supernode.routing]
fallback = "local-ollama"

[memchain.supernode.privacy]
default_level = "structured"
allow_full_for = []

api_key = "$ENV_VAR" is also supported. Keep secrets outside committed configuration.

OpenAI-compatible endpoint forms

All three forms resolve to the same Chat Completions endpoint:

• https://provider.example
• https://provider.example/v1
• https://provider.example/v1/chat/completions

For type = "anthropic", api_base may be omitted; the current runtime uses the official Anthropic Messages endpoint.

Failure and resource behavior

• Successful model responses are capped at 8 MiB before JSON parsing; non-success diagnostic bodies are capped at 64 KiB.
• Transport, API, parse, and empty-response failures place that provider in a 30-second monotonic cooldown instead of disabling it forever.
• HTTP 429 Retry-After is honored within a bounded 1-300 second window; an absent value uses 30 seconds.
• The router may use another healthy configured provider and automatically considers the failed provider again after its cooldown expires. No scheduled job is required.

Before restart, run aeronyx-server validate -c /etc/aeronyx/server.toml. The behavior described here is in the current source on main; an installed node must be rebuilt or upgraded to a containing release.

Privacy boundary: storage and relay nodes remain node-blind. The enabled cognitive worker and any external provider see the prompt allowed by the configured privacy level. Keep default_level = "structured"; leave allow_full_for empty unless full-content processing has explicit consent.