Shell Chain Testnet Operator Guide

This guide covers everything you need to run a shell-chain testnet node — from system requirements to monitoring and maintenance.

See also: Quickstart Guide · JSON-RPC API Reference · Post-Quantum Cryptography Guide · Prover Guide · Consensus Details · Block Pruning & Compression


Table of Contents

  1. System Requirements
  2. Installation from Source
  3. Docker Deployment
  4. Public Testnet Deployment
  5. Configuration
  6. Key Generation
  7. Genesis Initialization
  8. Starting a Node
  9. Health & Readiness Endpoints
  10. Nginx Reverse Proxy
  11. JSON Logging
  12. Monitoring Setup
  13. Upgrading and Maintenance
  14. Troubleshooting

System Requirements

Minimum (single validator)

Resource Requirement
OS Linux (Ubuntu 22.04+), macOS 13+, or Windows with WSL2
CPU 2 cores (x86_64 or ARM64)
RAM 4 GB
Disk 40 GB SSD (archive mode requires more over time)
Network 10 Mbps, port 30303/tcp open for P2P
Resource Recommendation
CPU 4+ cores
RAM 8 GB
Disk 200 GB NVMe SSD
Network 100 Mbps, static IP, ports 30303/tcp (P2P), 8545/tcp (RPC), 9090/tcp (metrics)

Software Prerequisites

  • Rust 1.75+ (with cargo)
  • Git
  • Docker & Docker Compose (for containerized deployment)
  • RocksDB system library (for persistent storage backend)

Installation from Source

1. Install the Rust toolchain

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o rustup-init.sh
less rustup-init.sh
sh rustup-init.sh
rm rustup-init.sh
source $HOME/.cargo/env
rustup update stable

2. Clone the repository

git clone https://github.com/ShellDAO/shell-chain.git
cd shell-chain

3. Build the release binary

cargo build --release -p shell-cli --bin shell-node --features libp2p

The libp2p feature is required for the --p2p validator and RPC-only node commands in this guide. A build without it supports local operation but rejects --p2p.

The binary is produced at target/release/shell-node.

Optionally install it system-wide:

cp target/release/shell-node /usr/local/bin/

4. Verify the installation

shell-node --version

Docker Deployment (Legacy Reference)

⚠️ Note: The Docker Compose setup below is provided as a reference. The public testnet and production deployments now use systemd-managed bare-metal topology. For new deployments, follow the Starting a Node section with a systemd unit.

Shell Chain ships with a Docker Compose file (docker-compose.prod.yml) that can deploy:

  • 3 validator nodes (node1, node2, node3)
  • 1 RPC-only node (rpc-node)
  • Prometheus metrics collector
  • Grafana dashboard

Quick start with Docker

# Copy and edit environment variables
cp .env.example .env
# Edit .env to set GF_ADMIN_PASSWORD for Grafana

# Start the full stack
docker compose -f docker-compose.prod.yml up -d

Service details

Service Ports Role
node1 8545 (RPC), 8546 (WS), 30303 (P2P) Genesis-creating validator
node2 8555 (RPC), 8556 (WS), 30304 (P2P) Validator
node3 8565 (RPC), 8566 (WS), 30305 (P2P) Validator
rpc-node 8548 (RPC), 8549 (WS), 30306 (P2P) Read-only RPC with debug/trace APIs
prometheus 127.0.0.1:9090 Metrics collection
grafana 3000 Dashboards (default password: changeme)

Resource limits

Each node container is limited to 2 CPUs and 2 GB RAM. Adjust in docker-compose.prod.yml if needed.

Health checks

All nodes use an eth_blockNumber JSON-RPC health check (10s interval, 5s timeout, 3 retries).

Networks

  • internal — Private bridge for inter-node and monitoring traffic.
  • frontend — Public-facing bridge; only rpc-node and grafana are attached.

Persistent volumes

node1-data, node2-data, node3-data, rpc-data, prometheus-data, grafana-data, and a shared genesis volume.


Public Testnet Deployment

Live public testnet: RPC https://testnet-rpc.shell.org · WSS wss://testnet-rpc.shell.org/ws · Faucet https://explorer.shell.org/faucet · Explorer https://explorer.shell.org Follow @orgShell for network updates.

Shell Chain testnet is deployed using systemd on bare-metal Linux. See Starting a Node for the recommended startup commands with --network testnet.

Joining the testnet

Set BOOTNODE to the complete bootstrap multiaddr supplied by the node operator, including its /p2p/<PEER_ID> suffix. For example, the address has the form /dns4/testnet.shell.org/tcp/30303/p2p/<PEER_ID>; replace <PEER_ID> with the operator's actual libp2p peer ID, not a validator account address.

For a bootstrap node you operate, start it with --log-level info and copy the Listening on .../p2p/... address from its startup log. If that address uses an unspecified or loopback host, use the node's reachable host when connecting from another machine while preserving its transport, port and peer ID. A host and port without the peer ID are rejected as a bootstrap address.

: "${BOOTNODE:?Set BOOTNODE to a full multiaddr ending in /p2p/PEER_ID}"
shell-node run \
  --datadir /var/lib/shell-chain \
  --network testnet \
  --keystore /etc/shell-chain/validator-key.json \
  --max-idle-interval 0 \
  --p2p \
  --bootnode "$BOOTNODE" \
  --rpc-addr 0.0.0.0:8545 \
  --ws --ws-port 8546 \
  --log-format json

Verify the node is running

# Check block height
curl -s http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

v0.24.x Upgrade Notes

⚠️ Breaking change: v0.24.0 changes the BLAKE3 wire-format. RocksDB data written by v0.23.x is incompatible and requires a fresh genesis on upgrade. Back up any state you need before upgrading.

# Stop the node
sudo systemctl stop shell-node

# Pull and build
cd shell-chain && git pull origin main
cargo build --release -p shell-cli --bin shell-node --features libp2p
cp target/release/shell-node /usr/local/bin/

# Wipe old data and re-init from updated genesis
shell-node removedb --datadir /var/lib/shell-chain --force
shell-node init --genesis /etc/shell-chain/genesis.json --datadir /var/lib/shell-chain

# Restart
sudo systemctl start shell-node

Health & Readiness Endpoints

Shell Chain exposes HTTP health and readiness probes on the metrics port (default: 9090). These are designed for use with container orchestrators (Docker, Kubernetes) and load balancers.

GET /health — Liveness Probe

Returns node liveness status. Use this for Docker HEALTHCHECK or Kubernetes livenessProbe.

curl http://localhost:9090/health

Response (200 OK):

{"status":"ok","version":"0.27.4","block_height":1234,"syncing":false,"production_ready":true}

The node is considered alive if the process is running and can respond to HTTP requests.

GET /ready — Readiness Probe

Returns whether the node is ready to serve traffic. Use this for Kubernetes readinessProbe or load balancer health checks.

curl http://localhost:9090/ready

Response (200 OK) — Ready:

{"ready":true}

Response (503 Service Unavailable) — Not Ready:

{"ready":false,"reason":"block synchronization is in progress"}

The node reports not-ready until it has imported or produced a block, while block sync is active, and whenever the consensus production gate is closed. Liveness may succeed before readiness during startup; wait for readiness before routing traffic to a validator.

Using with Docker Compose

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:9090/health"]
  interval: 10s
  timeout: 5s
  retries: 3

Nginx Reverse Proxy

For production deployments, place an nginx reverse proxy in front of the RPC endpoint. Shell Chain includes a reference configuration at docker/nginx.conf.

Reference configuration

The provided docker/nginx.conf handles:

  • TLS termination (configure your certificates)
  • Rate limiting for RPC requests
  • Proxy headers (X-Real-IP, X-Forwarded-For)
  • WebSocket upgrade for ws:// connections
  • CORS headers

Example setup

curl https://testnet-rpc.shell.org \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Manual nginx setup

If running nginx outside Docker, point it at the RPC endpoint:

upstream shell_rpc {
    server 127.0.0.1:8545;
}

server {
    listen 80;
    server_name testnet.shell.org;

    location / {
        proxy_pass http://shell_rpc;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

JSON Logging

Shell Chain supports structured JSON logging for integration with log aggregation systems (ELK, Loki, Datadog, etc.).

Enable JSON logging

shell-node run --log-format json --log-level info ...

Or in TOML config:

[logging]
format = "json"
level = "info"

JSON log format

Each log line is a JSON object:

{"timestamp":"2025-01-15T10:30:00Z","level":"INFO","target":"shell_node::consensus","message":"Block produced","block_number":1234,"block_hash":"0xabc...","tx_count":5}

Log rotation

When writing logs to a file, use logrotate or a similar tool:

shell-node run --config config.toml --log-format json 2>&1 | \
  tee -a /var/log/shell-chain/node.log

Configuration

Shell Chain is configured via a TOML file, CLI flags, or both. CLI flags always override config file values.

Reference configs are provided in examples/:

  • examples/config-validator.toml — Full validator node
  • examples/config-rpc.toml — Read-only RPC node

TOML config structure

[node]
datadir = "/data"               # Data directory for chain storage and keystore
chain_id = 10                   # Chain ID
block_time = 2000               # Block production interval (milliseconds)
keystore = "/data/keystore.json" # Path to encrypted keystore file (validators only)
db = "rocksdb"                  # Storage backend: "memory" or "rocksdb" (auto-selected by --network if omitted)
pruning = 0                     # State roots to retain (0 = archive mode)
node_role = "validator"         # Node role: "validator", "validator-prover", or "prover"

[rpc]
listen_addr = "0.0.0.0:8545"   # JSON-RPC HTTP listen address
ws_enabled = true               # Enable WebSocket RPC
ws_port = 8546                  # WebSocket listen port
cors_origins = ["*"]            # CORS allowed origins (⚠️ restrict in production!)
rate_limit = 100                # Max RPC requests per second per connection
api_modules = ["eth", "net", "web3", "shell"]  # Enabled API namespaces

[p2p]
enabled = true                  # Enable libp2p networking
listen_addr = "0.0.0.0:30303"  # P2P listen address
bootnodes = []                  # Bootstrap peer multiaddrs
enable_mdns = true              # mDNS local peer discovery (disable in cloud)

[consensus]
engine = "wpoa"                 # Consensus engine (Weighted Proof of Authority)
enable_stark_aggregation = false  # Enable async STARK proof generation (see PROVER_GUIDE.md)

[prover]
max_concurrent_proofs = 1       # Parallel proof jobs (validator-prover / prover roles only)
proving_priority = "sequential" # "sequential" or "latest-first"

[metrics]
enabled = true                  # Enable Prometheus metrics
listen_addr = "0.0.0.0:9090"   # Metrics HTTP endpoint

[logging]
level = "info"                  # Log level (trace, debug, info, warn, error)
format = "json"                 # Log format: "text" or "json"

Node roles

Shell Chain supports three operational roles, set via node.node_role or --node-role:

Role Block production Proves blocks Use case
validator (default) ✅ ❌ Standard block-producing authority node
validator-prover ✅ ✅ (idle slots) Validator that also contributes proof work
prover ❌ ✅ (full time) Standalone prover — no keys needed

A prover node syncs the chain, generates ProofAmendment proofs, and propagates them via P2P without producing blocks. See PROVER_GUIDE.md for details.

Validator vs RPC configuration differences

Setting Validator RPC Node
node.keystore Required (path to keystore) Omit (no block production)
node.block_time Set (e.g., 2000) Not needed
rpc.api_modules ["eth", "net", "web3", "shell"] ["eth", "net", "web3", "shell", "debug", "trace"]
rpc.rate_limit 100 50 (lower to protect from abuse)

CLI flag reference (for shell-node run)

Flag Default Description
--config <PATH> — Path to TOML config file
--datadir <PATH> shell-data Data directory
--rpc-addr <ADDR> 127.0.0.1:8545 RPC listen address
--block-time <MS> 2000 Block interval (ms)
--keystore <PATH> — Keystore file path
--chain-id <ID> 10 Chain ID
--db <BACKEND> network-dependent memory for --network dev; rocksdb for testnet/mainnet. Override explicitly with --db rocksdb or --db memory.
--max-idle-interval <SECS> 600 Seconds of empty mempool before producing a heartbeat block. Set to 0 for continuous production.
--ws disabled Enable WebSocket RPC
--ws-port <PORT> 8546 WebSocket port
--p2p disabled Enable P2P networking
--p2p-addr <ADDR> 0.0.0.0:30303 P2P listen address
--bootnode <MULTIADDR> — Bootstrap peer (repeatable)
--bootnodes <ADDRS> — Comma-separated bootstrap peers
--enable-mdns disabled Enable mDNS discovery
--pruning <N> 0 State roots retained (0 = archive)
--node-role <ROLE> validator validator, validator-prover, or prover
--enable-stark-aggregation profile default Enable STARK proof aggregation
--max-concurrent-proofs <N> 1 Parallel proof jobs (prover roles only)
--checkpoint-url <URL> — Checkpoint sync URL
--rpc-cors <ORIGINS> — CORS allowed origins
--rpc-rate-limit <N> — Rate limit (req/s)
--rpc-api <NS> — API namespaces (comma-separated)
--metrics-addr <ADDR> — Metrics listen address
--log-format <FMT> text text or json
--log-level <LEVEL> info Log level (overrides RUST_LOG)

Key Generation

Shell Chain uses ML-DSA-65 as its primary post-quantum signature scheme (see PQ Crypto Guide). Generate a keypair:

shell-node key generate --output my-validator-key.json
chmod 600 my-validator-key.json

You will be prompted for an encryption password. The command creates an encrypted keystore file containing:

  • ML-DSA-65 public key (1,952 bytes)
  • Encrypted secret key (4,032 bytes, encrypted with XChaCha20-Poly1305)
  • Derived 32-byte (0x + 64 hex) Shell address

Inspect a keystore

shell-node key inspect my-validator-key.json

This displays the address associated with the keystore without requiring the password.

Important: Back up your keystore file and password securely. There is no key recovery mechanism.


Genesis Initialization

1. Create a genesis.json

{
  "chain_id": 10,
  "chain_name": "shell-testnet",
  "timestamp": 1735689600,
  "gas_limit": 30000000,
  "extra_data": "shell-genesis",
  "consensus": {
    "engine": "wpoa",
    "authorities": [
      "0x<YOUR_VALIDATOR_ADDRESS_64_HEX>"
    ],
    "authority_pubkeys": [
      "0x<YOUR_VALIDATOR_PUBLIC_KEY_HEX>"
    ],
    "block_time_secs": 2,
    "epoch_length": 0
  },
  "alloc": {
    "0x<YOUR_VALIDATOR_ADDRESS_64_HEX>": {
      "balance": "0x3635c9adc5dea00000"
    }
  },
  "boot_nodes": []
}

The authorities array lists the initial validator addresses (derived from keystores). The authority_pubkeys array contains their full PQ public keys in the same order. Followers need these keys to verify block seals from their first import; an address alone is insufficient. The alloc section pre-funds accounts with an initial balance (in wei, hex-encoded).

Read the full public key for the corresponding validator without decrypting its secret key:

python3 - <<'PYKEY'
import json
from pathlib import Path

key = json.loads(Path("my-validator-key.json").read_text())
print("0x" + key["public_key"].removeprefix("0x"))
PYKEY

Replace both placeholders with the matching address and public key. Every node in a private network must use the same genesis file. When joining an existing public network, use its operator-supplied genesis rather than a new local one.

2. Initialize the data directory

shell-node init --genesis genesis.json --chain-id 10 --datadir shell-data

This creates the data directory structure and writes the genesis block.


Starting a Node

Validator node

A validator requires a keystore and produces blocks:

shell-node run \
  --config examples/config-validator.toml \
  --keystore my-validator-key.json \
  --datadir shell-data \
  --network testnet \
  --max-idle-interval 0 \
  --p2p \
  --p2p-addr 0.0.0.0:30303 \
  --rpc-addr 0.0.0.0:8545 \
  --ws --ws-port 8546 \
  --rpc-api eth,net,web3,shell \
  --metrics-addr 0.0.0.0:9090 \
  --log-format json

--network testnet automatically selects rocksdb as the storage backend and the correct chain defaults. You can override storage explicitly with --db rocksdb. You will be prompted for the keystore password on startup.

⚠️ Security: Binding to 0.0.0.0 exposes RPC to all network interfaces. In production, use a reverse proxy (nginx/caddy) with TLS and firewall rules to restrict access. For local-only access, use 127.0.0.1:8545.

RPC-only node

An RPC node syncs the chain but does not produce blocks. Omit --keystore and set BOOTNODE to the validator's complete multiaddr, including its peer ID:

: "${BOOTNODE:?Set BOOTNODE to a full multiaddr ending in /p2p/PEER_ID}"
shell-node run \
  --config examples/config-rpc.toml \
  --datadir shell-data \
  --network testnet \
  --p2p \
  --p2p-addr 0.0.0.0:30303 \
  --bootnode "$BOOTNODE" \
  --rpc-addr 0.0.0.0:8545 \
  --ws --ws-port 8546 \
  --rpc-api eth,net,web3,shell,debug,trace \
  --rpc-cors "*" \
  --rpc-rate-limit 50 \
  --metrics-addr 0.0.0.0:9090

Running as a systemd service

Create /etc/systemd/system/shell-node.service:

[Unit]
Description=Shell Chain Node
After=network.target

[Service]
Type=simple
User=shellchain
ExecStart=/usr/local/bin/shell-node run \
  --config /etc/shell-chain/config.toml \
  --datadir /var/lib/shell-chain
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now shell-node

Monitoring Setup

Shell Chain exposes Prometheus metrics when started with --metrics-addr. The monitoring/ directory provides a ready-to-use stack.

Prometheus

Scrape config (monitoring/prometheus.yml):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "shell-chain"
    static_configs:
      - targets:
          - "node1:9090"
          - "node2:9090"
          - "node3:9090"
          - "rpc-node:9090"
        labels:
          network: "shell-prod"
    metrics_path: "/metrics"

Grafana dashboard

The pre-built Shell Chain Overview dashboard (monitoring/grafana/dashboards/shell-chain.json) includes 6 panels:

Panel Metric Description
Block Height shell_block_height Current block number per node
Blocks Imported rate(shell_blocks_imported_total[1m]) Blocks/sec import rate
Transactions Received rate(shell_txs_received_total[1m]) Successful pool admissions/sec
Peer Count shell_peer_count Connected peers per node
Mempool Size shell_tx_pool_size Pending transactions
Block Production Latency shell_block_production_duration_seconds Avg and p95 latency

shell_txs_received_total counts successful admissions to the shared transaction pool from RPC, P2P, or reinsertions after a reorganization. Replacements count as new admissions; duplicate and invalid submissions do not. It resets when the node process restarts. shell_tx_pool_size is sampled from the current pool on every scrape, including before the next block. These corrections are in the current source; older binaries may omit RPC admissions and retain a stale pool size until a block or P2P transaction arrives.

To verify the metrics with a fresh isolated development node (Python 3 required):

cargo build -p shell-cli --features rocksdb
python3 tests/e2e/operator-monitoring.py

Run these commands from a current node source checkout. The script creates a temporary validator, submits a signed transaction, checks pending metrics, inclusion and readiness, then restarts the persistent node. It stops its node and prints the retained result directory on success or failure.

Access

  • Prometheus: http://localhost:9090
  • Grafana: http://localhost:3000 (default login: admin / password from GF_ADMIN_PASSWORD env var, default changeme)

Key metrics to alert on

  • shell_block_height not increasing → block production stalled
  • shell_peer_count == 0 → network isolation
  • shell_tx_pool_size growing unboundedly → mempool congestion
  • shell_block_production_duration_seconds p95 > block_time → node falling behind

Upgrading and Maintenance

Upgrading from source

cd shell-chain
git pull origin main
cargo build --release -p shell-cli --bin shell-node --features libp2p
# Stop the running node, replace the binary, restart
sudo systemctl restart shell-node

Upgrading Docker deployment

cd shell-chain
git pull origin main
docker compose -f docker-compose.prod.yml build
docker compose -f docker-compose.prod.yml up -d

Database management

Stop the node before opening its database with these commands. Snapshots use the persistent RocksDB database in the selected data directory; an in-memory node does not provide a persistent database to export.

Export state snapshot:

shell-node export-state --datadir shell-data --output snapshot.jsonl
# Optionally assert the expected canonical head (only when the head is 1000):
shell-node export-state --datadir shell-data --block 1000 --output snapshot.jsonl

--block must match the current canonical head. It does not select an arbitrary historical state: exporting a retained block below the head is rejected.

Import state snapshot:

Restore into a new data directory. Use the same trusted genesis.json and chain ID as the source network; the example below uses this guide's chain ID 10. Initialize the directory, then import before starting, syncing, or mining:

shell-node init --genesis genesis.json --chain-id 10 --datadir shell-data-restored
shell-node import-state --datadir shell-data-restored --snapshot snapshot.jsonl

Import verifies the checksum, chain identity, canonical head, and non-empty head state trie before publishing the restored height. It rejects an existing canonical head, a destination without trusted chain configuration, and a snapshot from a different genesis. Do not remove the source database to make an import succeed.

Start the restored node with --datadir shell-data-restored and the same network configuration as the source, using RocksDB. Compare its restored block hash, state root, account balances, nonce, and transaction receipts with the source snapshot height before allowing new blocks; then verify normal transaction processing resumes.

Remove the database:

shell-node removedb --datadir shell-data --force

Log rotation

When using --log-format json, pipe logs to a file and use logrotate or similar:

shell-node run --config config.toml --log-format json 2>&1 | \
  tee -a /var/log/shell-chain/node.log

Troubleshooting

Node won't start

Symptom Cause Solution
"genesis not initialized" Missing init step Run shell-node init --genesis genesis.json
"keystore not found" Wrong --keystore path Verify the path to your keystore file
"address in use" Port already bound Check for conflicting processes on 8545, 30303, or 9090
"wrong password" Incorrect keystore password Re-enter password; check for leading/trailing whitespace

No blocks produced

  • Verify your address is in the authorities list in genesis.json.
  • Check that --keystore is provided and the password is correct.
  • Confirm shell_mining returns true via eth_mining RPC call:
curl -s http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_mining","params":[],"id":1}'

Peers not connecting

  • Ensure port 30303/tcp is open and reachable.
  • Verify --p2p flag is enabled.
  • Check --bootnode multiaddr is correct (format: /dns4/<host>/tcp/30303/p2p/<PEER_ID> or /ip4/<ip>/tcp/30303/p2p/<PEER_ID>).
  • In cloud environments, disable mDNS (enable_mdns = false) and use explicit bootnodes.

RPC not responding

  • Confirm --rpc-addr is bound to 0.0.0.0 (not 127.0.0.1) if accessing remotely.
  • Check CORS settings with --rpc-cors if calling from a browser.
  • Verify the requested API namespace is enabled via --rpc-api.

High memory usage

  • Switch from --db memory to --db rocksdb for production.
  • Enable pruning: --pruning 1000 retains only the last 1,000 state roots.
  • Check mempool size via shell_pendingCount RPC.

Metrics not appearing in Grafana

  • Verify --metrics-addr is set and the port is reachable from Prometheus.
  • Check Prometheus targets at http://localhost:9090/targets.
  • Ensure the monitoring/prometheus.yml targets list matches your node addresses.

Last updated: 2026-06-17