Shell Chain Testnet Operator Guide
This guide covers everything you need to run a shell-chain testnet node — from system requirements to monitoring and maintenance.
See also: Quickstart Guide · JSON-RPC API Reference · Post-Quantum Cryptography Guide · Prover Guide · Consensus Details · Block Pruning & Compression
Table of Contents
- System Requirements
- Installation from Source
- Docker Deployment
- Public Testnet Deployment
- Configuration
- Key Generation
- Genesis Initialization
- Starting a Node
- Health & Readiness Endpoints
- Nginx Reverse Proxy
- JSON Logging
- Monitoring Setup
- Upgrading and Maintenance
- Troubleshooting
System Requirements
Minimum (single validator)
| Resource | Requirement |
|---|---|
| OS | Linux (Ubuntu 22.04+), macOS 13+, or Windows with WSL2 |
| CPU | 2 cores (x86_64 or ARM64) |
| RAM | 4 GB |
| Disk | 40 GB SSD (archive mode requires more over time) |
| Network | 10 Mbps, port 30303/tcp open for P2P |
Recommended (production validator)
| Resource | Recommendation |
|---|---|
| CPU | 4+ cores |
| RAM | 8 GB |
| Disk | 200 GB NVMe SSD |
| Network | 100 Mbps, static IP, ports 30303/tcp (P2P), 8545/tcp (RPC), 9090/tcp (metrics) |
Software Prerequisites
- Rust 1.75+ (with
cargo) - Git
- Docker & Docker Compose (for containerized deployment)
- RocksDB system library (for persistent storage backend)
Installation from Source
1. Install the Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o rustup-init.sh
less rustup-init.sh
sh rustup-init.sh
rm rustup-init.sh
source $HOME/.cargo/env
rustup update stable
2. Clone the repository
git clone https://github.com/ShellDAO/shell-chain.git
cd shell-chain
3. Build the release binary
cargo build --release -p shell-cli --bin shell-node --features libp2p
The libp2p feature is required for the --p2p validator and RPC-only node
commands in this guide. A build without it supports local operation but rejects
--p2p.
The binary is produced at target/release/shell-node.
Optionally install it system-wide:
cp target/release/shell-node /usr/local/bin/
4. Verify the installation
shell-node --version
Docker Deployment (Legacy Reference)
⚠️ Note: The Docker Compose setup below is provided as a reference. The public testnet and production deployments now use systemd-managed bare-metal topology. For new deployments, follow the Starting a Node section with a systemd unit.
Shell Chain ships with a Docker Compose file (docker-compose.prod.yml) that can deploy:
- 3 validator nodes (node1, node2, node3)
- 1 RPC-only node (rpc-node)
- Prometheus metrics collector
- Grafana dashboard
Quick start with Docker
# Copy and edit environment variables
cp .env.example .env
# Edit .env to set GF_ADMIN_PASSWORD for Grafana
# Start the full stack
docker compose -f docker-compose.prod.yml up -d
Service details
| Service | Ports | Role |
|---|---|---|
node1 |
8545 (RPC), 8546 (WS), 30303 (P2P) | Genesis-creating validator |
node2 |
8555 (RPC), 8556 (WS), 30304 (P2P) | Validator |
node3 |
8565 (RPC), 8566 (WS), 30305 (P2P) | Validator |
rpc-node |
8548 (RPC), 8549 (WS), 30306 (P2P) | Read-only RPC with debug/trace APIs |
prometheus |
127.0.0.1:9090 | Metrics collection |
grafana |
3000 | Dashboards (default password: changeme) |
Resource limits
Each node container is limited to 2 CPUs and 2 GB RAM. Adjust in docker-compose.prod.yml if needed.
Health checks
All nodes use an eth_blockNumber JSON-RPC health check (10s interval, 5s timeout, 3 retries).
Networks
internal— Private bridge for inter-node and monitoring traffic.frontend— Public-facing bridge; onlyrpc-nodeandgrafanaare attached.
Persistent volumes
node1-data, node2-data, node3-data, rpc-data, prometheus-data, grafana-data, and a shared genesis volume.
Public Testnet Deployment
Live public testnet: RPC
https://testnet-rpc.shell.org· WSSwss://testnet-rpc.shell.org/ws· Faucethttps://explorer.shell.org/faucet· Explorerhttps://explorer.shell.orgFollow @orgShell for network updates.
Shell Chain testnet is deployed using systemd on bare-metal Linux. See Starting a Node for the recommended startup commands with --network testnet.
Joining the testnet
Set BOOTNODE to the complete bootstrap multiaddr supplied by the node operator,
including its /p2p/<PEER_ID> suffix. For example, the address has the form
/dns4/testnet.shell.org/tcp/30303/p2p/<PEER_ID>; replace <PEER_ID> with the
operator's actual libp2p peer ID, not a validator account address.
For a bootstrap node you operate, start it with --log-level info and copy the
Listening on .../p2p/... address from its startup log. If that address uses an
unspecified or loopback host, use the node's reachable host when connecting from
another machine while preserving its transport, port and peer ID. A host and
port without the peer ID are rejected as a bootstrap address.
: "${BOOTNODE:?Set BOOTNODE to a full multiaddr ending in /p2p/PEER_ID}"
shell-node run \
--datadir /var/lib/shell-chain \
--network testnet \
--keystore /etc/shell-chain/validator-key.json \
--max-idle-interval 0 \
--p2p \
--bootnode "$BOOTNODE" \
--rpc-addr 0.0.0.0:8545 \
--ws --ws-port 8546 \
--log-format json
Verify the node is running
# Check block height
curl -s http://localhost:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
v0.24.x Upgrade Notes
⚠️ Breaking change: v0.24.0 changes the BLAKE3 wire-format. RocksDB data written by v0.23.x is incompatible and requires a fresh genesis on upgrade. Back up any state you need before upgrading.
# Stop the node
sudo systemctl stop shell-node
# Pull and build
cd shell-chain && git pull origin main
cargo build --release -p shell-cli --bin shell-node --features libp2p
cp target/release/shell-node /usr/local/bin/
# Wipe old data and re-init from updated genesis
shell-node removedb --datadir /var/lib/shell-chain --force
shell-node init --genesis /etc/shell-chain/genesis.json --datadir /var/lib/shell-chain
# Restart
sudo systemctl start shell-node
Health & Readiness Endpoints
Shell Chain exposes HTTP health and readiness probes on the metrics port (default: 9090). These are designed for use with container orchestrators (Docker, Kubernetes) and load balancers.
GET /health — Liveness Probe
Returns node liveness status. Use this for Docker HEALTHCHECK or Kubernetes livenessProbe.
curl http://localhost:9090/health
Response (200 OK):
{"status":"ok","version":"0.27.4","block_height":1234,"syncing":false,"production_ready":true}
The node is considered alive if the process is running and can respond to HTTP requests.
GET /ready — Readiness Probe
Returns whether the node is ready to serve traffic. Use this for Kubernetes readinessProbe or load balancer health checks.
curl http://localhost:9090/ready
Response (200 OK) — Ready:
{"ready":true}
Response (503 Service Unavailable) — Not Ready:
{"ready":false,"reason":"block synchronization is in progress"}
The node reports not-ready until it has imported or produced a block, while block sync is active, and whenever the consensus production gate is closed. Liveness may succeed before readiness during startup; wait for readiness before routing traffic to a validator.
Using with Docker Compose
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9090/health"]
interval: 10s
timeout: 5s
retries: 3
Nginx Reverse Proxy
For production deployments, place an nginx reverse proxy in front of the RPC endpoint. Shell Chain includes a reference configuration at docker/nginx.conf.
Reference configuration
The provided docker/nginx.conf handles:
- TLS termination (configure your certificates)
- Rate limiting for RPC requests
- Proxy headers (
X-Real-IP,X-Forwarded-For) - WebSocket upgrade for
ws://connections - CORS headers
Example setup
curl https://testnet-rpc.shell.org \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
Manual nginx setup
If running nginx outside Docker, point it at the RPC endpoint:
upstream shell_rpc {
server 127.0.0.1:8545;
}
server {
listen 80;
server_name testnet.shell.org;
location / {
proxy_pass http://shell_rpc;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
JSON Logging
Shell Chain supports structured JSON logging for integration with log aggregation systems (ELK, Loki, Datadog, etc.).
Enable JSON logging
shell-node run --log-format json --log-level info ...
Or in TOML config:
[logging]
format = "json"
level = "info"
JSON log format
Each log line is a JSON object:
{"timestamp":"2025-01-15T10:30:00Z","level":"INFO","target":"shell_node::consensus","message":"Block produced","block_number":1234,"block_hash":"0xabc...","tx_count":5}
Log rotation
When writing logs to a file, use logrotate or a similar tool:
shell-node run --config config.toml --log-format json 2>&1 | \
tee -a /var/log/shell-chain/node.log
Configuration
Shell Chain is configured via a TOML file, CLI flags, or both. CLI flags always override config file values.
Reference configs are provided in examples/:
examples/config-validator.toml— Full validator nodeexamples/config-rpc.toml— Read-only RPC node
TOML config structure
[node]
datadir = "/data" # Data directory for chain storage and keystore
chain_id = 10 # Chain ID
block_time = 2000 # Block production interval (milliseconds)
keystore = "/data/keystore.json" # Path to encrypted keystore file (validators only)
db = "rocksdb" # Storage backend: "memory" or "rocksdb" (auto-selected by --network if omitted)
pruning = 0 # State roots to retain (0 = archive mode)
node_role = "validator" # Node role: "validator", "validator-prover", or "prover"
[rpc]
listen_addr = "0.0.0.0:8545" # JSON-RPC HTTP listen address
ws_enabled = true # Enable WebSocket RPC
ws_port = 8546 # WebSocket listen port
cors_origins = ["*"] # CORS allowed origins (⚠️ restrict in production!)
rate_limit = 100 # Max RPC requests per second per connection
api_modules = ["eth", "net", "web3", "shell"] # Enabled API namespaces
[p2p]
enabled = true # Enable libp2p networking
listen_addr = "0.0.0.0:30303" # P2P listen address
bootnodes = [] # Bootstrap peer multiaddrs
enable_mdns = true # mDNS local peer discovery (disable in cloud)
[consensus]
engine = "wpoa" # Consensus engine (Weighted Proof of Authority)
enable_stark_aggregation = false # Enable async STARK proof generation (see PROVER_GUIDE.md)
[prover]
max_concurrent_proofs = 1 # Parallel proof jobs (validator-prover / prover roles only)
proving_priority = "sequential" # "sequential" or "latest-first"
[metrics]
enabled = true # Enable Prometheus metrics
listen_addr = "0.0.0.0:9090" # Metrics HTTP endpoint
[logging]
level = "info" # Log level (trace, debug, info, warn, error)
format = "json" # Log format: "text" or "json"
Node roles
Shell Chain supports three operational roles, set via node.node_role or --node-role:
| Role | Block production | Proves blocks | Use case |
|---|---|---|---|
validator (default) |
✅ | ❌ | Standard block-producing authority node |
validator-prover |
✅ | ✅ (idle slots) | Validator that also contributes proof work |
prover |
❌ | ✅ (full time) | Standalone prover — no keys needed |
A prover node syncs the chain, generates ProofAmendment proofs, and propagates them via P2P without producing blocks. See PROVER_GUIDE.md for details.
Validator vs RPC configuration differences
| Setting | Validator | RPC Node |
|---|---|---|
node.keystore |
Required (path to keystore) | Omit (no block production) |
node.block_time |
Set (e.g., 2000) |
Not needed |
rpc.api_modules |
["eth", "net", "web3", "shell"] |
["eth", "net", "web3", "shell", "debug", "trace"] |
rpc.rate_limit |
100 |
50 (lower to protect from abuse) |
CLI flag reference (for shell-node run)
| Flag | Default | Description |
|---|---|---|
--config <PATH> |
— | Path to TOML config file |
--datadir <PATH> |
shell-data |
Data directory |
--rpc-addr <ADDR> |
127.0.0.1:8545 |
RPC listen address |
--block-time <MS> |
2000 |
Block interval (ms) |
--keystore <PATH> |
— | Keystore file path |
--chain-id <ID> |
10 |
Chain ID |
--db <BACKEND> |
network-dependent | memory for --network dev; rocksdb for testnet/mainnet. Override explicitly with --db rocksdb or --db memory. |
--max-idle-interval <SECS> |
600 |
Seconds of empty mempool before producing a heartbeat block. Set to 0 for continuous production. |
--ws |
disabled | Enable WebSocket RPC |
--ws-port <PORT> |
8546 |
WebSocket port |
--p2p |
disabled | Enable P2P networking |
--p2p-addr <ADDR> |
0.0.0.0:30303 |
P2P listen address |
--bootnode <MULTIADDR> |
— | Bootstrap peer (repeatable) |
--bootnodes <ADDRS> |
— | Comma-separated bootstrap peers |
--enable-mdns |
disabled | Enable mDNS discovery |
--pruning <N> |
0 |
State roots retained (0 = archive) |
--node-role <ROLE> |
validator |
validator, validator-prover, or prover |
--enable-stark-aggregation |
profile default | Enable STARK proof aggregation |
--max-concurrent-proofs <N> |
1 |
Parallel proof jobs (prover roles only) |
--checkpoint-url <URL> |
— | Checkpoint sync URL |
--rpc-cors <ORIGINS> |
— | CORS allowed origins |
--rpc-rate-limit <N> |
— | Rate limit (req/s) |
--rpc-api <NS> |
— | API namespaces (comma-separated) |
--metrics-addr <ADDR> |
— | Metrics listen address |
--log-format <FMT> |
text |
text or json |
--log-level <LEVEL> |
info |
Log level (overrides RUST_LOG) |
Key Generation
Shell Chain uses ML-DSA-65 as its primary post-quantum signature scheme (see PQ Crypto Guide). Generate a keypair:
shell-node key generate --output my-validator-key.json
chmod 600 my-validator-key.json
You will be prompted for an encryption password. The command creates an encrypted keystore file containing:
- ML-DSA-65 public key (1,952 bytes)
- Encrypted secret key (4,032 bytes, encrypted with XChaCha20-Poly1305)
- Derived 32-byte (
0x+ 64 hex) Shell address
Inspect a keystore
shell-node key inspect my-validator-key.json
This displays the address associated with the keystore without requiring the password.
Important: Back up your keystore file and password securely. There is no key recovery mechanism.
Genesis Initialization
1. Create a genesis.json
{
"chain_id": 10,
"chain_name": "shell-testnet",
"timestamp": 1735689600,
"gas_limit": 30000000,
"extra_data": "shell-genesis",
"consensus": {
"engine": "wpoa",
"authorities": [
"0x<YOUR_VALIDATOR_ADDRESS_64_HEX>"
],
"authority_pubkeys": [
"0x<YOUR_VALIDATOR_PUBLIC_KEY_HEX>"
],
"block_time_secs": 2,
"epoch_length": 0
},
"alloc": {
"0x<YOUR_VALIDATOR_ADDRESS_64_HEX>": {
"balance": "0x3635c9adc5dea00000"
}
},
"boot_nodes": []
}
The authorities array lists the initial validator addresses (derived from
keystores). The authority_pubkeys array contains their full PQ public keys in
the same order. Followers need these keys to verify block seals from their first
import; an address alone is insufficient. The alloc section pre-funds accounts
with an initial balance (in wei, hex-encoded).
Read the full public key for the corresponding validator without decrypting its secret key:
python3 - <<'PYKEY'
import json
from pathlib import Path
key = json.loads(Path("my-validator-key.json").read_text())
print("0x" + key["public_key"].removeprefix("0x"))
PYKEY
Replace both placeholders with the matching address and public key. Every node in a private network must use the same genesis file. When joining an existing public network, use its operator-supplied genesis rather than a new local one.
2. Initialize the data directory
shell-node init --genesis genesis.json --chain-id 10 --datadir shell-data
This creates the data directory structure and writes the genesis block.
Starting a Node
Validator node
A validator requires a keystore and produces blocks:
shell-node run \
--config examples/config-validator.toml \
--keystore my-validator-key.json \
--datadir shell-data \
--network testnet \
--max-idle-interval 0 \
--p2p \
--p2p-addr 0.0.0.0:30303 \
--rpc-addr 0.0.0.0:8545 \
--ws --ws-port 8546 \
--rpc-api eth,net,web3,shell \
--metrics-addr 0.0.0.0:9090 \
--log-format json
--network testnet automatically selects rocksdb as the storage backend and the correct chain defaults. You can override storage explicitly with --db rocksdb.
You will be prompted for the keystore password on startup.
⚠️ Security: Binding to
0.0.0.0exposes RPC to all network interfaces. In production, use a reverse proxy (nginx/caddy) with TLS and firewall rules to restrict access. For local-only access, use127.0.0.1:8545.
RPC-only node
An RPC node syncs the chain but does not produce blocks. Omit --keystore
and set BOOTNODE to the validator's complete multiaddr, including its peer ID:
: "${BOOTNODE:?Set BOOTNODE to a full multiaddr ending in /p2p/PEER_ID}"
shell-node run \
--config examples/config-rpc.toml \
--datadir shell-data \
--network testnet \
--p2p \
--p2p-addr 0.0.0.0:30303 \
--bootnode "$BOOTNODE" \
--rpc-addr 0.0.0.0:8545 \
--ws --ws-port 8546 \
--rpc-api eth,net,web3,shell,debug,trace \
--rpc-cors "*" \
--rpc-rate-limit 50 \
--metrics-addr 0.0.0.0:9090
Running as a systemd service
Create /etc/systemd/system/shell-node.service:
[Unit]
Description=Shell Chain Node
After=network.target
[Service]
Type=simple
User=shellchain
ExecStart=/usr/local/bin/shell-node run \
--config /etc/shell-chain/config.toml \
--datadir /var/lib/shell-chain
Restart=on-failure
RestartSec=5
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now shell-node
Monitoring Setup
Shell Chain exposes Prometheus metrics when started with --metrics-addr. The monitoring/ directory provides a ready-to-use stack.
Prometheus
Scrape config (monitoring/prometheus.yml):
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "shell-chain"
static_configs:
- targets:
- "node1:9090"
- "node2:9090"
- "node3:9090"
- "rpc-node:9090"
labels:
network: "shell-prod"
metrics_path: "/metrics"
Grafana dashboard
The pre-built Shell Chain Overview dashboard (monitoring/grafana/dashboards/shell-chain.json) includes 6 panels:
| Panel | Metric | Description |
|---|---|---|
| Block Height | shell_block_height |
Current block number per node |
| Blocks Imported | rate(shell_blocks_imported_total[1m]) |
Blocks/sec import rate |
| Transactions Received | rate(shell_txs_received_total[1m]) |
Successful pool admissions/sec |
| Peer Count | shell_peer_count |
Connected peers per node |
| Mempool Size | shell_tx_pool_size |
Pending transactions |
| Block Production Latency | shell_block_production_duration_seconds |
Avg and p95 latency |
shell_txs_received_total counts successful admissions to the shared transaction
pool from RPC, P2P, or reinsertions after a reorganization. Replacements count as
new admissions; duplicate and invalid submissions do not. It resets when the
node process restarts. shell_tx_pool_size is sampled from the current pool on
every scrape, including before the next block. These corrections are in the
current source; older binaries may omit RPC admissions and retain a stale pool
size until a block or P2P transaction arrives.
To verify the metrics with a fresh isolated development node (Python 3 required):
cargo build -p shell-cli --features rocksdb
python3 tests/e2e/operator-monitoring.py
Run these commands from a current node source checkout. The script creates a temporary validator, submits a signed transaction, checks pending metrics, inclusion and readiness, then restarts the persistent node. It stops its node and prints the retained result directory on success or failure.
Access
- Prometheus:
http://localhost:9090 - Grafana:
http://localhost:3000(default login:admin/ password fromGF_ADMIN_PASSWORDenv var, defaultchangeme)
Key metrics to alert on
shell_block_heightnot increasing → block production stalledshell_peer_count == 0→ network isolationshell_tx_pool_sizegrowing unboundedly → mempool congestionshell_block_production_duration_secondsp95 > block_time → node falling behind
Upgrading and Maintenance
Upgrading from source
cd shell-chain
git pull origin main
cargo build --release -p shell-cli --bin shell-node --features libp2p
# Stop the running node, replace the binary, restart
sudo systemctl restart shell-node
Upgrading Docker deployment
cd shell-chain
git pull origin main
docker compose -f docker-compose.prod.yml build
docker compose -f docker-compose.prod.yml up -d
Database management
Stop the node before opening its database with these commands. Snapshots use the persistent RocksDB database in the selected data directory; an in-memory node does not provide a persistent database to export.
Export state snapshot:
shell-node export-state --datadir shell-data --output snapshot.jsonl
# Optionally assert the expected canonical head (only when the head is 1000):
shell-node export-state --datadir shell-data --block 1000 --output snapshot.jsonl
--block must match the current canonical head. It does not select an arbitrary
historical state: exporting a retained block below the head is rejected.
Import state snapshot:
Restore into a new data directory. Use the same trusted genesis.json and chain
ID as the source network; the example below uses this guide's chain ID 10.
Initialize the directory, then import before starting, syncing, or mining:
shell-node init --genesis genesis.json --chain-id 10 --datadir shell-data-restored
shell-node import-state --datadir shell-data-restored --snapshot snapshot.jsonl
Import verifies the checksum, chain identity, canonical head, and non-empty head state trie before publishing the restored height. It rejects an existing canonical head, a destination without trusted chain configuration, and a snapshot from a different genesis. Do not remove the source database to make an import succeed.
Start the restored node with --datadir shell-data-restored and the same network
configuration as the source, using RocksDB. Compare its restored block hash,
state root, account balances, nonce, and transaction receipts with the source
snapshot height before allowing new blocks; then verify normal transaction
processing resumes.
Remove the database:
shell-node removedb --datadir shell-data --force
Log rotation
When using --log-format json, pipe logs to a file and use logrotate or similar:
shell-node run --config config.toml --log-format json 2>&1 | \
tee -a /var/log/shell-chain/node.log
Troubleshooting
Node won't start
| Symptom | Cause | Solution |
|---|---|---|
| "genesis not initialized" | Missing init step |
Run shell-node init --genesis genesis.json |
| "keystore not found" | Wrong --keystore path |
Verify the path to your keystore file |
| "address in use" | Port already bound | Check for conflicting processes on 8545, 30303, or 9090 |
| "wrong password" | Incorrect keystore password | Re-enter password; check for leading/trailing whitespace |
No blocks produced
- Verify your address is in the
authoritieslist in genesis.json. - Check that
--keystoreis provided and the password is correct. - Confirm
shell_miningreturnstrueviaeth_miningRPC call:
curl -s http://localhost:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_mining","params":[],"id":1}'
Peers not connecting
- Ensure port 30303/tcp is open and reachable.
- Verify
--p2pflag is enabled. - Check
--bootnodemultiaddr is correct (format:/dns4/<host>/tcp/30303/p2p/<PEER_ID>or/ip4/<ip>/tcp/30303/p2p/<PEER_ID>). - In cloud environments, disable mDNS (
enable_mdns = false) and use explicit bootnodes.
RPC not responding
- Confirm
--rpc-addris bound to0.0.0.0(not127.0.0.1) if accessing remotely. - Check CORS settings with
--rpc-corsif calling from a browser. - Verify the requested API namespace is enabled via
--rpc-api.
High memory usage
- Switch from
--db memoryto--db rocksdbfor production. - Enable pruning:
--pruning 1000retains only the last 1,000 state roots. - Check mempool size via
shell_pendingCountRPC.
Metrics not appearing in Grafana
- Verify
--metrics-addris set and the port is reachable from Prometheus. - Check Prometheus targets at
http://localhost:9090/targets. - Ensure the
monitoring/prometheus.ymltargets list matches your node addresses.
Last updated: 2026-06-17