Running an Archive Server

Please see The Data Waterfall for an overview of the different sources of archive data. This page is dedicated to operational details on running an Archive Server.

Recommended footprint

For anyone looking to reliably serve historical transactional data, it is recommended to run multiple Archive Servers, fed by multiple full nodes running the Archive Writer process.

Suggested configuration:

• 2 Archive Servers running MongoDB + monad-indexer
• 2 Archive Writer nodes, aka full node + monad-archiver
• Many full nodes serving RPC requests, connected to both Archive Servers for historical transactional data

Recommended Archive Server host specs

• CPU: 16 cores
• RAM: minimum 64GB; prefer >512GB for best performance
• Storage: minimum recommended 16TB; preferred 32TB+
  • NVMe SSD
  • RAID 10
  • Minimum 10K IOPS sustained
• Network: >1GbE, scale higher as needed when serving more RPC servers

Fresh installation

These instructions assume an existing full node (for Archive Writer) and a new host (for Archive Server).

On your Archive Server:

1. Add the following section to /home/monad/.env

   # Replace <db username> and <db pwd> with your actual values
   DB_USER="<your db username>"
   DB_PWD="<your db pwd>"
   DB_VOLUME="~/archive-db-data"
   
   # Note: source and sink should generally be the same local MongoDB.
   # Indexer reads from:
   BLOCK_DATA_SOURCE="mongodb mongodb://${DB_USER}:${DB_PWD}@0.0.0.0:27017 archive-db"
   # Indexer writes to:
   ARCHIVE_SINK="mongodb mongodb://${DB_USER}:${DB_PWD}@0.0.0.0:27017 archive-db"
   OTEL_ENDPOINT="http://0.0.0.0:4317"
   

2. Run mongodb. Instructions show how to run in Docker, but operators can run however they choose.

   archive-db:
     image: mongo:latest
     command: mongod --bind_ip 0.0.0.0
     networks:
       - host
     environment:
       MONGO_INITDB_ROOT_USERNAME: ${DB_USER}
       MONGO_INITDB_ROOT_PASSWORD: ${DB_PWD}
     ports:
       - "27017:27017"
     volumes:
       - ${DB_VOLUME}:/data/db
     logging:
       driver: journald
       options:
         tag: "mongo"
   

3. Start the db

   # create directory from .env db volume
   source /home/monad/.env
   sudo mkdir -p $DB_VOLUME
   sudo chown -R monad:monad $DB_VOLUME
   sudo chmod 700 $DB_VOLUME
   
   docker compose up archive-db -d
   

On your Archive Writer host:

4. Add the following section to /home/monad/.env

   ###############################################################
   ################### Vars that do not change ###################
   ###############################################################
   # Replace with your Archive Server credentials
   ARCHIVE_DB_USER="<your db username>"
   ARCHIVE_DB_PWD="<your db pwd>"
   ARCHIVE_DB_HOST="<your db hostname>"
   
   ## Where block data gets written (your Archive Server)
   ARCHIVE_SINK="mongodb mongodb://${ARCHIVE_DB_USER}:${ARCHIVE_DB_PWD}@${ARCHIVE_DB_HOST}:27017 archive-db"
   ## Change this if your ledger folder is in a different location
   BFT_BLOCK_PATH=~/monad-bft/ledger
   OTEL_ENDPOINT=http://0.0.0.0:4317
   ## This can be significantly higher during backfill
   MAX_CONCURRENT_BLOCKS=50
   
   
   ###############################################################
   ####### Backfill from Genesis (initial configuration) #########
   ###############################################################
   BLOCK_DATA_SOURCE="aws testnet-can-004-0-aavn9ll 50" # testnet
   BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet
   ## Alternatively you can use your other ArchiveDB if it has already been backfilled
   ## If using another Archive Server, define variables for it and use them:
   #OTHER_DB_USER="<other db username>"
   #OTHER_DB_PWD="<other db pwd>"
   #OTHER_DB_HOST="<other db hostname>"
   #BLOCK_DATA_SOURCE="mongodb mongodb://${OTHER_DB_USER}:${OTHER_DB_PWD}@${OTHER_DB_HOST}:27017 archive-db"
   FALLBACK_BLOCK_DATA_SOURCE="aws testnet-can-004-0-aavn9ll-0 50" # testnet
   FALLBACK_BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet
   
   ###############################################################
   ###################### Normal Operation #######################
   ###############################################################
   ## Archiver checks triedb first for block data
   BLOCK_DATA_SOURCE="triedb /dev/triedb 5000"
   
   ### FALLBACK_BLOCK_DATA_SOURCE ###
   ## This is used whenever data is missing from BLOCK_DATA_SOURCE
   ## Normally used after state-sync
   ## If you have another Archive Server for redundancy, configure it here:
   #OTHER_DB_USER="<other db username>"
   #OTHER_DB_PWD="<other db pwd>"
   #OTHER_DB_HOST="<other db hostname>"
   #FALLBACK_BLOCK_DATA_SOURCE="mongodb mongodb://${OTHER_DB_USER}:${OTHER_DB_PWD}@${OTHER_DB_HOST}:27017 archive-db"
   
   ## Alternatively you can use category-labs aws bucket
   ## Note: YOU pay for S3 egress costs if pulling from this bucket
   FALLBACK_BLOCK_DATA_SOURCE="aws testnet-can-004-0-aavn9ll 50" # testnet
   FALLBACK_BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet
   

5. Backfilling:

   As of v0.12.3, --start-block is no longer supported as a daemon argument. Instead, use the set-start-block subcommand to set the starting block marker imperatively before starting the daemon. This prevents accidental progress loss from systemd overrides.

   # [Optional] Set a specific start block if needed
   # This is typically only needed for initial setup or recovery scenarios
   # Example: start archiving from block 1000000
   monad-archiver set-start-block --block 1000000 --archive-sink "${ARCHIVE_SINK}"
   
   # For async backfill marker, add the --async-backfill flag:
   # monad-archiver set-start-block --block 1000000 --archive-sink "${ARCHIVE_SINK}" --async-backfill
   
   # Start the archiver daemon
   sudo systemctl start monad-archiver
   # Should show it running
   # Double check the arguments look correct based on the above ^
   systemctl status monad-archiver
   # Expect many:
   # > INFO: Successfully archived block block_num=X
   journalctl -u monad-archiver -o cat
   

6. Once monad-archiver has caught up to the chain tip you will see:

   INFO: Nothing to process

7. This means backfilling is done and you should

   1. Comment out BACKFILL section in your .env and uncomment Normal Operation
   2. Restart: systemctl restart monad-archiver
   3. Verify status and logs as above

On your ArchiveDB host

8. Start monad-indexer

   sudo systemctl start monad-indexer
   systemctl status monad-indexer
   # Expect many:
   # > INFO: Indexing block...
   # > INFO: Index spot-check successful
   journalctl -u monad-indexer -o cat
   

[Optional] Set up monad-archive-checker

Typically only 1 checker is run per organization, but more can be run if desired

1. Add the following systemd override to monad-archive-checker

   sudo systemctl edit monad-archive-checker
   
   [Service]
   ExecStart=
   ExecStart=/usr/local/bin/monad-archive-checker \
       # Example --bucket my-org-mainnet-checker
       # Note: storing checker state in local mongo or filesystem
       #       will be supported in a future release
       --bucket <S3 Bucket for storing checker state>
       checker
       # Example of comparing a local self-hosted mongo against a Category Labs aws bucket
       # --init-replicas "<aws mainnet-deu-009-0 50,mongodb mongodb://<username>:<pwd>@<db hostname>:27017 archive-db>,mongodb mongodb://<username>:<pwd>@<second db hostname>:27017 archive-db>"
       --init-replicas "<ArchiveDB 1>,<ArchiveDB 2>"
   

2. Reload and Restart

   sudo systemctl daemon-reload
   sudo systemctl restart monad-archive-checker
   systemctl status monad-archive-checker
   
   # Check for errors
   journalctl -u monad-archive-checker -o cat -f