Running the stack, conceptually

This lesson does not tell you how to install anything. Installation instructions belong in the official documentation, where they are maintained by people who change them when the software changes.

What this covers is the shape of the system, so that when something breaks you have a model of what could be broken.

The components and what they need.

graph-node needs chain access, Postgres, and more disk and memory than you first budgeted. It is the component that does the actual indexing, and it is where most resource problems originate. Every subgraph it indexes is a schema in Postgres that grows without bound.

Postgres is not an afterthought here. Query latency, indexing throughput and your backup story all run through it. Treat it as a first-class database with a real operator, because that is what it is.

Chain access is the cost that surprises people. Indexing history means reading history, which means archive access. Run your own and you are operating another substantial piece of infrastructure. Buy it and it is a recurring bill that scales with what you index.

indexer-agent needs an operator key, network access, and your indexing rules. It takes the decisions.

indexer-service-rs is the public face. It is stateless, so it scales horizontally, and it needs to be reachable and fast.

indexer-tap-agent runs as a single instance and turns receipts into redeemable vouchers. If it is down, you serve queries for free.

Operator keys and stake.

Your stake sits in one wallet. The day-to-day actions, opening allocations, submitting POIs, redeeming vouchers, are performed by an operator key that is authorised to act for you without being able to move the stake.

Use it. Separating the key that runs the machine from the key that holds the capital is the single highest-value operational decision available to you, and it costs one transaction to set up.

Which failures are loud, and which are silent.

The loud ones look after themselves. A process that dies gets restarted, and disk filling triggers whatever you already have.

The silent ones are the expensive category, because in every case the system appears healthy and the money stops:

Note what they have in common. Not one is a systems failure. Every one is an economic failure that a systems dashboard reports as green.

What to build before you scale.

In order, and the order matters:

1. POI age alerting per allocation, warning well before 28 days.
2. Deployment health, distinguishing syncing, synced and failed. A failed subgraph is invisible in aggregate metrics.
3. Payment pipeline monitoring. Unaggregated value and unredeemed RAV age.
4. POI divergence checking against other indexers, before a dispute rather than after.
5. Postgres backups you have actually restored from. An untested backup is a belief, not a backup.

Only then add another chain.

On running it alone.

Worth saying plainly, since most material skips it. Chains do not pause for holidays, illness or a bad week. An indexer with one operator has a single point of failure who needs to sleep.

Options exist: an arrangement with another operator to watch each other’s alerts, automation that closes or reallocates on failure, running a smaller allocation set that degrades gracefully. Any of these is better than intending to be vigilant.