Firehose is the layer that gets data out of a chain node efficiently. Most developers never interact with it directly, and understanding it explains a great deal about everything above.
The problem it solves.
A chain node’s RPC interface was designed for asking specific questions: give me this block, this receipt, these logs. It is request-response, and every answer is a round trip.
Reading an entire chain history through that interface means an enormous number of round trips. The bottleneck is not the data volume, it is the protocol. You are asking a question and waiting, several million times.
Firehose instruments the node to emit everything as it processes, writing an ordered stream of flat files. Reading history then becomes reading files sequentially, which is what disks and networks are good at.
What this makes possible upstream.
Three things, and each one is a property of a layer above:
Substreams parallelism. Because the input is a file stream rather than a stateful cursor into a node, different block ranges can be read by different machines at the same time.
Deterministic replay. The stream is a fixed artefact. Running the same transformation over it twice gives the same result, which is the property the whole protocol’s verification depends on.
Cheaper backfill. Reading history stops being a per-block negotiation with a node and becomes a sequential read.
Who actually runs it.
Infrastructure operators, mostly. If you are running indexing infrastructure at scale, or integrating a chain that does not have support yet, you are in Firehose territory. If you are building an application, you are not.
It is Apache-2.0 licensed and developed in the open, which matters for the same reason it matters that subgraph mappings are deterministic: an extraction layer nobody can inspect would be a trust assumption underneath everything.
What to take from this.
If somebody says a pipeline is Firehose-based, they mean the slow part was replaced with a sequential read of a fixed artefact. That is why Substreams can parallelise, why backfills are faster, and why the results are reproducible.
That is genuinely all you need unless you are operating the infrastructure, in which case the specification and the chain integration process are the places to go.
What is the bottleneck Firehose removes?
The problem is the protocol shape, not the data volume. Millions of round trips become a sequential read of flat files, which is what disks and networks are good at.
Which property above Firehose depends on it emitting a fixed, ordered artefact?
A fixed input stream is what makes transformations reproducible, and reproducibility is what the protocol's verification rests on. It also enables the range-parallelism Substreams uses.
0 of 2 answered