Building a Blockchain Indexer That Survives Reorgs
An indexer looks simple until the chain it indexed a minute ago changes. How I would structure one for an EVM chain, who owns progress, what happens during a reorg, and the protocol facts behind each decision.
An indexer looks simple. Subscribe to new blocks, pull the events you care about, write them somewhere. The hard part is that the chain you indexed a minute ago may not be the chain that exists now. Recent blocks get replaced, WebSocket connections drop, and the process restarts halfway through a batch.
This is how I would design an indexer for an EVM chain such as Ethereum, and the protocol facts that drive each decision.
One process, several loops
I would start with one application, deployed as a single container task, running several asynchronous workers. Splitting each step into its own service adds network hops, deployments and failure modes without buying anything at this scale. The shape is closer to a set of Kubernetes controllers in one binary than to a set of microservices.
- A supervisor owns startup, shutdown, cancellation and worker health.
- A head listener receives new-block notifications over a WebSocket subscription.
- Fetch workers pull block headers and matching logs over RPC.
- A chain controller tracks continuity, detects gaps and reorgs, and decides what gets fetched.
- A publisher sends decoded events to a stream such as Kinesis or Kafka.
- A checkpoint controller records durable progress.
- A recovery worker refetches missed block ranges.
- The head listener receives new-block notices from the node over WebSocket.
- The chain controller checks continuity, detects gaps and reorgs, and schedules blocks.
- Fetch workers pull headers and logs; the publisher writes decoded events to the stream.
- When every record for a block is acknowledged, the checkpoint controller records its height and hash.
- A recovery worker refetches missed ranges; a restart resumes from the checkpoint.
When block 105 arrives, the listener hands the notification to the chain controller, which schedules the block. A fetch worker retrieves its header and logs. The publisher sends the events, and only after the stream confirms the write does the checkpoint controller record block 105 as complete.
Confirming the write means every record. A Kinesis PutRecords call can accept some records and reject others, so the publisher retries only the rejected ones, and the checkpoint moves only when every record for the block is acknowledged. PutRecords also does not preserve record order, so ordering cannot come from the write.
The workers talk through bounded in-memory queues. Work runs concurrently, but one rule holds throughout. Only the checkpoint controller advances progress. Concurrency is for throughput; progress has a single owner.
Progress is a height and a hash
The checkpoint stores the block hash as well as the height. Two different blocks can share a height on competing forks. A checkpoint that only says 105 cannot tell whether the block 105 it processed is the block 105 the chain has now.
This checkpoint records what the indexer has published, nothing more. The consumer's progress and a verified downstream projection are separate checkpoints, and a stream acknowledgement says nothing about whether the business state built from it is correct.
What happens during a reorg
A reorganisation happens when the network replaces recent blocks with a different branch. The chain controller sees it when a new block's parent hash does not match the hash it recorded for the previous height. From there it does four things.
- Walks back until it finds the common ancestor, the last block both branches share.
- Cancels pending work for blocks on the old branch.
- Fetches the replacement blocks, publishes their events, and publishes a correction marking the old branch's events as no longer canonical.
- Holds the checkpoint until the replacement range is complete.
- Blocks 101A and 102A were indexed on top of block 100.
- The chain now has 101B and 102B on the same ancestor; a parent hash no longer matches.
- The indexer walks back to block 100, the common ancestor.
- It publishes the replacement events and a correction marking 101A and 102A as no longer canonical.
- The store keeps every observation and moves the canonical flag.
Events already published cannot be pulled back out of a stream. Consumers have to handle corrections, so the downstream store keeps every observation and tracks which ones are canonical, rather than deleting rows. That also leaves an audit trail of what the indexer saw and when.
A correction and the events it replaces may travel on different shards or partitions, so consumers cannot assume the correction arrives first. They need version guards and completeness checks rather than arrival order.
Reorg depth is a signal. Short reorgs near the head are normal. A reorg that rewrites finalized history on Ethereum mainnet is not routine recovery. Treat it as a provider or consensus integrity problem and verify it before accepting it.
Structured concurrency
In Python I would use asyncio, because the work is almost all waiting on the network. Running every worker inside one asyncio.TaskGroup, available from Python 3.11, gives structured concurrency. If one worker fails with an unhandled exception, the group cancels and awaits the rest, then raises the errors together as an ExceptionGroup. The process stops instead of running half-broken, with a listener receiving blocks that nobody publishes. A clean stop still needs workers whose cleanup is safe to cancel.
Production code also needs graceful shutdown, a timeout on every RPC call, retries with backoff for expected transient failures, and bounded queues, so a slow publisher applies backpressure instead of growing memory.
What in-memory queues can and cannot do
The queues are for coordination, not recovery. If the task crashes, whatever is queued is gone. That is acceptable when four things are true.
- The checkpoint is durable and conservative, so it never claims a block that was not fully published.
- On restart, the indexer refetches everything after the checkpoint from RPC.
- Consumers handle duplicates idempotently, since a block may be published twice across a crash.
- The RPC source can still serve the historical range.
A recovery worker inside the same process is fine for small gaps. Backfilling millions of blocks is a different job. Run extra tasks from the same codebase in a backfill mode over fixed ranges, and keep the live indexer simple.
The protocol facts behind the design
Most of the decisions above follow from a short list of protocol facts. These are the ones I check first when reviewing an indexer.
- Height is not identity. The block hash identifies a block. The height is only its position, and competing blocks can share it.
- A transaction is not an event. A transaction triggers execution, and that execution can emit zero, one or many logs. An indexer watching Transfer logs is not indexing every transaction.
- Canonical is not finalized. Canonical means the current winner of fork choice, which can still be replaced. Ethereum's latest, safe and finalized block tags expose the difference. Finality is reached at epoch checkpoints, with 12-second slots and 32-slot epochs, and typically takes about 15 minutes. Decide which level each consumer needs.
- Subscriptions are not replay. WebSocket notifications are lost on disconnect, so gaps are recovered with historical queries such as eth_getLogs over a block range.
- Historical logs are not historical state. Reading past Transfer logs does not need an archive node, but it does need a node that kept block and receipt history: since July 2025 execution clients can drop pre-Merge history. Asking what a balance was at an old block usually needs an archive node. Providers also differ in how much log history they serve and how wide a range one eth_getLogs call may cover.
- The ABI has to match the block. Logs are raw bytes decoded with the contract's ABI. Upgradeable proxies can change the implementation, so decoding years of history with today's ABI can produce wrong results.
- A log needs a stable identity. Chain ID, block hash and log index identify one log occurrence. The same event on another fork is a different occurrence, with a different block hash. That makes a good deduplication key.
- Transfers do not always add up to balances. ERC-20 recommends a Transfer from the zero address when tokens are minted, and burning to the zero address is a common convention, but rebasing and other non-standard tokens need direct state queries.
- Pending is not done. Mempool transactions can be dropped or replaced, and an included transaction can still revert and emit no logs. Index events from included blocks, not pending ones.
On AWS
The indexer is one part of a data system. On AWS the rest can be managed services, each with one job, so the only long-running process you operate is the indexer itself.
- The indexer runs as an ECS task, reads the node and writes events to Kinesis with PutRecords.
- A Lambda function consumes each shard through an event source mapping and writes idempotently to Aurora PostgreSQL.
- Aurora holds observations, the canonical view and the projections the API reads.
- The stream is archived to S3 so history outlives the stream's retention.
- A reconciler job compares stored state with the chain, replays from the archive and repairs a range.
- The indexer runs as an ECS task and writes with the AWS SDK's PutRecords call, up to 500 records per call. A producer does not need the Kinesis Client Library; that library is for consumers.
- Kinesis Data Streams carries the events. Each shard accepts up to 1,000 records per second, so a partition key with little spread, such as one contract address for every event, creates a hot shard. Key by something with more spread, and size shards from measured peaks.
- A Lambda function consumes the stream through an event source mapping, which avoids running the Kinesis Client Library, a Java library. Lambda processes each shard in order and stops on a failing batch. By default it checkpoints only when a whole batch succeeds, so turn on partial batch responses, and keep writes idempotent because records that succeeded can still be retried.
- Aurora PostgreSQL holds every observation, the current canonical view and the projections the API reads. A unique constraint on the event identity stops duplicates, and a projection advances only when its block is complete. A connection pool such as RDS Proxy keeps Lambda concurrency from exhausting database connections.
- An S3 archive keeps the raw observations, so history can be replayed after records leave the stream. Kinesis can deliver records to S3 directly, but only for on-demand streams.
- A reconciler runs as a separate job when needed. It compares stored state with the chain and repairs or rebuilds a range without pausing live ingestion.
There are three checkpoints here, and they mean different things: the last block the indexer published, the consumer's position in the stream, and the last block whose projection is verified. Monitor them separately. AWS warns that once a consumer's iterator age passes half of the stream's retention period, records are at risk of expiring before they are read, so alarm well before that. Retention is 24 hours by default and can be raised to 365 days.
Leave out what the workload has not asked for: leader election, enhanced fan-out, a separate projection service, cross-region active-active. Add each one when a measurement shows the need.
Beyond one EVM chain
The event envelope can be the same across chains. The semantics underneath cannot. Solana works in slots rather than block numbers, a slot may have no block, and RPC calls take a commitment level of processed, confirmed or finalized. Layer 2 networks add a sequencer, and its acceptance of a transaction is not the same as settlement on the base chain. A token with the same symbol on two chains is two contracts, linked only by whatever bridge or issuer relationship actually exists.
One more boundary matters for anything compliance-related. A wallet address is not a legal identity, and an on-chain record proves that something executed, not that it was allowed. Identity, authorisation and policy come from off-chain sources, and the system should keep them separate from chain evidence.
Questions for a design review
- Does the checkpoint store the block hash, and does only one component advance it?
- How is a reorg detected, and what happens to events already published?
- What is the deduplication key, and does it tell the same event on two forks apart?
- Which finality level does each consumer need?
- After a crash or disconnect, how does the indexer find and fill the gap?
- Which ABI decodes a log from three years ago?
- Can a backfill run without touching the live indexer?
None of this needs a distributed system on day one. One process, a single owner for progress, a durable checkpoint and honest handling of finality are a sound starting point.
Ethereum JSON-RPC API (opens in a new tab)Ethereum proof-of-stake and finality (opens in a new tab)Python asyncio task groups (opens in a new tab)Solana RPC and commitment levels (opens in a new tab)