[ChainOrchestrator] Monitor performance of recursive syncing of blocks

[frisitano]

Overview

In #185 we recursively request blocks from the network using the header.parent_hash field to ensure we fetch the correct chain. This could be a bottleneck, and we need to validate that the latency is sufficiently low to ensure we do not enter an endless cycle due to high latency.

TODO: Add code reference @frisitano

[Thegaram]

My concern is that while we're fetching the missing ancestors we stop processing newly gossiped blocks, resulting in another gap, and so on. This is exacerbated by sub-second block times, where I believe longer gaps will be more common.

If this proves to be an issue, there are a few techniques we can try:

1. Fetch the full block range in one query, instead of querying ancestors recursively.
2. While fetching the block gap, still accept gossiped blocks and cache them for future processing.

[frisitano]

We no longer recursively sync blocks, we use a single query to fetch blocks over the network. Current performance appeas to be stable.