Story Form

As a Block Node Operator
I want to ensure that my archive plugin continues to function if it fails to archive a batch and notifies me for the failure
So that I will not lose the next batches and be able to take an action for the failed one

Technical Notes

• currently the files.historic plugin does not allow gaps in the zips and also does not proceed with the next batch until it finishes this one.
• effectively this means that if we cannot supply a block for the current batch, the archive will "halt".
• a gap is not the only condition for a failed or incomplete batch.
• no matter why a batch may have failed or is incomplete, that should not halt the processing of the next batch.
• an alert has to be sent (i.e. log error) when a batch has failed to archive or is closed incomplete.
• we also need a way to ensure that we do not lose a batch that fails to archive or is incomplete.

We need to allow the files.historic plugin to continue with the next batch if the current batch fails or is incomplete. We also need to alert operators for the situation. Also, we need to ensure that we do not lose batches that we have failed to archive or that are incomplete. Furthermore, the failed or incomplete batch needs to be "set aside" to try again later (and in future updates we might send a notification to request high priority backfill of the missing blocks).