Streaming aggregated TimeMap

[ibnesayeed]

It may be useful to have the aggregator stream TimeMaps as they accumulate from various upstream archives. Sorting in this case will not be needed (or even practical). This can be utilized in some visualizations or tools like Mink where the number may dynamically change as more and more responses arrive from upstream archives. From the implementation perspective, chunked encoding could be one of many other ways to transmit the data to the client.

[machawk1]

@ibnesayeed This would be very useful for tools like Mink where a partial result is often better than waiting for a comprehensive result if the latter is not needed by the user.

[ibnesayeed]

The ORS/CDXJ format will be a good candidate for the stream serialization. Perhaps we should only support that format for streaming.

[machawk1]

@ibnesayeed Agreed, and this highlights one of the benefits of using CDXJ over Link (or JSON) since a partial TimeMap is still valid and useful.

[ibnesayeed]

There is a potential of utilizing the chunk extension to provide metadata about each batch of response that might contain the name of the archive and number of mementos after the batch size.

2a7; provider=IA; memento-count=15;
... ORS/CDXJ lines...