Prefix protocol metadata to avoid that internal metadata fields are accidentally overwritten

[sebastian-nagel]

HTTP protocol implementations add all HTTP response header fields to the response metadata. The FetcherBolt merges the response metadata into the metadata which is passed forward in the topology as part of the tuples <url, content, metadata>. Existing key-value(s) pairs (persisted in the status index) are overwritten and later eventually stored in the status index (if listed in "metadata.persist") or even transferred to outlinks ("metadata.transfer").

This allows to easily use the response header values for requests, e.g. cookies or "ETag".

However, webadmins are free to send any header back. This may cause unwanted collisions with metadata keys used by crawler-internal classes. E.g. if the server responds with non-standard headers "HostName" or "Depth". Or even standard headers such as "Last-modified" which require to follow a specific format for internal use.

To avoid collisions: Why not prefix protocol metadata: protocol.content-type or http.content-type? This would also make it clear which component sets the metadata - similar to the prefixes fetch. and parse. already used. The draw-back would be that users are required to update the configuration.

[jnioche]

Good summary of the existing behaviour and interesting point, thanks. Not sure how frequent this issue would arise but certainly worth considering, although as you pointed out there would be a cost in updating the configs and code.

[sebastian-nagel]

Not sure how frequent this issue would arise

Not very often for non-standard headers, but it happens. I've counted HTTP header names for a random selection of 250 WARC files and got about 14,000 unique lowercased header names, among them:

• 220 x hostname - luckily without effect because metadata.hostname (old configuration, now field key outside metadata and safe) is always set from the URL (to address StatusUpdaterBolt to use provided domain name for routing #684 by reindexing was the correct decision)
• 5 x ip, e.g. https://dnews.dn.ua/news/735755 :

HTTP/1.1 200 OK
Server: nginx
...
ip: 107.23.218.4

If URLs are partitioned "byIP", the ip header field would overwrite the persisted IP eventually causing duplicates in the status index, same URL in various partitions.

So, it's probably more a matter of time and size of the crawl until a collision happens. It's also a minor security issue. The prefix would then also mark all prefixed metadata names as unsafe, same as for parse.keywords - you cannot trust the value unless you crawl your own content.

a cost in updating the configs and code.

For the code: At a first glance, there aren't so many changes: the protocol implementations and few more places.
For the user config: A configurable prefix if empty would allow to delay any changes. That's important if you want to keep your status index and cannot reindex immediately.

An alternative solution would be to force users to explicitly configure the HTTP header names put into the response metadata:

  # lists the (HTTP) protocol response headers put into
  # response metadata as pairs <key, value(s)>
  # Header names are lowercased. Metadata persisted under the
  # same name is overwritten by the response metadata.
  metadata.protocol.response:
   - etag
   - set-cookie

[jnioche]

hi @sebastian-nagel

An alternative solution would be to force users to explicitly configure the HTTP header names put into the response metadata:

Probably easier to use the prefix instead, there's already enough config everywhere ;-)