New URL filter implementation based on JSON file and organised per hostname or domain

[jnioche]

One benefit is that we won't need to go through all the patterns and can just check the ones for a particular domain which could be quicker. Being based on a JSON file, means that thanks to #569 and we'll even be able to refresh the resources from ES without restarting the topology.

[noerw]

Even more flexibility would be given, if the filters were organized by arbitrary keys, that would be matched to metadata key/values.
This would for example allow the use case where sets of (seed) URLs are grouped by an external ID (eg metadata.crawlId), which maps to specific crawl settings / filters for that group.

[jnioche]

That's a good suggestion. I'll think about a nice way to configure per host/domain/arbitrary metadata in the same config file but if you have any suggestion, that would be welcome.

[sebastian-nagel]

Maybe the urlfilter-fast plugin in CC's fork of Nutch could be a source of inspiration. Matching regular expressions only on path (and query) is also faster than matching on the longer URL string. Currently, urlfilter-fast uses a HashMap to hold rules per host/domain. That scales well up to 100,000s of hosts/domains given that most of them share the simple {{DenyPath .*}} - I'm trying to replace the hash by a trie and hope to be able to blacklist few millions of domains using only about 100-200 MB of Java heap memory.

[jnioche]

@noerw I've put an initial version in a separate branch, please have a look and let me know what you think of it

[noerw]

Amazing, thank you! This looks great, two remarks though:

• When generating the required JSON, it would be easier to have the scope type, key and value in separate JSON fields instead of having to build strings. But thats just preference I guess.

  [
         {
  		"scope": "GLOBAL",
  		"patterns": ["DenyPathQuery \\.jpg"]
  	},
  	{
  		"scope": "DOMAIN",
  		"value": "stormcrawler.net",
  		"patterns": ["DenyPath .+"]
  	},
  	{
  		"scope": "METADATA",
  		"value": "bar",
  		"key": "foo",
  		"patterns": ["DenyPath .+"]
         }
  ]

• To implement a whitelist, a directive AllowPath could easily be added to checkScope(), right?

[jnioche]

it would be easier to have the scope type, key and value in separate JSON fields instead of having to build strings

I considered it but thought it was simpler like that. The value (whatever it is) is part of the scope and easier to build and understand.

To implement a whitelist, a directive AllowPath could easily be added to checkScope(), right?

Could do. I'll have a look at that.

Thanks for your feedback

[jnioche]

@noerw see #581

[jnioche]

Merged