[Visualizer/Playbook] Crawl results

[faux-eccles]

Name

Crawl Result

Playbooks

No applicable playbooks exists, however I suggest we create one with the purpose of conducting crawls or fetching live sites of a given URL

Why should we create it

Phishing pages will often have multiple different infrastructures linked to the main page, it'll be useful to directly pivot between related URLs or run other playbooks on related infra

Possible implementation

Render the result of a url scan to include interesting pivotable information from the scan results. including the redirect chain if present, dependent requests, as well as hosting information as detailed by urlscan

[faux-eccles]

I was partially already working on this for personal stuff but figured I should at least have it tracked here so I don't double work if someone else is already working on it. Happy to be assigned to it as well

https://github.com/faux-eccles/IntelOwl/tree/urlscan

Still a long way to go with it

I'd also like to discuss adding a new VisualizableObject for embedding remote images, URLScan offers screen shots of the pages scanned which would be convenient to show in the report, but I think that's a complex enough change to warrant a separate discussion as there are potential security implications with that

[faux-eccles]

Thinking about this I might make the visualization more service agnostic, so we combine the observed infrastructure from multiple crawl sources in the one place and just have a link out to the source of the observations

[drosetti]

Hi @faux-eccles,

• What is your proposal ? If I understood correctly, you want to create a playbook with the analyzers related to the phishing services and create a visualizer that aggregates data from all of them ? Do you want to create a pivot for each analyzer to analyze some info extracted from the analyzers ?
• Talking about the changes to the visualizer framework: I think we could create another component (both frontend and backend) for the images, you can open another issue

[faux-eccles]

Hi @drosetti,

That's about right, more specifically it's the playbook for any service that renders a webpage and provides details about how the page is loaded and the requests involved. The details should be in the result from the existing analyzer so shouldn't require a new connector or anything just the analyzer to display the results.

The details I'm looking to display using a urlscans result as an example,:

• url observables and ip observables involved in the redirect chain when loading the page https://urlscan.io/result/01999ef4-8e55-74f1-bb0d-fe89de2f708c/#redirects:

• The list of XHR requests, websocket connections, and JS files that the loaded page connects to: https://urlscan.io/result/01999ef4-8e55-74f1-bb0d-fe89de2f708c/#transactions

• Optionally a screen shot of the rendered page

• Optionally a list urls observables that link to other parts of the rendered web page: https://urlscan.io/result/01999ef4-8e55-74f1-bb0d-fe89de2f708c/#links

All this information is already available via the API object when you perform a scan with the existing urlscans.io analyzer, so doesn't need additional processing to display the results.

If it's service agnostic then we can submit a crawl to multiple services (or query existing results via the urlscans search analyzer) and render all the results side by side to drive investigations

[drosetti]

Ok, I think the best think way is: create a playbook with N (number of services) analyzers and the visualizer. In the visualizer we could use these components:

• VList for redirect chain elements
• Vlist for the connections. we could think about multiple elements based on type of connectios: One for js, one for images, one for ws, and so on
• VList for Optionally a list urls observables that link to other parts of the rendered web page
• The new component with the screenshot of the page

Maybe we could have multiple pages, each one for each service.
How does it sounds ? Do you seen any problem ?

[faux-eccles]

Yeah I like the sounds of that, and happy to pick it up and continue working on it

[github-actions]

This issue has been marked as stale because it has had no activity for 30 days. If you are still working on this, please provide some updates.