Add Rapid7 Velociraptor artifacts plugin

[Zawadidone]

Adds support for Velociraptor artifacts: https://docs.velociraptor.app/docs/vql/artifacts/.

[Zawadidone]

I have a few questions:

• Why doesn't this work (https://github.com/fox-it/dissect.target/pull/698/files#diff-d21a3f7b5a8a12a1531cd0193f1d5d28eed550ad78782baaf78ce79f25b66631R135-R139)? The file uploads.json is mapped correctly and exists in the root filesystem of the target but the other files in the results directory do not.
• Why is there no record type dict, otherwise I will just use str? https://github.com/fox-it/dissect.target/pull/698/files#diff-b1b22daa6721c32ea013bfb6fcaabe0e1fc80d9d38988451a01e44503c7ac220R48

[Zawadidone]

@Schamper, I am trying to map files into a different location so that these files can be used by plugins.

• https://github.com/fox-it/dissect.target/pull/698/files#diff-d21a3f7b5a8a12a1531cd0193f1d5d28eed550ad78782baaf78ce79f25b66631R138
• https://github.com/fox-it/dissect.target/pull/698/files#diff-d21a3f7b5a8a12a1531cd0193f1d5d28eed550ad78782baaf78ce79f25b66631R142

The function target.fs.map_file triggers the exception FileNotFoundError when the mapped files are accessed, is this a known issue?

[Zawadidone]

@EinatFox can somebody take a look at this PR? I don't know if the code that I want to implement will even work.

[Zawadidone]

@Schamper any response?

[Schamper]

@Schamper any response?

I've brought this PR up internally with the team several weeks ago as I don't have much time to look into this until the new year, but doesn't look like anyone responded yet.

@EinatFox pinging again.

[EinatFox]

Hi @Zawadidone and sorry for the late response. As we are very low on resources and this has low priority for us internally, I cannot allocated resources for this soon. However, we understand the value for the community. Let's connect in January to see how we can help you finalize this PR submission.

[Schamper]

Can you also move acquire_hash.py and acquire_handles.py into a single plugins/apps/edr/acquire.py and make it a namespace? acquire.hashes and acquire.handles sounds nice.

[Zawadidone]

This is not blocking but the function iterdir works fine but when using glob the FS entries do not exist, which results in the exception FileNotFoundError: https://github.com/fox-it/dissect.target/pull/698/files#diff-b1b22daa6721c32ea013bfb6fcaabe0e1fc80d9d38988451a01e44503c7ac220R84-R86.

Is this normal behaviour of the filesystem?

[codecov]

Codecov Report

Attention: Patch coverage is 88.03419% with 14 lines in your changes missing coverage. Please review.

Project coverage is 77.86%. Comparing base (bf3860a) to head (c74c753).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dissect/target/plugins/apps/edr/velociraptor.py	84.28%	11 Missing ⚠️
dissect/target/plugins/apps/edr/acquire.py	94.28%	2 Missing ⚠️
dissect/target/loaders/velociraptor.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #698      +/-   ##
==========================================
+ Coverage   77.84%   77.86%   +0.01%     
==========================================
  Files         357      357              
  Lines       32367    32435      +68     
==========================================
+ Hits        25196    25255      +59     
- Misses       7171     7180       +9     

Flag	Coverage Δ
unittests	77.86% <88.03%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Schamper]

This is not blocking but the function iterdir works fine but when using glob the FS entries do not exist, which results in the exception FileNotFoundError: https://github.com/fox-it/dissect.target/pull/698/files#diff-b1b22daa6721c32ea013bfb6fcaabe0e1fc80d9d38988451a01e44503c7ac220R84-R86.

Is this normal behaviour of the filesystem?

Do you have a full stacktrace or otherwise a small reproducible snippet? I'm not really sure what you mean from just this description.

[Zawadidone]

Listing the files with the function iterdir works correctly, but then an extra line of code is needed to skip non JSON files. The function glob could resolve this but it does not work correctly as shown in the error below.

I am curious whether this a bug in the filesystem implementation of Dissect or related to the implementation of the VelociraptorLoader

diff --git i/dissect/target/plugins/apps/edr/velociraptor.py w/dissect/target/plugins/apps/edr/velociraptor.py
index 0a6250f9..08b96445 100644
--- i/dissect/target/plugins/apps/edr/velociraptor.py
+++ w/dissect/target/plugins/apps/edr/velociraptor.py
@@ -81,10 +81,7 @@ class VelociraptorPlugin(Plugin):
         References:
             - https://docs.velociraptor.app/docs/vql/artifacts/
         """
-        for artifact in self.results.iterdir():
-            if not artifact.name.endswith(".json"):
-                continue
-
+        for artifact in self.results.glob("*.json"):
             # "Windows.KapeFiles.Targets%2FAll\ File\ Metadata.json" becomes "windows_kapefiles_targets"
             artifact_name = (
                 urllib.parse.unquote(artifact.name.removesuffix(".json")).split("/")[0].lower().replace(".", "_")

Traceback (most recent call last):
  File /dissect.target/dissect/target/helpers/compat/path_310.py", line 261, in get
    return self._entry
AttributeError: 'TargetPath' object has no attribute '_entry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dissect.target/.tox/py310/bin/target-query", line 8, in <module>
    sys.exit(main())
  File "/dissect.target/dissect/target/tools/utils.py", line 249, in wrapper
    return func(*args, **kwargs)
  File "/dissect.target/dissect/target/tools/query.py", line 386, in main
    for record in record_generator:
  File "/dissect.target/dissect/target/plugins/apps/edr/velociraptor.py", line 97, in velociraptor
    for line in artifact.open("rt"):
  File "/dissect.target/dissect/target/helpers/compat/path_310.py", line 391, in open
    return self._accessor.open(self, mode, buffering, encoding, errors, newline)
  File "/dissect.target/dissect/target/helpers/compat/path_310.py", line 91, in open
    return path_common.io_open(path, mode, buffering, encoding, errors, newline)
  File "/dissect.target/dissect/target/helpers/compat/path_common.py", line 216, in io_open
    raw = path.get().open()
  File "/dissect.target/dissect/target/helpers/compat/path_310.py", line 263, in get
    self._entry = self._fs.get(str(self))
  File "/dissect.target/dissect/target/filesystem.py", line 1628, in get
    entry = super().get(path, relentry)
  File "dissect.target/dissect/target/filesystem.py", line 1479, in get
    raise FileNotFoundError(full_path)
dissect.target.exceptions.FileNotFoundError: /$velociraptor_results$/Windows.Network.NetstatEnriched%2FNetstat.json

[Schamper]

I can't reproduce, should the unit test trigger this already? I made some small other changes but I don't think those should affect this.

[Zawadidone]

The unit tests are not affected and the error occurred when using target-query on a Velociraptor collection.

When using a virtual file system the issue does not occur, so the commit 9ec0fd1 (#698) fixed the issue.

[Schamper]

Weird, but okay. If you address that last comment then this is good to go.