jitdump output not accepted by ARM streamline

[RobertHenry6bev]

Description

dotnet 6.0 running on ARM64 ubuntu 21.10 linux, with the environment variable DOTNET_PerfMapEnabled=1, produces a jit dump file in /tmp/jit-*.dump as expected.

However, when the ARM Streamline Performance Analyzer v7.7.0 from https://developer.arm.com/tools-and-software/embedded/arm-development-studio/components/streamline-performance-analyzer is given this jitdump file, then streamline complains:

com.arm.streamline.analysis.processor.profile.JITDumpParserException: JIT dump file contains out of order records
at com.arm.streamline.analysis.processor.profile.JITDumpStateBuilder.validateTimestamp(JITDumpStateBuilder.java:614)
at com.arm.streamline.analysis.processor.profile.JITDumpStateBuilder.processRecord(JITDumpStateBuilder.java:598)
at com.arm.streamline.analysis.processor.profile.Profile.lambda$1(Profile.java:70)
at com.arm.streamline.analysis.processor.profile.JITDumpParser.parse(JITDumpParser.java:714)
at com.arm.streamline.analysis.processor.profile.Profile.loadJITDumps(Profile.java:70)
at com.arm.streamline.analysis.processor.profile.Profile.process(Profile.java:187)
at com.arm.streamline.analysis.AnalysisBase.run(AnalysisBase.java:92)
at java.base/java.lang.Thread.run(Thread.java:834)

Reproduction Steps

export DOTNET_PerfMapEnabled=1
perf record --call-graph=fp -k CLOCK_MONOTONIC --count 1 dotnet_aot_binary.x
scp /tmp/jit*.dump perf.data host_with_streamline:.

On the host_with_streamline, run streamline with input from perf.data and analyze that pointing the analyzer to the jitdump file. Observe the thrown exception in streamline's stderr.

Expected behavior

Streamline reads the jitdump file, anid is able to map pc's in dotnet JITted code to meaningful source lines.

Actual behavior

An exception is thrown; streamline continues on in degraded mode, as there are no pc maps.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

dotnet 6.0 running on ARM64 ubuntu 21.10 linux, with the environment variable DOTNET_PerfMapEnabled=1, produces a jit dump file in /tmp/jit-*.dump as expected.

However, when the ARM Streamline Performance Analyzer v7.7.0 from https://developer.arm.com/tools-and-software/embedded/arm-development-studio/components/streamline-performance-analyzer is given this jitdump file, then streamline complains:

com.arm.streamline.analysis.processor.profile.JITDumpParserException: JIT dump file contains out of order records
at com.arm.streamline.analysis.processor.profile.JITDumpStateBuilder.validateTimestamp(JITDumpStateBuilder.java:614)
at com.arm.streamline.analysis.processor.profile.JITDumpStateBuilder.processRecord(JITDumpStateBuilder.java:598)
at com.arm.streamline.analysis.processor.profile.Profile.lambda$1(Profile.java:70)
at com.arm.streamline.analysis.processor.profile.JITDumpParser.parse(JITDumpParser.java:714)
at com.arm.streamline.analysis.processor.profile.Profile.loadJITDumps(Profile.java:70)
at com.arm.streamline.analysis.processor.profile.Profile.process(Profile.java:187)
at com.arm.streamline.analysis.AnalysisBase.run(AnalysisBase.java:92)
at java.base/java.lang.Thread.run(Thread.java:834)

Reproduction Steps

export DOTNET_PerfMapEnabled=1
perf record --call-graph=fp -k CLOCK_MONOTONIC --count 1 dotnet_aot_binary.x
scp /tmp/jit*.dump perf.data host_with_streamline:.

On the host_with_streamline, run streamline with input from perf.data and analyze that pointing the analyzer to the jitdump file. Observe the thrown exception in streamline's stderr.

Expected behavior

Streamline reads the jitdump file, anid is able to map pc's in dotnet JITted code to meaningful source lines.

Actual behavior

An exception is thrown; streamline continues on in degraded mode, as there are no pc maps.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	RobertHenry6bev
Assignees:	-
Labels:	area-Diagnostics-coreclr, untriaged
Milestone:	-

[RobertHenry6bev]

I wrote a C program to scan the jit dump files produced by dotnet looking for issues. The file consists of a sequence of records. Each record has a timestamp. streamtrace fails when the timestamp of record N+1 is < the timestamp on record N, eg, the next record is marked as earlier than the previous record. My C program now overwrites the tiny fraction of imestamps that fail the test, so that timestamp(N+1) = timestamp(N)+1

This seems to make streamline happy.

AFAICT, thie jit dump specificationi https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jitdump-specification.txt is silent on this matter.

[RobertHenry6bev]

If I pin the dotnet program being profiled and jitdumped to a single core, then the timestamps are indeed monotonic. If I put the core number in the upper 12 bits of the timestamp (which is otherwise 0), I can see the core assignment bounce around.

I run the dotnet app under perf record -k CLOCK_MONOTONIC, following what ARM suggests one does for java programs https://developer.arm.com/documentation/101816/0707/JIT-profiling-support/JIT-profiling-support I don't see a noted change in behavior if I use CLOCK_MONOTONIC_RAW.

My suspicion is that the arm streamline tool doesn't need to have the records sorted strictly by timestamp. It is conceivable that i the clocks are skewed, then it may appear to streamline that code is executing (eg, a stack snapshot from perf record) before the timestamp of the code itself, as recorded in the JIT dump file.

[bengaineyarm]

Hi @RobertHenry6bev, I'm the lead engineer for Streamline here at arm.

We've tracked this internally as a bug, and the fix will appear in the next release (which due to different release cycles for Arm's various tool suites will be Streamline 8.0).

[JulieLeeMSFT]

We've tracked this internally as a bug, and the fix will appear in the next release (which due to different release cycles for Arm's various tool suites will be Streamline 8.0).

Hi @bengaineyarm, thanks for the fix. What is the approximate release date for Streamline 8.0?

[bengaineyarm]

Hi @JulieLeeMSFT ; the internal release process for Streamline is being done now, but it will be distributed within Arm Development Studio, which typically releases about a month after us, so I'd imagine towards the end of March / start April.

[JulieLeeMSFT]

Hi @JulieLeeMSFT ; the internal release process for Streamline is being done now, but it will be distributed within Arm Development Studio, which typically releases about a month after us, so I'd imagine towards the end of March / start April.

That sounds great. Much earlier than I thought.