Wrong energy reports on a two sockets system

[PierreRust]

Bug description

It seems to be a problem on socket-level energy reporting, potentially when several sockets are available, but that's only an hypothesis.

On my two-sockets workstation ( two Xeon Gold 5118) when i run scaphandre stdout (inside docker) I get the following. Core energy is listed, but not uncore (nor DRAM) but I suspect :

Host:	23.81615 W	Core		Uncore		DRAM
Socket0	15.061086 W	1.214757 W	
Socket1	8.743361 W	0.888699 W

When looking at the json report, core consumption is listed for the first socket, but not uncore, and it's the opposite for the second socket:

    "sockets": [{
            "id": 0,
            "consumption": 17822326.0,
            "domains": [{
                    "name": "core",
                    "consumption": 1738454.0
            }]
        }, {
            "id": 1,
            "consumption": 14311183.0,
            "domains": [{
                    "name": "uncore",
                    "consumption": 1509426.0
                }]
     }]

When I run the same commands on a single-socket laptop system with a i7-10875H I get both core and uncore, and even DRAM, both with sdtout and json

Host:	11.849977 W	Core		Uncore		DRAM
Socket0	11.850316 W	2.098711 W	8.857136 W	0.312643 W

    "sockets": [
        {
            "id": 0,
            "consumption": 12735075.0,
            "domains": [
                {
                    "name": "core",
                    "consumption": 2297552.0
                },
                {
                    "name": "uncore",
                    "consumption": 9526411.0
                },
                {
                    "name": "dram",
                    "consumption": 442801.0
                }
            ]   }   ]

I'd be glad to help on this issue, if you can point me to the right direction ;)

To Reproduce

The exact command I've been using, on both system, is:

docker run --name scaphandre --rm -ti -v /sys/class/powercap:/sys/class/powercap -v /proc:/proc hubblo/scaphandre:latest json

Environment

Both workstation and laptop are running ubuntu 20.04 LTS on kernel 5.8.0-44-generic

[bpetit]

Hi !

Interesting bug, thanks for noticing that.

Looking at it in a rush, I'd say interesting parts of the code are:

• the get_domains_power function in the json exporter which is where the action of filling the data for domains is triggered
• the get_records_diff_power_microwatts function is the sensor module which is where we actually get the consumption data for the domain and return it

To go deeper if the bug is not in the get_records_diff_power_microwatts, I'd say it would be worth looking in the refresh_records function of Domain which is where the actual data is gathered and stored in a vec/buffer

This is really a quick overview, hope it could lead to interesting investigations if you want to.

I'm thinking of organizing peer programming or debugging sessions on discord or some other platform to help in contributions, would you be interested in ?

[bpetit]

Quick update on that, I'm looking at it and it seems that #87 fixes part of what you describe (the wrong allocations of domains to sockets). Thanks @jdrouet !

However, I don't see how we could properly get all the domains on one CPU, and not on another. Could you check what you have in /sys/class/powercap please ? Do you actually have 4 folders for each socket, on the 2-socket system ? For socket 0 it would be intel-rapl:0 intel-rapl:0:0 (core) intel-rapl:0:1 (uncore) and intel-rapl:0:2 (dram).

I'm testing right now on a bi-socket machine and I only have the core domain. I'm not sure if it is a limitation of this implementation of rapl or powercap that could'nt get all the data, but I think we should assume all the domains are not always properly showed by powercap on all machines. Is this your case @PierreRust ? Thanks !

EDIT: browsing a bit about this situation, it seems that not having all domains available for reading could be related to a too old BIOS version, but it is surely not the only possible reason

[PierreRust]

I've checked on two systems (laptop and workstation) and I must say I'm lost here ! I'll keep digging but I don't understand why domains are missing on my workstation.

In powerAPI, rapl reading is done using perf, not by reading the filesystem interface. It requires a bit more rights (but nothing more than scaphandre already requires for other needs I think) but seems to be more reliable, I have no issue running powerAPI on that workstation

On a two sockets workstation: I have both sockets but only one domain for each:

tree -d -L 2 /sys/class/powercap/intel-rapl/   
/sys/class/powercap/intel-rapl/
├── intel-rapl:0
│   ├── device -> ../../intel-rapl
│   ├── intel-rapl:0:0
│   ├── power
│   └── subsystem -> ../../../../../class/powercap
├── intel-rapl:1
│   ├── device -> ../../intel-rapl
│   ├── intel-rapl:1:0
│   ├── power
│   └── subsystem -> ../../../../../class/powercap
├── power
└── subsystem -> ../../../../class/powercap

Using powercap-info, I see both sockets, but only with the dram domain ?!

powerapi@powerapi-Workstation:~$ powercap-info -p intel-rapl  
Zone 0
  name: package-0
  enabled: 1
  Constraint 0
  Constraint 1
  Zone 0:0
    name: dram
    enabled: 0
    Constraint 0
Zone 1
  name: package-1
  enabled: 1
  Constraint 0
  Constraint 1
  Zone 1:0
    name: dram
    enabled: 0
    Constraint 0

When I look at the actual name in /sys/class/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:0/name, it is indeed the dram domain (so one must definitely look at the name of the domain and not simply the index !)

When running the same command on my laptop, things are completely different and are what I would expect. A single socket and 3 domains:

> tree -d -L 2 /sys/class/powercap/intel-rapl/                                                                                       
/sys/class/powercap/intel-rapl/
├── intel-rapl:0
│   ├── device -> ../../intel-rapl
│   ├── intel-rapl:0:0
│   ├── intel-rapl:0:1
│   ├── intel-rapl:0:2
│   ├── power
│   └── subsystem -> ../../../../../class/powercap
├── intel-rapl:1
│   ├── device -> ../../intel-rapl
│   ├── power
│   └── subsystem -> ../../../../../class/powercap
├── power
└── subsystem -> ../../../../class/powercap

powercap-info -p intel-rapl                                                                                                   7s  lun. 19 avril 2021 08:55:10
Zone 0
  name: package-0
  enabled: 1
  Constraint 0
  Constraint 1
  Zone 0:0
    name: core
    enabled: 0
    Constraint 0
  Zone 0:1
    name: uncore
    enabled: 0
    Constraint 0
  Zone 0:2
    name: dram
    enabled: 0
    Constraint 0
Zone 1
  name: psys
  enabled: 0
  Constraint 0
  Constraint 1

[PierreRust]

A quick update on this issue, you're right, #87 fixes the issue with 2 sockets.
About the missing domain, as far as I an tell this is actually "normal" : all other tools (perf list , powercap-info -p intel-rapl , etc. ) also only report package and dram domain and it seems to be the way recent server-class cpu are designed. When looking at intel's documentation for this CPU, only these two domains are mentioned.
So I think you can close this issue, everything is ok now :) thanks !

[bpetit]

Thanks a lot for the update !