Amount of traffic reported is incorrect (much lower than real)

[SnapshotCiTy]

FreeBSD 14.1-RELEASE-p6
Bandwhich 0.23.1

Traffic reported by systat -if 1 is correct and matches what we see on the switch / router
Traffic reported by bandwhich is about 4x lower than what is currently being transferred /second

the traffic is a single zfs send stream through ssh

Screenshot attached.

[cyqsimon]

Maybe related to #360?

On BSD-like OSes we use lsof to read system connections. I suspect this is the source of the problem, or maybe we are just using it wrong. See https://github.com/imsnif/bandwhich/blob/main/src/os/lsof_utils.rs.

Can you please try running bandwhich with --log-to and one level of --verbose? There may be some interesting logs we can take a look at.

[SnapshotCiTy]

here is the log for bandwhich running for a bit while there was some zfs send through ssh transfers under way. Hope it helps.

bandwhich.log.tar.gz

[cyqsimon]

Yeah it seems like some source IP/host of lsof output is in an unhandled format. Although I must admit, the current logging infrastructure lacks sufficient granularity for me to fix it.

I will make some changes to error handling and logging and come back to you. Then you can either self-compile or run the CI build so that we can see what exactly the problem is.

[cyqsimon]

Can you please try out #451? I realised you are on FreeBSD where we unfortunately don't have a CI build available; you would have to self-compile.

Can you please:

1. Check if the problem still persists? I found and fixed a problem during the rewrite which could explain your observations.
2. And take another log for me so I can confirm there are no other issues?

Thanks.

[ulidtko]

Hi @cyqsimon — I'm seeing pretty much the same issue on Linux.

For best reproducibility, measuring over the loopback interface lo.

Here's a test I ran: curl for client, and server is a quick throw-away thing that just streams 128-byte HTTP chunked transfer-encoding frames, every 31.25 ms. I wrote my own, to know for sure.

The server code you don't need, as trivial to (re-)write 📝

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE NumericUnderscores #-}
{-# LANGUAGE RankNTypes #-}

module Stream where

import Foundation

import Conduit
import Control.Concurrent (threadDelay)
import Control.Monad.State.Strict as State
import Data.ByteString.Builder (Builder)
import Data.Fixed (Fixed, HasResolution, Micro)
import Data.Text (Text)
import Data.Time.Clock (NominalDiffTime, nominalDiffTimeToSeconds)
import Data.Time.Clock.POSIX (getPOSIXTime)
import Text.Printf (printf)
import Yesod.Core

getStreamR :: Handler TypedContent
getStreamR = respondSource typeHtml responseBody

responseBody :: ConduitT () (Flush Builder) Handler ()
responseBody = do
  yield (Chunk "hello, stream!\n")
  yield Flush
  threadDelayUs 0.500 -- 500 ms
  now <- liftIO getPOSIXTime
  evalStateC now
     $ yieldMany [1..]
    .| awaitForever (\i -> do
        -- sleep for targetInterval, adjusting for drift
        pre <- State.get
        now <- liftIO getPOSIXTime
        let delay_s = nominalDiffTimeToSeconds (pre + targetInterval - now)
        let delay_us = max 0 (castFixed delay_s) :: Micro
        threadDelayUs delay_us
        State.put (pre + targetInterval)
        -- output 128-byte chunked frame
        let line = printf "%119d\n" (i :: Integer) :: String
        sendChunk line
        sendFlush
    )
    -- 128 bytes every 125 ms = 1.000 kiB per second.
    -- overhead of Transfer-Encoding:chunked -- 8 bytes per frame.
    -- thus, 120-byte chunk every 62.5 ms = 2.000 kiB / s.
    -- and, 120-byte chunk every 31.25 ms = 4.000 kiB / s.

targetInterval :: NominalDiffTime
targetInterval = 0.031_250

-- | Type-inference-driven helper for cross-converting fixed-resolution numbers
-- (a.k.a. fixpoint fractionals) from the 'Data.Fixed' base module.
--
-- In conversions with loss of precision -- rounds down with truncation.
--
-- Converts via 'Integer' without overflow.
castFixed :: forall resA resB. (HasResolution resA, HasResolution resB)
          => Fixed resA -> Fixed resB
castFixed = fromRational . toRational
{-# INLINE castFixed #-}

-- | A variant of 'threadDelay' taking fixed-precision 'Micro'seconds.
threadDelayUs :: MonadIO m => Micro -> m ()
threadDelayUs = liftIO . threadDelay . fromEnum
{-# INLINE threadDelayUs #-}

Thus, I'm expecting download rate of 128 / 0.03125 = 4.000 kiB / s.

What WireShark says

In WireShark, Statistics › TCP Stream Graphs › Throughput — measures ~32700 bits per second:

This is within 1% of the correct value (32768 bps). The graph fluctuates a little, so the precision isn't perfect.

What curl says

Curl itself measures stable 3840 bytes per second — which is bang on the expected value 🎯
(accounting for chunked framing: 32 frames a second with 8 bytes of overhead each).

What bandwhich says

I've cargo installed it, and granted SELinux permissions; --version says 0.23.1.

Then, in the same test, bandwhich -i lo says 2.00 kiB/s 🤷‍♂

— which is wildly off; by exactly a factor of 0.5× (or 2×, if you like).

This is certainly incorrect. For such a tool — super not great, you had one job