Pipeline monitoring

This script can be used to monitor the capture task while it's running (you can start this script first, then the pipeline):

  adp@adp:~/bifrost/scripts$ ./print_Depacketize_stats.py adp:7777

where the specified address is the one on which the capture process is running.

Output of script

   RECV    GOOD MISSING   NRECV NIGNORE   NLATE  OVRWRTN  PENDING  MISSING    RATIO
   Gb/s    Gb/s Bytes/s   pkt/s   pkt/s   pkt/s    Bytes    Bytes    Bytes         
  0.000   0.000 0.00e+00       0       0       0 0.00e+00 1.23e+08 0.00e+00 0
  0.401   0.000 0.00e+00   32322       0       0 0.00e+00 1.84e+08 0.00e+00 0
  0.717   0.000 0.00e+00   57785       0       0 0.00e+00 2.46e+08 0.00e+00 0
  0.451   0.958 2.15e+08   36323       0       0 0.00e+00 1.75e+08 2.37e+08 1.03e+01

If you're capturing from all roaches, the RECV Gb/s should be around 14.746. There is another script in ~/monitor_p5p1.py which will print out the rate of packets being received at the system level. If this is also too low then there may be something wrong with the roach config (e.g., I've had roaches get into bad states where they needed to be rebooted).

Ideally the GOOD column matches the RECV column, and MISSING is zero. The RATIO column is a hack that gives the fraction of missing data out of 16 (so an integer would indicate whole roaches). IGNORE means bad packets, LATE means packets with sequence numbers behind where it's currently supposed to be up to, and OVRWRTN means the pipeline was running too slowly and the buffer filled up. PENDING is just how much data is sitting in internal buffers waiting to be completed.

The stalling after a pause will be proportional to the time since the sequence numbers were at 0. Generally it will break things and shouldn't be expected to work unless the pause is say <= O(1 sec) while the pipeline is left running.

The pipeline should roughly max out 3 threads and use a few more as well. I also found that the disk IO could continue long after things appeared to have finished, visible via iotop.