[benchmark] fix issues with benchmark running locally

[jennyf19]

[VmPeter]

Hi @jennyf19 , you created this but then closed it again. I see you are working on the 3.7.0 release. Did you see the other issue here: #3245

What issues did you see and did you fix them?