Batch evaluations

[RobotSail]

Currently it seems like the evaluations are processed sequentially and could be made more efficient. We should consider running the evaluations with some kind of batching or parallelism to speed up how quickly we can evaluate models.

[yurakuratov]

Hi! Thank you for your suggestion and for taking the time to share your insights.

You're absolutely right that implementing batching or parallelism could speed up evaluations, especially for smaller models and shorter sequences. However, we've chosen to use a simple evaluation process with batch_size=1 to maintain compatibility across a wide range of use cases (from relatively small 1B models to large 70B+ models) and hardware configurations. For example, our setup supports models running locally via HuggingFace, or via API with frameworks like llama.cpp or vllm, as well as remote model requests.

Possibly, one way could be to integrate our bench into lm-evaluation-harness to support more features.

If you have a specific model in mind that you want to evaluate regularly with batching, I would suggest implementing a custom evaluation loop based on our evaluation code to save the model's predictions.