[FEATURE] Add lm-eval to our Metrics

[sdiazlor]

‼️ If you want to work on this issue: please comment below and wait until a maintainer assigns this issue to you before opening a PR to avoid several contributions on the same issue. Thanks! 😊

✨ What You’ll Do

Adding more metrics makes Pruna’s evaluations more insightful, so we’re excited to welcome evaluation harness into our metric lineup! 🎉

---

📐 Metric Class

For a detailed guide, refer to the documentation on implementing a metric: https://docs.pruna.ai/en/stable/docs_pruna/user_manual/customize_metric.html.
Also, take a look at the metric code repository here: https://github.com/EleutherAI/lm-evaluation-harness
Quick Overview

1. File & Class
   • Create the new file and define the class.
2. Configuration
   • Choose a sensible call_type 
   • Define the state and update functions.
   • Don't forget to register the metric
3. Testing & Docs
   • Write the unit tests.

And that's all!

✅ Acceptance Criteria

• Correct Output: compute(...) returns a sensible result
• Tests & Docs: All existing and new unit tests pass, and the documentation is updated

And don’t forget to give us a ⭐️!

---

❓ Questions?

Feel free to jump into our Discord channel if you hit any roadblocks. Can’t wait to see your contribution of this metric! 🚀

---

Share on X

[sky-2002]

I will give it a try @sdiazlor , will ping here if I am dropping it in a day.

[sky-2002]

@johannaSommer looking forward for review!