Skip to content

feat: 271 feature implement hps hpsv2 VQA #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat: 271 feature implement hps hpsv2 VQA #272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a6d6cb8
feat: add ImageReward metric for text-to-image evaluation
davidberenstein1957 Jul 24, 2025
1b98ffb
feat: add ImageReward metric for text-to-image generation evaluation
davidberenstein1957 Aug 30, 2025
0b07425
chore: remove `timm` dependency from `pyproject.toml`
davidberenstein1957 Jul 24, 2025
e0bd484
chore: update image-reward dependency source in pyproject.toml and uv…
davidberenstein1957 Jul 22, 2025
5a40613
feat: add HPS and HPSv2 metrics for text-to-image evaluation
davidberenstein1957 Jul 22, 2025
806d2ed
refactor: streamline reward metrics imports and update test structure
davidberenstein1957 Jul 22, 2025
c18e279
feat: add VQA metric for text-to-image generation evaluation
davidberenstein1957 Aug 30, 2025
6ac22c6
refactor: enhance reward metrics initialization with model loading ar…
davidberenstein1957 Jul 22, 2025
36b4e35
refactor: update test_reward_metric to include model loading arguments
davidberenstein1957 Jul 22, 2025
81e2120
refactor: clean up imports in task.py by removing redundant ImageRewa…
davidberenstein1957 Jul 24, 2025
b587517
refactor: update HPSv2Metric to use partial function for model scoring
davidberenstein1957 Jul 24, 2025
f3fbce7
feat: add SharpnessMetric to evaluation metrics
davidberenstein1957 Jul 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ gptqmodel = [
{ index = "pruna_internal", marker = "sys_platform != 'darwin' or platform_machine != 'arm64'"},
{ index = "pypi", marker = "sys_platform == 'darwin' and platform_machine == 'arm64'"},
]
image-reward = { git = "https://github.com/PrunaAI/ImageReward" }
hpsv2 = { git = "https://github.com/PrunaAI/HPSv2" }
Comment on lines +63 to +64
Copy link
Member Author

@davidberenstein1957 davidberenstein1957 Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were some issues with both repositories.
ImageReward, had pushed their fixes to main
HPSv2 had very strict constraints for protobuf where they set it to lower than 6

Copy link
Member Author

@davidberenstein1957 davidberenstein1957 Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@johnrachwan123 perhaps we should publish these on our own index?

Copy link
Member Author

@davidberenstein1957 davidberenstein1957 Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also we could add t2v-metrics with a more relaxed Python constraint


[project]
name = "pruna"
Expand All @@ -70,7 +72,7 @@ authors = [
]
license = {file = "LICENSE"}
readme = "README.md"
requires-python = ">=3.9,<3.13"
requires-python = ">=3.10,<3.13"
Copy link

@cursor cursor bot Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Python Version Mismatch in Package Metadata

The pyproject.toml declares Python 3.9 support in its classifiers, which conflicts with the requires-python field now set to >=3.10,<3.13. This creates a mismatch in the package metadata.

Fix in Cursor Fix in Web

keywords = ["AI", "machine learning", "model optimization", "pruning"]
classifiers = [
"Development Status :: 4 - Beta",
Expand Down Expand Up @@ -106,7 +108,6 @@ dependencies = [
"codecarbon",
"pynvml",
"thop",
"timm",
"bitsandbytes; sys_platform != 'darwin' or platform_machine != 'arm64'",
Copy link
Member

@begumcig begumcig Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we remove timm from the dependencies?

"optimum-quanto>=0.2.5",
"ctranslate2==4.6.0",
Expand All @@ -115,11 +116,14 @@ dependencies = [
"torchao",
"llmcompressor",
"gliner; python_version >= '3.10'",
"piq",
"opencv-python",
"kernels",
"aenum"

"aenum",
"image-reward",
"piq",
"clip",
"hpsv2",
"t2v-metrics",
]

[project.optional-dependencies]
Expand Down
6 changes: 6 additions & 0 deletions src/pruna/evaluation/metrics/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
from pruna.evaluation.metrics.metric_memory import DiskMemoryMetric, InferenceMemoryMetric, TrainingMemoryMetric
from pruna.evaluation.metrics.metric_model_architecture import TotalMACsMetric, TotalParamsMetric
from pruna.evaluation.metrics.metric_pairwise_clip import PairwiseClipScore
from pruna.evaluation.metrics.metric_reward import HPSMetric, HPSv2Metric, ImageRewardMetric
from pruna.evaluation.metrics.metric_sharpness import SharpnessMetric
from pruna.evaluation.metrics.metric_torch import TorchMetricWrapper

__all__ = [
Expand All @@ -37,4 +39,8 @@
"TotalMACsMetric",
"PairwiseClipScore",
"CMMD",
"ImageRewardMetric",
"HPSMetric",
Copy link
Member

@sharpenb sharpenb Aug 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If HPS does not work, I would prefer to leave it out for now, or fix it.

"HPSv2Metric",
"SharpnessMetric",
]
Loading
Loading