TRACE Model Fine-Tuning Evaluation Issue: Reported vs. Observed Metrics Mismatch

[Rorwey]

Hello, I encountered an issue where the results I obtained using the fine-tuned model provided by the authors (MODEL_DIR="model/trace-ft-youcook2") are much closer to the TRACE-UNI baseline and significantly different from the values reported in the paper. Specifically, my metrics are SODA_c_2: 2.3, F1_Score: 18.5, and CIDER: 7.5, whereas the paper reports SODA_c_2: 6.7, F1_Score: 31.8, and CIDER: 35.5.
I followed the evaluation script (trace/eval/eval.sh) as instructed. Could there be any specific parameters or settings required for evaluating the fine-tuned model that I might have overlooked?
Any guidance would be greatly appreciated.

[gyxxyg]

Hi, we did not set specific parameters or configurations for the fine-tuned models, and I believe you have used the correct evaluation script. Could you please double-check if the correct checkpoint was used for evaluation? It seems odd that the fine-tuned models produce the same results as the non-fine-tuned models.

[Rorwey]

Thank you for your reply. I simply changed MODEL_DIR='model/trace-uni' to MODEL_DIR="model/trace-ft-youcook2". The relevant checkpoints mentioned by the author were all downloaded from https://huggingface.co/. The evaluation results of trace-uni are close to the zero-shot results reported in the paper. Do I need to adjust any other parameters?

I noticed that when the checkpoint is trace-uni, load_pretrained_model does not print any additional information. However, when loading model/trace-ft-youcook2, it warns: You are using a model of type videollama2_mistral to instantiate a model of type trace_mistral. This is not supported for all configurations of models and can yield errors. Could this affect the accuracy?

I will continue testing the official pre-trained checkpoints on other datasets to further investigate the issue.

[gyxxyg]

In fact, we did not adjust any other parameters for evaluation. The warning occurs because the model name is 'videollama2_mistral' instead of 'trace_mistral'. You can try modifying the config.json file to resolve this issue. By the way, were you able to successfully reproduce our results for other fine-tuned checkpoints?

[Rorwey]

First, my sincere apologies for the delayed response. This was due to overlapping experimental schedules and some personal commitments that required urgent attention.

Regarding the fine-tuning experiments:
​​- YouCook2-BB & Charades​​: Results are consistent with the paper’s claims.
​​- QVHighlights​​: We’re unable to reproduce the reported performance despite extensive attempts.

[gyxxyg]

I think the all things should be correct if the Youcook and Charades results can be reproduced. For QVHighlights, did you use the JSON file we provided here?