ViQAgent

Data

The benchmarking data was not uploaded due to its size and autorship. But can be accesed on the corresponding repositories of each dataset. The ViQAgent solution was evaluated on four benchmarks, namely:

• ActivityNet-QA (Yu, Zhou, et al. 2019)
• NExT-QA (Xiao, Junbin, et al. 2021)
• EgoSchema (Mangalam, Karttikeya, et al. 2023)
• iVQA (Yang, Antoine, et al. 2022)

These datasets were not used for training, therefore only the test/validation splits were used. Both the NExT-QA and EgoSchema datasets are closed-ended, but the ActivityNet-QA and iVQA datasets don't contain answer options.

Datasets (Question-Answer)

Store the dataset repositories, in a subfolder named data.

Click for details on how to download...

ActivityNet-QA

git clone https://github.com/MILVLG/activitynet-qa ActivityNet_QA

NExT-QA

git clone https://github.com/doc-doc/NExT-QA NExT_QA

EgoSchema

git clone https://github.com/egoschema/EgoSchema EgoSchema

iVQA

The dataset is available in a drive folder shared by the authors in the repository.

Store the dataset in a folder named iVQA, inside the data folder.

Videos

Store the videos of each dataset in the corresponding dataset folder (data/<DATASET>), in a subfolder named videos.

Click for details on how to download...

ActivityNet-QA

Must request the videos to the ActivityNet team, through this form.

NExT-QA

The videos are available in a drive folder shared by the authors in the repository.

EgoSchema

The videos are available in a drive folder shared by the authors in the repository.

iVQA

The dataset's column video_id contains the video's id in the YouTube platform. The videos can be downloaded using the yt-dlp library; for ease, the following script can be used:

import yt_dlp as yt

def download_video(video_id):
    url = f"https://www.youtube.com/watch?v={video_id}"
    ydl_opts = {
        'format': f'best[height<={720}]',
        'outtmpl': './videos/%(id)s.%(ext)s',  
    }

    with yt.YoutubeDL(ydl_opts) as ydl:
        ydl.download([url])

However, as some of the videos are private or have been removed, the videos can be requested to the authors through the form in the same drive folder shared in the repository.