Agent-based Video Trimming

Agent-based Video Trimming
Lingfeng Yang^1†, Zhenyuan Chen^2†, Xiang Li^3,2∗, Peiyang Jia⁴, Liangqu Long⁴, Jian Yang^1∗
1Nanjing University of Science and Technology, ²VCIP, CS, Nankai University, ³NKIARI, Shenzhen Futian, ⁴Insta360
{yanglfnjust, csjyang}@njust.edu.cn, [email protected],
[email protected], [email protected], [email protected]
[Arxiv] [Project Page] [Demo]

---

A comparison between our new task and existing video tasks: (a) Highlight Detection retrieves clips above a saliency threshold. (b) Moment Retrieval identifies the start and end for intervals related to a given query. (c) Video Summarization extracts keyframes for each theme of the video. (d) Video Trimming addresses more than just a retrieval task by also filtering wasted footage and logically composing the selected segments.

The overall framework of AVT. The approach first (a) converts sampled video content into structured captions and attributes, then (b) discards defective clips, and finally (c) organizes the remaining clips into a coherent final cut.

🛠 Data Preparation

data
├── avt
│   ├── caption
│   ├── output
│   │   ├── caption
│   │   ├── evaluation
│   │   └── story
│   ├── section_data
│   └── video
│        ├── daily_life-family-CapperCoolCooper
│        │   └── KRqR6LSoP8.mp4
│        ├── daily_life-family-Earls_Family_Vlogs
│        │   └── MyLwV1V19WY.mkv
│         ...
├── youtube_highlights
│   │    
│    ...
└── tvsum
    │   
     ...

🚀 How to start

Get Section Data

python tools/get_section_data.py --config data/avt/get_section_data.yaml

This command would generate ./data/avt/section_data.

Video Structuring

python tools/get_caption.py --config data/avt/get_caption.yaml

This command would generate ./data/avt/caption.

Story Composition

python tools/get_story.py --config data/avt/get_story.yaml

This command would generate ./data/avt/output/caption and ./data/avt/output/story.

Output Video

python tools/get_video.py --config data/avt/get_video.yaml

This command would generate ./data/avt/output/story/*/output_video.mp4.

Video Evaluation

python tools/get_evaluation.py --config data/avt/get_evaluation.yaml

This command would generate ./data/avt/output/evaluation.

🎥 Visualization

Visualization of trimmed videos on the video trimming dataset. AVT creates a more complete storyline with more highlight footage and less wasted footage.

Checkout the videos on our project page :)

⭐ Citation

If you find our paper or repo helpful for your research, please consider citing the following paper and giving this repo a star. Thank you!

@article{avt,
  title={Agent-based Video Trimming},
  author={Yang, Lingfeng and Chen, Zhenyuan and Li, Xiang and Jia, Peiyang and Long, Liangqu and Yang, Jian},
  journal={arXiv preprint arXiv:2412.09513},
  year={2024}
}