[WIP] Initiate animation component by ctao456 · Pull Request #400 · opea-project/GenAIComps

ctao456 · 2024-08-02T17:37:22Z

Description

Add a new component called "Animation", first PR includes Wav2Lip+GFPGAN model

Issues

opea-project/docs#59

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

Wav2Lip-GFPGAN
opea-project/GenAIExamples#523

Tests

Once microservice starts, user can use below script to validate the running microservice.

export ip_address=$(hostname -I | awk '{print $1}')
curl http://${ip_address}:7860/v1/animation -X POST -H "Content-Type: application/json" -d @comps/animation/assets/audio/sample_question.json

or

cd comps/animation
python3 test_animation_server.py

The expected output is a message prompting

"Video generated successfully, check $OUTFILE for the result."

Please find "outputs/result.mp4" in the current directory as a reference generated video.

Signed-off-by: Chun Tao <[email protected]>

Signed-off-by: ZhangJianyu <[email protected]>

Signed-off-by: Chun Tao <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Chun Tao <[email protected]>

…into ctao/animation Signed-off-by: Chun Tao <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Chun Tao <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Chun Tao <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Chun Tao <[email protected]>

…into ctao/animation

Signed-off-by: Chun Tao <[email protected]>

for more information, see https://pre-commit.ci

louie-tsai

looks good. still need more time to look into annimation.py.

.gitignore

comps/animation/Dockerfile_hpu

comps/animation/README.md

comps/animation/animation.py

comps/animation/test_animation_server.py

comps/animation/assets/flowchart.png

comps/animation/animation.py

louie-tsai · 2024-08-30T19:05:40Z

comps/animation/animation.py

+        full_frames = [cv2.imread(args.face)]
+        fps = args.fps
+    else:
+        video_stream = cv2.VideoCapture(args.face)


do you know where CV2 runs on? do they leverage Gaudi?

Could we leverage Gaudi media pipeline?
https://docs.habana.ai/en/latest/Media_Pipeline/Media_Pipeline.html?highlight=media

CV2 runs on cpu by default. If we want gpu support, we need to build CV2 from source. I don't see CV2 on Gaudi in the documentation.

Media pipeline is possible. Added as a TO-DO.

louie-tsai · 2024-08-30T19:16:21Z

comps/animation/animation.py

+            if args.resize_factor > 1:
+                frame = cv2.resize(frame, (frame.shape[1] // args.resize_factor, frame.shape[0] // args.resize_factor))
+            if args.rotate:
+                frame = cv2.rotate(frame, cv2.ROTATE_90_CLOCKWISE())


good to have an estimation how long CV2 tasks take.

louie-tsai · 2024-08-30T19:19:02Z

comps/animation/animation.py

+    # one single video frame corresponds to 80/25*0.01 = 0.032 seconds (or 32 milliseconds) of audio
+    # 30 fps video will match closer to audio, than 25 fps
+    mel_chunks = []
+    mel_idx_multiplier = 80.0 / fps


what does 80 mean here? good to give it a variable or definition instead of magic number

80 is the batch size of mel chunks during training, to be fused with 1 second video.

Paper here: https://arxiv.org/pdf/2008.10010
So 80/fps is how many mel chunks are used to select to associate one image frame.

louie-tsai · 2024-08-30T19:21:15Z

comps/animation/animation.py

+
+            frame_h, frame_w = full_frames[0].shape[:-1]
+            if args.inference_mode == "wav2clip_only":
+                out = cv2.VideoWriter("temp/result.avi", cv2.VideoWriter_fourcc(*"DIVX"), fps, (frame_w, frame_h))


also good to understand the time taken by cv2 here.

comps/animation/docker_run.sh

comps/animation/requirements.txt

comps/animation/utils.py

ctao456 and others added 30 commits June 23, 2024 16:43

add gitignore

3949911

Signed-off-by: Chun Tao <[email protected]>

Merge branch 'opea-project:main' into main

af823fa

Merge branch 'opea-project:main' into main

cbf26e2

first commit

5e1d51c

Signed-off-by: ZhangJianyu <[email protected]>

add TTS

79cb6dd

Signed-off-by: ZhangJianyu <[email protected]>

update

e6a62ef

Signed-off-by: Chun Tao <[email protected]>

update

65d4057

Signed-off-by: Chun Tao <[email protected]>

update

06fba67

Signed-off-by: Chun Tao <[email protected]>

update requirements.txt

bfa3ce9

Signed-off-by: Chun Tao <[email protected]>

update

6e529cc

Signed-off-by: Chun Tao <[email protected]>

update docker and requirements

f26db03

Signed-off-by: Chun Tao <[email protected]>

update

4f8f382

Signed-off-by: Chun Tao <[email protected]>

update pre-trained model download paths

c2f17f5

Signed-off-by: Chun Tao <[email protected]>

update

170c932

Signed-off-by: Chun Tao <[email protected]>

update app

e58fc51

Signed-off-by: Chun Tao <[email protected]>

updates

58835de

Signed-off-by: Chun Tao <[email protected]>

update path

01c4727

Signed-off-by: Chun Tao <[email protected]>

update path

9f1bb66

Signed-off-by: Chun Tao <[email protected]>

add custom patches

e0d0d7a

Signed-off-by: Chun Tao <[email protected]>

update

befd902

Signed-off-by: Chun Tao <[email protected]>

update

a52184d

Signed-off-by: Chun Tao <[email protected]>

update

34b4125

Signed-off-by: Chun Tao <[email protected]>

update

f49fffa

Signed-off-by: Chun Tao <[email protected]>

updates

132ee6a

Signed-off-by: Chun Tao <[email protected]>

updates

29d0b1c

Signed-off-by: Chun Tao <[email protected]>

update

7c1e68b

Signed-off-by: Chun Tao <[email protected]>

update

d7a0942

Signed-off-by: Chun Tao <[email protected]>

updates

0b76255

Signed-off-by: Chun Tao <[email protected]>

updates

72c7b29

Signed-off-by: Chun Tao <[email protected]>

update to include gdown paths

c73e132

Signed-off-by: Chun Tao <[email protected]>

ctao456 mentioned this pull request Aug 4, 2024

[WIP] Add a Avatar Chatbot (Audio) example opea-project/GenAIExamples#523

Closed

4 tasks

ctao456 and others added 11 commits August 4, 2024 13:48

delete unnecessary ui file (included in Megaservice)

4345f8d

Signed-off-by: Chun Tao <[email protected]>

add ffmpeg-python in requirements.txt

cf6ec7e

Signed-off-by: Chun Tao <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

19ca82e

for more information, see https://pre-commit.ci

Use ffmpeg-python

53ab0b8

Signed-off-by: Chun Tao <[email protected]>

Merge branch 'ctao/animation' of https://github.com/ctao456/GenAIComps …

1589fd6

…into ctao/animation Signed-off-by: Chun Tao <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

eac9d60

for more information, see https://pre-commit.ci

sample ffmpeg

b42878e

Signed-off-by: Chun Tao <[email protected]>

ffmpeg

a6e3771

Signed-off-by: Chun Tao <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

5e774de

for more information, see https://pre-commit.ci

Merge branch 'opea-project:main' into ctao/animation

ebd588a

Merge branch 'main' into ctao/animation

04b8f9e

ctao456 requested a review from lvliang-intel as a code owner August 5, 2024 14:27

ctao456 mentioned this pull request Aug 5, 2024

RFC - AI Avatar Animation Design opea-project/docs#59

Merged

ctao456 and others added 12 commits August 5, 2024 15:55

switch from git clone to pip install wav2lip, gfpgan

b2a97e5

Signed-off-by: Chun Tao <[email protected]>

add --no-deps flag in pip install command

920fdd2

Signed-off-by: Chun Tao <[email protected]>

test

b509869

Signed-off-by: Chun Tao <[email protected]>

test

6b10015

Signed-off-by: Chun Tao <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9cac555

for more information, see https://pre-commit.ci

test

42b310a

Signed-off-by: Chun Tao <[email protected]>

Merge branch 'ctao/animation' of https://github.com/ctao456/GenAIComps …

9e800a0

…into ctao/animation

test

9b7c340

Signed-off-by: Chun Tao <[email protected]>

pip install

643a07a

Signed-off-by: Chun Tao <[email protected]>

update

b98e032

Signed-off-by: Chun Tao <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3140a34

for more information, see https://pre-commit.ci

Merge branch 'opea-project:main' into ctao/animation

41c4990

louie-tsai reviewed Aug 19, 2024

View reviewed changes

.gitignore Show resolved Hide resolved

comps/animation/Dockerfile_hpu Show resolved Hide resolved

comps/animation/README.md Show resolved Hide resolved

comps/animation/animation.py Show resolved Hide resolved

comps/animation/test_animation_server.py Show resolved Hide resolved

ctao456 closed this Aug 21, 2024

louie-tsai reviewed Aug 30, 2024

View reviewed changes

ctao456 mentioned this pull request Oct 9, 2024

Initiate "animation" component #775

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Initiate animation component#400

[WIP] Initiate animation component#400
ctao456 wants to merge 120 commits intoopea-project:mainfrom
ctao456:ctao/animation

ctao456 commented Aug 2, 2024 •

edited

Loading

Uh oh!

louie-tsai left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

louie-tsai Aug 30, 2024

Uh oh!

ctao456 Oct 9, 2024

Uh oh!

louie-tsai Aug 30, 2024

Uh oh!

louie-tsai Aug 30, 2024

Uh oh!

ctao456 Oct 9, 2024 •

edited

Loading

Uh oh!

louie-tsai Aug 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ctao456 commented Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

louie-tsai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

louie-tsai Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

ctao456 Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

louie-tsai Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

ctao456 Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louie-tsai Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ctao456 commented Aug 2, 2024 •

edited

Loading

ctao456 Oct 9, 2024 •

edited

Loading