Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
544f082
Update audioqna gateway to print text, in gateway.py
ctao456 Sep 26, 2024
7dd0b8a
Merge branch 'opea-project:main' into main
ctao456 Oct 2, 2024
254071d
Merge branch 'opea-project:main' into main
ctao456 Oct 7, 2024
f0c7a02
updates needed for demo
ctao456 Oct 7, 2024
81a5943
Merge branch 'ctao/demo' of https://github.com/ctao456/GenAIComps int…
ctao456 Oct 7, 2024
dbf8ad3
original pr content
ctao456 Oct 8, 2024
98cbb1c
Merge branch 'opea-project:main' into ctao/opea
ctao456 Oct 8, 2024
9d7c4f9
Merge branch 'opea-project:main' into main
ctao456 Oct 8, 2024
d85f33a
Revert "updates needed for demo"
ctao456 Oct 8, 2024
a8363b7
remove improper images
ctao456 Oct 8, 2024
4c22b40
Addressed some comments on previous pr
ctao456 Oct 8, 2024
dc59a77
Add Dockerfile for cpu support
ctao456 Oct 9, 2024
bf38204
CODEOWNER: Update comp CODEOWNER (#757)
hteeyeoh Oct 8, 2024
b932b94
Add stable diffusion microservice (#729)
XinyuYe-Intel Oct 8, 2024
7fb8c6b
Compatible with different platforms. (#766)
ZePan110 Oct 8, 2024
817c0f1
Optimize path and link validity check. (#745)
ZePan110 Oct 9, 2024
a14c76f
Add timeout for ut test (#773)
chensuyue Oct 9, 2024
8fae482
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 9, 2024
b9420c7
Merge branch 'opea-project:main' into ctao/opea
ctao456 Oct 9, 2024
1879a1d
test hyperlink
ctao456 Oct 10, 2024
cc938fb
test hyperlink
ctao456 Oct 10, 2024
697fdbd
test hyperlink issue
ctao456 Oct 10, 2024
120edae
test hyperlink issue
ctao456 Oct 10, 2024
e20ef65
Merge branch 'opea-project:main' into main
ctao456 Oct 10, 2024
4bfd1f3
put back hyperlinks in readme
ctao456 Oct 10, 2024
10b3b34
remove possible error hyperlink
ctao456 Oct 10, 2024
09ab934
put hyperlink back
ctao456 Oct 10, 2024
901adcc
Merge branch 'opea-project:main' into ctao/opea
ctao456 Oct 14, 2024
303eb1f
major update to use FastAPI for wav2lip, and structure component format
ctao456 Oct 15, 2024
d058496
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 15, 2024
d6b3508
Merge branch 'opea-project:main' into ctao/opea
ctao456 Oct 15, 2024
ff0ec2d
Add dockerfiles in animation-compose-cd.yaml
ctao456 Oct 16, 2024
87f51e6
Fix end of file issue in animation-compose-cd.yaml
ctao456 Oct 16, 2024
3520b9e
Merge branch 'main' into ctao/opea
chensuyue Oct 17, 2024
c4f43da
Fix Docker deployment on Xeon
ctao456 Oct 18, 2024
f41f974
Merge branch 'ctao/opea' of https://github.com/ctao456/GenAIComps int…
ctao456 Oct 18, 2024
b7752b5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 18, 2024
22cc67a
add versioning for all pip packages
ctao456 Oct 18, 2024
fd214e7
e2e test script for animation
ctao456 Oct 18, 2024
48a23f6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 18, 2024
a75bcc6
update e2e test script
ctao456 Oct 18, 2024
46d293e
Merge branch 'ctao/opea' of https://github.com/ctao456/GenAIComps int…
ctao456 Oct 18, 2024
f637e1e
update e2e test script
ctao456 Oct 18, 2024
a3e2148
Merge branch 'opea-project:main' into ctao/opea
ctao456 Oct 18, 2024
d9f152b
update readme
ctao456 Oct 18, 2024
e08f0fb
Merge branch 'ctao/opea' of https://github.com/ctao456/GenAIComps int…
ctao456 Oct 18, 2024
f2e6b34
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 18, 2024
72a6527
update
ctao456 Oct 19, 2024
dd849ef
update
ctao456 Oct 19, 2024
fecedab
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 19, 2024
f3f3343
update gateway
ctao456 Oct 19, 2024
8790139
udpate gateway
ctao456 Oct 19, 2024
26deb88
Fix AVATAR_CHATBOT
ctao456 Oct 19, 2024
5d8fef8
update gateway
ctao456 Oct 19, 2024
a0b4d13
update
ctao456 Oct 19, 2024
9a705ca
test
ctao456 Oct 19, 2024
e177ba2
update
ctao456 Oct 19, 2024
d36d0fe
update
ctao456 Oct 19, 2024
e90f5cf
update gateway
ctao456 Oct 19, 2024
dda2e31
fix max_tokens in AvatarChatbot gateway
ctao456 Oct 19, 2024
83186de
test
ctao456 Oct 19, 2024
38e4d46
update
ctao456 Oct 19, 2024
7aa56f4
update
ctao456 Oct 19, 2024
155ef8e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 19, 2024
0b7f385
Model download script moved from Dockerfiles to Docker entrypoint
ctao456 Oct 21, 2024
f3b08ca
update paths
ctao456 Oct 21, 2024
9b392bd
Correct paths in readme
ctao456 Oct 21, 2024
c19d8f8
Merge branch 'main' of https://github.com/ctao456/GenAIComps into cta…
ctao456 Oct 21, 2024
4181392
Merge branch 'opea-project:main' into main
ctao456 Oct 21, 2024
560a8fc
Merge branch 'main' of https://github.com/ctao456/GenAIComps into cta…
ctao456 Oct 21, 2024
8252bed
revert changes to audioqna gateway
ctao456 Oct 21, 2024
746778d
longer wait time after docker run
ctao456 Oct 22, 2024
fd8b026
add mount volume in test scripts
ctao456 Oct 22, 2024
00ce8a8
add volume mount in test scripts
ctao456 Oct 22, 2024
c45acbf
udpate test script
ctao456 Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions comps/animation/wav2lip/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ export WAV2LIP_PORT=7860
export ANIMATION_PORT=9066
export INFERENCE_MODE='wav2lip+gfpgan'
export CHECKPOINT_PATH='/usr/local/lib/python3.11/site-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
export FACE="assets/img/avatar1.jpg"
# export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
export AUDIO='None'
export FACESIZE=96
export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
export OUTFILE="assets/outputs/result.mp4"
export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
export UPSCALE_FACTOR=1
export FPS=10
Expand All @@ -61,11 +61,11 @@ export WAV2LIP_PORT=7860
export ANIMATION_PORT=9066
export INFERENCE_MODE='wav2lip+gfpgan'
export CHECKPOINT_PATH='/usr/local/lib/python3.10/dist-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
export FACE="assets/img/avatar1.jpg"
# export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
export AUDIO='None'
export FACESIZE=96
export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
export OUTFILE="assets/outputs/result.mp4"
export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
export UPSCALE_FACTOR=1
export FPS=10
Expand All @@ -78,13 +78,13 @@ export FPS=10
- Xeon CPU

```bash
docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
```

- Gaudi2 HPU

```bash
docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
```

## 2.2 Run Animation Microservice
Expand All @@ -100,8 +100,8 @@ Once microservice starts, user can use below script to validate the running micr
## 3.1 Validate Wav2Lip service

```bash
cd GenAIComps/comps/animation/wav2lip
python3 dependency/check_wav2lip_server.py
cd GenAIComps
python3 comps/animation/wav2lip/dependency/check_wav2lip_server.py
```

## 3.2 Validate Animation service
Expand All @@ -115,14 +115,14 @@ curl http://${ip_address}:9066/v1/animation -X POST -H "Content-Type: applicatio
or

```bash
cd GenAIComps/comps/animation/wav2lip
python3 check_animation_server.py
cd GenAIComps
python3 comps/animation/wav2lip/dependency/check_animation_server.py
```

The expected output is a message similar to the following:

```bash
{'wav2lip_result': '.../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
{'wav2lip_result': '....../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
```

Please find "comps/animation/wav2lip/assets/outputs/result.mp4" as a reference generated video.
Binary file modified comps/animation/wav2lip/assets/outputs/result.mp4
Binary file not shown.
2 changes: 1 addition & 1 deletion comps/animation/wav2lip/check_animation_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
outfile = os.environ.get("OUTFILE")

# Read the JSON file
with open("assets/audio/sample_question.json", "r") as file:
with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
data = json.load(file)

response = requests.post(url=endpoint, json=data, headers={"Content-Type": "application/json"}, proxies={"http": None})
Expand Down
2 changes: 0 additions & 2 deletions comps/animation/wav2lip/dependency/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.11/site-packages/gfpgan"

# Download pre-trained models
WORKDIR /usr/local/lib/python3.11/site-packages
RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh

# Install pip dependencies
RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt
Expand Down
2 changes: 0 additions & 2 deletions comps/animation/wav2lip/dependency/Dockerfile.intel_hpu
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.10/dist-packages/gfpgan"

# Download pre-trained models
WORKDIR /usr/local/lib/python3.10/dist-packages
RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh

# Install pip dependencies
RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
outfile = os.environ.get("OUTFILE")

# Read the JSON file
with open("assets/audio/sample_question.json", "r") as file:
with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
data = json.load(file)

inputs = {"audio": data["byte_str"]}
Expand Down
17 changes: 16 additions & 1 deletion comps/animation/wav2lip/dependency/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,14 +1,29 @@
#!/bin/sh
cd /home/user/comps/animation/wav2lip/ || exit

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

# Check the value of $DEVICE and cd to the download path accordingly
if [ "$DEVICE" = "hpu" ]; then
cd /usr/local/lib/python3.10/dist-packages
else
cd /usr/local/lib/python3.11/site-packages
fi

# Download model weights
wget https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth -O Wav2Lip/face_detection/detection/sfd/s3fd.pth
mkdir -p Wav2Lip/checkpoints
wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/Eb3LEzbfuKlJiR600lQWRxgBIY27JZg80f7V9jtMfbNDaQ?download=1" -O Wav2Lip/checkpoints/wav2lip.pth
wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/EdjI7bZlgApMqsVoEUUXpLsBxqXbn5z8VTmoxp55YNDcIA?download=1" -O Wav2Lip/checkpoints/wav2lip_gan.pth
wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth -P gfpgan/experiments/pretrained_models
echo "Face Detector, Wav2Lip, GFPGAN weights downloaded."

# Environment variables
export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES=1

# Wav2Lip, GFPGAN
cd /home/user/comps/animation/wav2lip/ || exit
python3 dependency/wav2lip_server.py \
--device $DEVICE \
--port $((WAV2LIP_PORT)) \
Expand Down
9 changes: 5 additions & 4 deletions tests/animation/test_animation_wav2lip.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ function build_docker_images() {
else
echo "opea/wav2lip built successful"
fi
docker build -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
docker build --no-cache -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
if [ $? -ne 0 ]; then
echo "opea/animation built fail"
exit 1
Expand All @@ -43,12 +43,13 @@ function start_service() {
export UPSCALE_FACTOR=1
export FPS=10

docker run -d --name="test-comps-animation-wav2lip" -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
docker run -d --name="test-comps-animation" -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
sleep 1m
docker run -d --name="test-comps-animation-wav2lip" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
docker run -d --name="test-comps-animation" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
sleep 3m
}

function validate_microservice() {
cd $WORKPATH
result=$(http_proxy="" curl http://localhost:9066/v1/animation -X POST -H "Content-Type: application/json" -d @comps/animation/wav2lip/assets/audio/sample_question.json)
if [[ $result == *"result.mp4"* ]]; then
echo "Result correct."
Expand Down