Skip to content

Dynamic wait for lvm service in MultimodalQnA tests #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mhbuehler merged 5 commits into from
Dec 18, 2024
Merged

Dynamic wait for lvm service in MultimodalQnA tests #33

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2ff09e2
Adds dynamic wait to MMQnA tests
mhbuehler Dec 12, 2024
8079628
Fix for empty grep result
mhbuehler Dec 16, 2024
ba8b0f0
Adjusted test for gaudi
mhbuehler Dec 16, 2024
e62695a
Fixed for Gaudi service check
mhbuehler Dec 17, 2024
e8afc1b
Added warning to check_service_ready function
mhbuehler Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion MultimodalQnA/tests/test_compose_on_gaudi.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,24 @@ export image_fn="apple.png"
export video_fn="WeAreGoingOnBullrun.mp4"
export caption_fn="apple.txt"

function check_service_ready() {
local container_name="$1"
local max_retries="$2"
local log_string="$3"

for i in $(seq 1 "$max_retries")
do
service_logs=$(docker logs "$container_name" 2>&1 | grep "$log_string" || true)
if [[ -z "$service_logs" ]]; then
echo "The $container_name service is not ready yet, sleeping 30s..."
sleep 30s
else
echo "$container_name service is ready"
break
fi
done
}

function build_docker_images() {
cd $WORKPATH/docker_image_build
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
Expand Down Expand Up @@ -201,7 +219,13 @@ function validate_microservices() {
"retriever-multimodal-redis" \
"{\"text\":\"test\",\"embedding\":${your_embedding}}"

sleep 3m
echo "Wait for tgi-llava-gaudi-server service to be ready"
check_service_ready "tgi-llava-gaudi-server" 10 "Connected"

if [[ $i -ge 10 ]]; then
echo "WARNING: Max retries reached when waiting for the lvm-llava service to be ready"
Copy link
Copy Markdown
Collaborator

@dmsuehir dmsuehir Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "WARNING: Max retries reached when waiting for the lvm-llava service to be ready"
echo "WARNING: Max retries reached when waiting for the tgi-llava-gaudi-server service to be ready"

docker logs lvm-llava >> ${LOG_PATH}/lvm_llava_file.log
Copy link
Copy Markdown
Collaborator

@dmsuehir dmsuehir Dec 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
docker logs lvm-llava >> ${LOG_PATH}/lvm_llava_file.log
docker logs tgi-llava-gaudi-server >> ${LOG_PATH}/tgi_llava_gaudi_server_file.log

fi

# llava server
echo "Evaluating LLAVA tgi-gaudi"
Expand Down
21 changes: 20 additions & 1 deletion MultimodalQnA/tests/test_compose_on_xeon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,24 @@ export image_fn="apple.png"
export video_fn="WeAreGoingOnBullrun.mp4"
export caption_fn="apple.txt"

function check_service_ready() {
local container_name="$1"
local max_retries="$2"
local log_string="$3"

for i in $(seq 1 "$max_retries")
do
service_logs=$(docker logs "$container_name" 2>&1 | grep "$log_string" || true)
if [[ -z "$service_logs" ]]; then
echo "The $container_name service is not ready yet, sleeping 30s..."
sleep 30s
else
echo "$container_name service is ready"
break
fi
done
}

function build_docker_images() {
cd $WORKPATH/docker_image_build
git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
Expand Down Expand Up @@ -199,7 +217,8 @@ function validate_microservices() {
"retriever-multimodal-redis" \
"{\"text\":\"test\",\"embedding\":${your_embedding}}"

sleep 3m
echo "Wait for lvm-llava service to be ready"
check_service_ready "lvm-llava" 10 "Uvicorn running on http://"

Comment thread
dmsuehir marked this conversation as resolved.
# llava server
echo "Evaluating lvm-llava"
Expand Down