Skip to content

Commit c84b2cd

Browse files
use official docker image (#10)
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
1 parent 8b29234 commit c84b2cd

2 files changed

Lines changed: 12 additions & 3 deletions

File tree

ChatQnA/README.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,16 @@ This ChatQnA use case performs RAG using LangChain, Redis vectordb and Text Gene
33
# Environment Setup
44
To use [🤗 text-generation-inference](https://github.com/huggingface/text-generation-inference) on Habana Gaudi/Gaudi2, please follow these steps:
55

6-
## Build TGI Gaudi Docker Image
6+
## Prepare Docker
7+
8+
Getting started is straightforward with the official Docker container. Simply pull the image using:
9+
10+
```bash
11+
docker pull ghcr.io/huggingface/tgi-gaudi:1.2.1
12+
```
13+
14+
Alternatively, you can build the Docker image yourself with:
15+
716
```bash
817
bash ./serving/tgi_gaudi/build_docker.sh
918
```

ChatQnA/serving/tgi_gaudi/launch_tgi_service.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ volume=$PWD/data
2727

2828
# Build the Docker run command based on the number of cards
2929
if [ "$num_cards" -eq 1 ]; then
30-
docker_cmd="docker run -p $port_number:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy tgi_gaudi --model-id $model_name"
30+
docker_cmd="docker run -p $port_number:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/tgi-gaudi --model-id $model_name"
3131
else
32-
docker_cmd="docker run -p $port_number:80 -v $volume:/data --runtime=habana -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy tgi_gaudi --model-id $model_name --sharded true --num-shard $num_cards"
32+
docker_cmd="docker run -p $port_number:80 -v $volume:/data --runtime=habana -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e HTTPS_PROXY=$https_proxy -e HTTP_PROXY=$https_proxy ghcr.io/huggingface/tgi-gaudi --model-id $model_name --sharded true --num-shard $num_cards"
3333
fi
3434

3535
# Execute the Docker run command

0 commit comments

Comments
 (0)