ULIP2 Encoder

A containerized ULIP2 (Unified Language-Image-Point cloud) encoder for generating embeddings from 3D models, point clouds, and text descriptions. This tool enables multimodal similarity search and retrieval tasks between 3D geometry and natural language.

NOTE: To run it as a containerized local server with additional functionalities, check the server branch!

Features

• Multi-format 3D Support: Accepts .npy, .obj files and more
• Text Encoding: Generate embeddings from natural language descriptions
• Batch Processing: Encode entire directories of 3D files
• Similarity Comparison: Compute cosine similarity between 3D models and text
• Docker Support: Fully containerized with GPU acceleration
• Flexible Output: Save embeddings to custom locations or alongside input files

Requirements

• Docker with GPU support (NVIDIA Container Toolkit)
• NVIDIA GPU with CUDA support
• 10G disk space to hold model weights

Quick Start

1. Setup

Clone the repository with ULIP submodule:

git clone --recurse-submodules <repository-url>
cd ulip2_encoder

Download the pretrained model weights (in total ~10G) from sfxx/ulip huggingface:

bash ./download_models.sh

• uses ULIP-2-PointBERT-10k-xyzrgb-pc-vit_g-objaverse_shapenet-pretrained.pt by default
• also credits to the OpenCLIP ViT model ULIP2 uses: laion/CLIP-ViT-bigG-14-laion2B-39B-b160k

2. Build Docker Image

docker build -t ulip2_encoder .

3. Run Container

Basic usage (with model weights mounted):

docker run -it --gpus all \
  -v $(pwd)/weights/ulip_models:/app/ULIP/ulip_models \
  ulip2_encoder

With payload folder (recommended for processing your own files):

docker run -it --gpus all \
  -v $(pwd)/weights/ulip_models:/app/ULIP/ulip_models \
  -v $(pwd)/payload:/app/ULIP/payload \
  ulip2_encoder

If experiencing OOM errors, try adding --shm-size=16G flag after gpus all.

Usage Examples

Single File Encoding

Encode an OBJ file:

python ulip2_encoder.py --obj_file payload/model.obj

Encode a point cloud:

python ulip2_encoder.py --pc_file payload/pointcloud.ply

Encode text description:

python ulip2_encoder.py --text "a red wooden chair with metal legs"

Batch Processing

Encode all 3D files in a directory:

python ulip2_encoder.py --obj_dir payload/models/

Encode directory with custom output location:

python ulip2_encoder.py --obj_dir payload/models/ --output_dir embeddings/

Similarity Comparison

Compare 3D model with text description:

python ulip2_encoder.py --obj_file payload/chair.obj --text "a comfortable office chair" --compare

Custom Output Locations

Specify output directory:

python ulip2_encoder.py --obj_file model.obj --output_dir embeddings/

Specify exact output file:

python ulip2_encoder.py --obj_file model.obj --output_file my_embedding.npy

Command Line Options

Option	Description	Default
--model_path	Path to ULIP2 pretrained model	ulip_models/ULIP-2-PointBERT-10k-xyzrgb-pc-vit_g-objaverse_shapenet-pretrained.pt
--device	Device for inference (cuda or cpu)	cuda
--num_points	Number of points to sample from 3D models	10000
--obj_file	Single OBJ file to encode	-
--obj_dir	Directory containing 3D files	-
--pc_file	Single point cloud file to encode	-
--text	Text description to encode	-
--output_dir	Directory to save embeddings	Next to input file
--output_file	Specific output file for single embedding	-
--compare	Compare 3D model with text description (prints out similarity score)	false

Supported File Formats

3D Models

• OBJ files (.obj): Wavefront OBJ format with optional MTL materials
• PLY files (.ply): Polygon File Format [not tested]
• PCD files (.pcd): Point Cloud Data format [not tested]
• NumPy arrays (.npy): Point cloud data as numpy arrays
• Text files (.txt): Space-separated XYZ or XYZRGB coordinates [not tested]

Output Format

• Embeddings: NumPy arrays saved as .npy files
• Shape: (1, 1280) - 1280-dimensional feature vectors
• Normalization: L2-normalized for cosine similarity computation

Troubleshooting

Common Issues

"Model file not found" error:

• Ensure you ran bash ./download_models.sh
• Verify the model weights are properly mounted in Docker

CUDA out of memory:

• Reduce --num_points parameter
• Use --device cpu for CPU-only inference

File Transfer

Copy files to running container:

docker cp local_file.obj container_name:/app/ULIP/payload/

Copy embeddings from container:

docker cp container_name:/app/ULIP/payload/embeddings/ ./embeddings/

Advanced Usage

Custom Model Paths

Other model weights from ULIP2 HF are not tested:

python ulip2_encoder.py --model_path custom_models/another_ulip2_model.pt --obj_file model.obj

Development

Applying Patches

The repository includes patches for the ULIP codebase:

./apply_patches_to_ulip.sh  # Applied automatically in Dockerfile

• for patches, reaon not opening a PR instead, is that the original repo has been claimed to be archived and not actively maintained anymore.

Restoring Original ULIP submodule

./restore_ulip.sh

Limitations

• Color Information: Uncolored point clouds receive dummy RGB values (gray)

License

This project uses the ULIP framework. Please refer to the original ULIP repository for licensing information.

Notes

Bumped a lot of dependencies to latest (torch 2.7.1 and cuda 12.8) and fixed a lot of bugs in the original code base (hence the patch/ folder)...

This will not be possible without:

• ULIP
• issues, pull requests and discussion in the original ULIP repository
• Copilot with Claude Sonnet 4