HuggingFace Model Deployment

Deploy HuggingFace models with one click. Get an OpenAI-compatible API endpoint in minutes.

Overview

The HuggingFace integration automatically:

1. Provisions a GPU instance
2. Downloads your selected model
3. Starts a vLLM inference server
4. Exposes an OpenAI-compatible API endpoint

Quick Start

1. Click HuggingFace in the sidebar
2. Search for a model or browse the catalog
3. Select a model
4. Choose your GPU configuration
5. Click Deploy

Your model will be ready in 5-10 minutes (depending on model size).

Recommended Models

These models are tested and optimized for Packet.ai deployment:

General Purpose

Coding Specialists

Small & Fast

Deployment Options

When deploying, you can configure:

Gated Models

Some models on HuggingFace require accepting terms before use. These include Llama, Gemma, and other popular models.

Setup Steps

1. Accept License: Visit the model page on HuggingFace and click "Agree and access repository"
   • Llama 3.1
   • Gemma 2
2. Create Access Token: Go to huggingface.co/settings/tokens
   • Click "New token"
   • Select "Read" access
   • Copy the token (starts with hf_)
3. Enter Token: Paste your token when deploying the model

Using Your Deployed Model

Once deployed, you'll receive an API endpoint like:

http://35.190.160.152:20000/v1

cURL

curl http://YOUR-IP:PORT/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "max_tokens": 100
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://YOUR-IP:PORT/v1",
    api_key="not-needed"  # No auth required for direct endpoint
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about GPUs"}
    ],
    max_tokens=100,
    temperature=0.7
)

print(response.choices[0].message.content)

Streaming Responses

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
    max_tokens=500
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

JavaScript/TypeScript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://YOUR-IP:PORT/v1',
  apiKey: 'not-needed',
});

const response = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Hello!' }
  ],
  maxTokens: 100,
});

console.log(response.choices[0].message.content);

API Endpoints

Your vLLM server exposes these endpoints:

Deployment Status

Your deployment goes through these stages:

GPU Sizing Guide

Model size determines GPU requirements. Use this guide to choose the right configuration:

Monitoring

View Deployment Logs

Click the Logs button on your deployment card to view real-time logs. You can also expand to full screen for detailed debugging.

Check Server Status

# Health check
curl http://YOUR-IP:PORT/health

# List loaded models
curl http://YOUR-IP:PORT/v1/models

# Check vLLM version
curl http://YOUR-IP:PORT/version

SSH Access for Debugging

For detailed debugging, SSH into your instance:

# Connect to instance
ssh -p <port> ubuntu@<host>

# View vLLM logs
tail -f ~/hf-workspace/vllm.log

# Check GPU utilization
nvidia-smi

# Watch GPU in real-time
watch -n 1 nvidia-smi

Using Persistent Storage

Enable persistent storage to:

• Cache downloaded models - Faster restarts (minutes → seconds)
• Save conversation logs - Keep inference logs
• Store fine-tuned adapters - Use custom LoRA adapters

With persistent storage, model downloads are cached and subsequent starts load directly from storage.

Troubleshooting

Model Not Loading

Check the deployment logs for errors. Common issues:

API Not Responding

1. Check if deployment status is "Running"
2. Verify the port is exposed (check the endpoint URL)
3. Wait for model loading to complete (check logs)
4. Try the health endpoint: curl http://YOUR-IP:PORT/health

Slow Responses

• First request slow: Model loading into GPU memory, subsequent requests faster
• Consistently slow: Check GPU utilization with nvidia-smi
• High latency: Consider a smaller model or more GPUs
• Timeouts: Reduce max_tokens or add streaming

Out of Memory

• Increase GPU count (Scale feature)
• Use a quantized model version (AWQ, GPTQ)
• Reduce max_model_len in vLLM config
• Try a smaller model variant

Need Help?

Contact us at support@packet.ai