fix: add health check + clarify volume mounts

- nginx: return 200 OK on GET / for load balancer health checks
- Dockerfile: set HF_HOME=/app/models so Hub downloads persist in mounted volume
- Dockerfile: add /app/data directory and volume declaration
- docker-compose: explicit volume mounts for models, data, lora, output
- README: document where to put training files and find output
- .gitignore: exclude volume mount directories (models/, data/, lora/, output/)
This commit is contained in:
Daniel Cox
2026-08-11 21:46:24 +09:30
parent 85df305079
commit da5b2da097
5 changed files with 100 additions and 25 deletions

6
.gitignore vendored
View File

@@ -5,3 +5,9 @@ voxcpm.egg-info
.DS_Store .DS_Store
./pretrained_models/ ./pretrained_models/
app_local.py app_local.py
# Docker volume mount directories (large files, user-specific)
models/
data/
lora/
output/

View File

@@ -38,14 +38,15 @@ RUN pip install --no-cache-dir -e .
COPY . /app/ COPY . /app/
# Create default directories and declare volumes # Create default directories and declare volumes
RUN mkdir -p /app/lora /app/models /app/output RUN mkdir -p /app/lora /app/models /app/output /app/data
VOLUME ["/app/models", "/app/lora", "/app/output"] VOLUME ["/app/models", "/app/lora", "/app/output", "/app/data"]
EXPOSE 7860 EXPOSE 7860
# Environment variables for configuration # Environment variables for configuration
ENV GRADIO_SERVER_PORT=7860 ENV GRADIO_SERVER_PORT=7860
ENV GRADIO_ROOT_PATH="" ENV GRADIO_ROOT_PATH=""
ENV HF_HOME=/app/models
# Default: launch training WebUI # Default: launch training WebUI
CMD ["python", "lora_ft_webui.py"] CMD ["python", "lora_ft_webui.py"]

View File

@@ -21,6 +21,75 @@ This starts:
Access the WebUI at **http://localhost/webui/**. Access the WebUI at **http://localhost/webui/**.
## Volume Mounts
The compose file maps host directories to container paths. Create these directories at the project root before starting:
```
VoxCPM/
├── docker/
│ ├── docker-compose.yml
│ ├── Dockerfile
│ └── nginx.conf
├── models/ ← Pretrained model weights (or auto-downloaded via HF)
│ ├── openbmb__VoxCPM2/
│ └── openbmb__VoxCPM1.5/
├── data/ ← Training manifests + audio files
│ ├── train.jsonl
│ ├── val.jsonl (optional)
│ └── audio/
│ ├── speaker1_001.wav
│ └── ...
├── lora/ ← LoRA training output (created automatically)
│ └── my-voice-2024/
│ ├── checkpoints/
│ ├── logs/
│ └── train_config.yaml
└── output/ ← Additional training artifacts
```
### Mount Reference
| Host Path | Container Path | Purpose |
|-----------|---------------|---------|
| `./models/` | `/app/models` | Pretrained model weights and HF cache (`HF_HOME`). Pre-populate with model dirs (e.g., `openbmb__VoxCPM2/`) or leave empty — models auto-download on first run and persist here. |
| `./data/` | `/app/data` | Training data. Put JSONL manifests and audio files here. In the WebUI, reference paths as `/app/data/train.jsonl`. |
| `./lora/` | `/app/lora` | LoRA checkpoint output. After training, find results in `lora/<run-name>/checkpoints/`. Also used to resume training from existing checkpoints. |
| `./output/` | `/app/output` | Miscellaneous training artifacts. |
### Training Data Format
The train manifest is a JSONL file where each line references an audio file:
```json
{"audio_path": "/app/data/audio/speaker1_001.wav", "text": "Hello world", "speaker": "speaker1"}
```
Use absolute container paths (`/app/data/...`) in your manifest so the container can find the files.
### Models
If `models/openbmb__VoxCPM2/` exists on the host, the app loads directly from that path — no network access needed. If the directory is empty or missing, `from_pretrained` falls back to `snapshot_download` from HuggingFace Hub.
The Dockerfile sets `HF_HOME=/app/models` so any Hub downloads land in the same mounted volume (matching the pattern in `deploy/Dockerfile.voxcpm-unified`). This means models persist across container restarts regardless of whether they were pre-populated or auto-downloaded.
**Recommended:** Pre-populate to avoid first-run download delay:
```bash
huggingface-cli download openbmb/VoxCPM2 --local-dir ./models/openbmb__VoxCPM2
```
The Dockerfile creates empty `/app/models`, `/app/lora`, `/app/output` directories, but the volume mounts override them with your host directories.
## Health Check
The nginx proxy responds with `200 OK` on `GET /` for load balancer health checks (AWS ALB, etc.). This is separate from the WebUI at `/webui/`.
```bash
curl http://localhost/
# OK
```
## Direct Access (no proxy) ## Direct Access (no proxy)
If you want to bypass nginx and access Gradio directly: If you want to bypass nginx and access Gradio directly:
@@ -40,23 +109,12 @@ docker build -f docker/Dockerfile -t voxcpm-training .
# Run with GPU access (no reverse proxy) # Run with GPU access (no reverse proxy)
docker run --gpus all -p 7860:7860 \ docker run --gpus all -p 7860:7860 \
-v ./models:/app/models \ -v ./models:/app/models \
-v ./data:/app/data \
-v ./lora:/app/lora \ -v ./lora:/app/lora \
-v ./output:/app/output \ -v ./output:/app/output \
voxcpm-training voxcpm-training
``` ```
## Model Weights
Models are **auto-downloaded** from HuggingFace Hub on first use. The `/app/models` volume persists them across container restarts so they don't need to be re-downloaded.
To pre-populate (avoids download at startup):
```
models/
├── openbmb__VoxCPM2/ # VoxCPM2 (preferred)
└── openbmb__VoxCPM1.5/ # VoxCPM1.5 (fallback)
```
## Environment Variables ## Environment Variables
| Variable | Default | Description | | Variable | Default | Description |
@@ -88,17 +146,10 @@ Training subprocess output is streamed to stdout, visible via:
docker compose -f docker/docker-compose.yml logs -f training-webui docker compose -f docker/docker-compose.yml logs -f training-webui
``` ```
## Volumes
| Mount Point | Purpose |
|-------------|---------|
| `/app/models` | Pre-trained model weights (read-only OK) |
| `/app/lora` | LoRA checkpoints — training output is saved here |
| `/app/output` | Additional training artifacts |
## Troubleshooting ## Troubleshooting
- **"no NVIDIA GPU detected"**: Ensure the NVIDIA Container Toolkit is installed and `docker run --gpus all nvidia-smi` works. - **"no NVIDIA GPU detected"**: Ensure the NVIDIA Container Toolkit is installed and `docker run --gpus all nvidia-smi` works.
- **OOM errors**: Reduce batch size in the WebUI or use a GPU with more VRAM. - **OOM errors**: Reduce batch size in the WebUI or use a GPU with more VRAM.
- **WebUI not accessible**: Check that port 80 (nginx) or 7860 (direct) isn't blocked by a firewall. - **WebUI not accessible**: Check that port 80 (nginx) or 7860 (direct) isn't blocked by a firewall.
- **WebSocket errors behind proxy**: Ensure your proxy forwards `Upgrade` and `Connection` headers (the included nginx.conf handles this). - **WebSocket errors behind proxy**: Ensure your proxy forwards `Upgrade` and `Connection` headers (the included nginx.conf handles this).
- **Health check failing**: Ensure nginx is running — `curl http://localhost/` should return `OK`.

View File

@@ -8,9 +8,20 @@ services:
ports: ports:
- "7860:7860" - "7860:7860"
volumes: volumes:
- ../models:/app/models # Pre-downloaded model weights # Pretrained model weights + HF cache (HF_HOME=/app/models in Dockerfile).
- ../lora:/app/lora # LoRA checkpoints (input/output) # Pre-populate with model dirs, or leave empty — auto-downloads on first run.
- ../output:/app/output # Training output artifacts - ../models:/app/models
# Training data: JSONL manifests and audio files.
# Reference paths inside the container as /app/data/train.jsonl etc.
- ../data:/app/data
# LoRA training output — checkpoints, configs, logs.
# Results appear in lora/<run-name>/checkpoints/ after training.
- ../lora:/app/lora
# Additional training artifacts.
- ../output:/app/output
deploy: deploy:
resources: resources:
reservations: reservations:

View File

@@ -2,6 +2,12 @@ server {
listen 80; listen 80;
server_name _; server_name _;
# Health check for load balancers (AWS ALB, etc.)
location = / {
return 200 'OK\n';
add_header Content-Type text/plain;
}
location /webui/ { location /webui/ {
proxy_pass http://training-webui:7860/; proxy_pass http://training-webui:7860/;
proxy_set_header Host $host; proxy_set_header Host $host;