Community project by @kadirb4rut integrating VoxCPM2 as the TTS/voice-cloning
backend of a local-first video dubbing pipeline (issue #384).
Co-authored-by: Labmem-Zhouyx <913703649@qq.com>
- nginx: return 200 OK on GET / for load balancer health checks
- Dockerfile: set HF_HOME=/app/models so Hub downloads persist in mounted volume
- Dockerfile: add /app/data directory and volume declaration
- docker-compose: explicit volume mounts for models, data, lora, output
- README: document where to put training files and find output
- .gitignore: exclude volume mount directories (models/, data/, lora/, output/)
Add Docker infrastructure for running the training WebUI in containers:
- Dockerfile based on PyTorch CUDA base image with layer-cached deps
- docker-compose.yml with GPU support and nginx reverse proxy
- nginx.conf with WebSocket support for Gradio
Code fixes for container environments:
- Stream training subprocess stdout/stderr to Docker logs
- Support GRADIO_ROOT_PATH env var for reverse proxy (nginx/Traefik)
- Echo startup URL to stdout for container log discovery
All changes are backward-compatible: without Docker or env vars,
behavior is identical to before.
Add audio.cpp (ggml-based unified C++ inference framework) to the
Ecosystem & Community table in both README.md and README_zh.md.
VoxCPM2 is a released, supported model in that project.
Requested in #354.
Co-authored-by: Labmem-Zhouyx <913703649@qq.com>
The fine-tuning resume path in scripts/train_voxcpm_finetune.py called
torch.load() without weights_only=True for the LoRA checkpoint, full-model
checkpoint, optimizer, and scheduler. Loading an attacker-supplied checkpoint
directory would therefore execute arbitrary code during unpickling.
Every inference-time loader (model/voxcpm.py, model/voxcpm2.py, LoRA loading)
already passes weights_only=True, and the project ships a test asserting LoRA
loading rejects malicious pickle payloads. This closes the remaining gap so the
resume path matches that posture.
Also:
- app.py: add a --host flag so the Gradio server can bind to 127.0.0.1 instead
of being hardwired to 0.0.0.0 (default unchanged to preserve current behavior).
The UI exposes an unauthenticated api_name="generate" endpoint.
- tests/test_torch_load_safety.py: AST guard asserting every torch.load across
src/, scripts/, app.py and lora_ft_webui.py sets weights_only=True, plus a
behavioral check that a malicious pickle is blocked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
from_pretrained(lora_weights_path=...) built a default r=8 LoRAConfig and crashed for checkpoints trained with other ranks (e.g. r=32). Load the checkpoint's lora_config.json so r/alpha match; fall back to the default.
- Exposed 'seed' parameter in VoxCPMModel and VoxCPM2Model generation methods.
- Added PyTorch RNG seed setting before inference runs.
- Handled 'retry_badcase' seed adjustment by incrementing the seed value on retries.
- Exposed 'self.last_successful_seed' as a model attribute for UI integrations.
- Propagated 'seed' parameter to high-level pipeline class and CLI tools (cli.py).
- Added '--seed' flag to full-finetune and LoRA inference scripts.
- Configured validation audio generation in training script to use a fixed seed for objective comparison on TensorBoard.
- Added comprehensive unit tests in CLI test files to validate seed parsing and propagation.
- Updated English and Chinese READMEs with seed usage examples.
Pass the selected LoRA checkpoint to load_model() on first load so the
model initializes with the correct rank from lora_config.json instead of
always defaulting to r=32.
On subsequent LoRA hot-swaps, detect rank incompatibility and
automatically reload the model with the new checkpoint's config,
preventing tensor shape mismatch errors (fixes#283).
Made-with: Cursor
Move `import soundfile as sf` and `from voxcpm.core import VoxCPM` from
module-level into the functions that require model inference (load_model,
_run_single, cmd_batch), so `voxcpm validate` can run without loading
the model/inference stack.
Pass manifest path via --manifest flag (required) instead of as a
positional argument, so the test exercises cmd_validate rather than
argparse error handling. Also assert returncode==1 and check stderr
for the FAILED/error message to prevent false positives.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Invalid audio rows (bad path or sample-rate mismatch) no longer
increment valid_samples; has_error is now set on any audio failure
- _check_audio_file now enforces the expected sample rate when soundfile
is available, making --sample-rate actually useful
- ref_audio missing-file warning is emitted for every invalid entry
independently, not only before the first valid one is seen
- New tests cover each of the four corrected behaviours: invalid audio
count, sample-rate mismatch, mixed ref_audio, and CLI exit code
Drop "half" from _VALID_DTYPE_OVERRIDES / _LOW_PRECISION_DTYPES.
get_dtype() has never accepted "half", so VOXCPM_MPS_DTYPE=half would
pass override validation and then crash downstream with
"Unsupported dtype: half". The remaining aliases (bfloat16/bf16,
float16/fp16, float32/fp32) already cover the intended dtype space.
Adds a standalone unit check under scripts/ to guard the invariant
that every accepted override parses through get_dtype().
Addresses review feedback on #263.
Added regex to strip parentheses from control instructions in the text synthesis method to ensure compatibility with the expected prompt format. This change improves the robustness of the input handling.
LoRA is a first-class workflow in VoxCPM, and the project already prefers
safetensors plus weights-only fallback loading for base model artifacts. The
legacy LoRA .ckpt/.pth path was the remaining place that still deserialized
arbitrary pickle objects, so this switches it to weights_only=True and adds
focused regression coverage for both model loaders.
Constraint: Must preserve compatibility with tensor-only legacy LoRA checkpoints
Rejected: Remove .ckpt/.pth support entirely | too disruptive for existing users
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep LoRA artifact handling aligned with the existing safetensors-first, weights-only loading pattern
Tested: python3 -m pytest -q tests/test_lora_checkpoint_loading.py tests/test_model_utils.py -q
Not-tested: Full end-to-end LoRA hot-load with heavyweight model assets
Document vLLM-Omni as a production serving option for VoxCPM2
alongside the existing Nano-vLLM reference. Mirrors the addition in
README_zh.md, and adds an ecosystem table entry.
Install snippet follows the upstream vLLM-Omni installation guide
(from source, since vllm-omni is rapidly evolving).
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
- StreamingVAEDecoder caches CausalConv1d/CausalTransposeConv1d left-pad
state between calls — one patch in, one patch out, no overlap
- _inference yields single-patch latents in streaming mode
- 2x faster streaming VAE decode, more accurate (max diff 0.0005 vs 0.0011)
VoxCPM checkpoints default to bfloat16. Following commit e4e0496 which
added MPS device routing, running with `device=mps` selects bf16 on
Apple Silicon. On Metal, bf16 introduces enough numerical drift in the
diffusion AR loop that the synthesized audio is glitched and trips the
model's badcase detector, which retries until the per-call retry budget
is exhausted. Effectively MPS support is unusable in the default config.
This patch adds a single helper, `pick_runtime_dtype(device, dtype)`,
that promotes any low-precision dtype to float32 when the resolved
device is `mps`. CUDA and CPU paths are untouched. An opt-out env var
`VOXCPM_MPS_DTYPE` lets users force a specific dtype on MPS once future
PyTorch / macOS releases improve bf16 stability.
Both VoxCPMModel and VoxCPM2Model adopt the helper in their __init__,
replacing what would otherwise be duplicated inline checks.
Verified locally on Apple M5 Max, PyTorch 2.11, macOS 15:
- VoxCPM2 (2B): clean output, RTF ~0.78 steady state
- VoxCPM 0.5B: clean output, RTF ~0.92
- No badcase retries fired in any test
- VOXCPM_MPS_DTYPE=bfloat16 round-trips and reproduces the original
glitched output, confirming the override path.
