mirror of
https://github.com/OpenBMB/VoxCPM.git
synced 2026-09-20 02:43:33 +08:00
docs: add llama.cpp-omni on-device inference section
This commit is contained in:
Dennis Huang
39
README.md
39
README.md
@@ -71,6 +71,7 @@ Chinese Dialect: 四川话, 粤语, 吴语, 东北话, 河南话, 陕西话, 山
|
||||
- [CLI Usage](#cli-usage)
|
||||
- [Web Demo](#web-demo)
|
||||
- [Production Deployment](#-production-deployment-nano-vllm)
|
||||
- [On-Device Inference (llama.cpp-omni)](#-on-device-inference-llamacpp-omni)
|
||||
- [Models & Versions](#-models--versions)
|
||||
- [Performance](#-performance)
|
||||
- [Fine-tuning](#%EF%B8%8F-fine-tuning)
|
||||
@@ -293,6 +294,43 @@ curl http://localhost:8000/v1/audio/speech \
|
||||
|
||||
> Built on the upstream vLLM scheduler, with batched concurrent requests, streaming chunk delivery, and multi-GPU deployment out of the box. See the [VoxCPM2 example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm2) for full deployment recipes.
|
||||
|
||||
### 📱 On-Device Inference (llama.cpp-omni)
|
||||
|
||||
For on-device / edge deployment without Python, use **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** — a high-performance C++ inference engine built on llama.cpp, with native VoxCPM2 GGUF support on **CPU / Metal / CUDA / Vulkan**.
|
||||
|
||||
**1. Download GGUF weights** from [HuggingFace](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) | [ModelScope](https://modelscope.cn/models/DennisHuang/VoxCPM2-GGUF) — you need one **BaseLM** (F16 or Q8_0) + the **Acoustic** file. Q8_0 halves the download with negligible quality loss.
|
||||
|
||||
**2. Build**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tc-mb/llama.cpp-omni.git && cd llama.cpp-omni
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --target voxcpm2-cli -j
|
||||
```
|
||||
|
||||
> CMake auto-detects Metal (macOS) or CUDA (Linux with NVIDIA GPU).
|
||||
|
||||
**3. Run**
|
||||
|
||||
```bash
|
||||
# Basic TTS
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "Hello, this is VoxCPM2 running through llama.cpp-omni." \
|
||||
-o output.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
|
||||
# Voice cloning (reference audio)
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "Cloned voice." -r speaker.wav -o clone.wav \
|
||||
VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
|
||||
# Ultimate cloning (reference audio + transcript)
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "Target text." --prompt-wav speaker.wav --prompt-text "transcript of speaker.wav" \
|
||||
-o clone.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
```
|
||||
|
||||
> **RTF ~1.76 (Q8_0) on Apple M4 Pro / Metal.** Key flags: `--cfg` (guidance scale), `--timesteps` (CFM steps), `--seed`, `--temperature`, `--stream`. See the [llama.cpp-omni repo](https://github.com/tc-mb/llama.cpp-omni) and [GGUF weights page](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) for full details.
|
||||
|
||||
> **Full parameter reference, multi-scenario examples, and voice cloning tips →** [Quick Start Guide](https://voxcpm.readthedocs.io/en/latest/quickstart.html) | [Usage Guide](https://voxcpm.readthedocs.io/en/latest/usage_guide.html) | [Cookbook](https://voxcpm.readthedocs.io/en/latest/cookbook.html)
|
||||
|
||||
---
|
||||
@@ -566,6 +604,7 @@ Full documentation: **[voxcpm.readthedocs.io](https://voxcpm.readthedocs.io/en/l
|
||||
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| **[Nano-vLLM](https://github.com/a710128/nanovllm-voxcpm)** | High-throughput and Fast GPU serving |
|
||||
| **[vLLM-Omni](https://github.com/vllm-project/vllm-omni)** | Official vLLM omni-modal serving for VoxCPM2 — PagedAttention, OpenAI-compatible API |
|
||||
| **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** | Full-duplex omni inference engine — VoxCPM2 GGUF on CPU / Metal / CUDA / Vulkan |
|
||||
| **[VoxCPM.cpp](https://github.com/bluryar/VoxCPM.cpp)** | GGML/GGUF: CPU, CUDA, Vulkan inference |
|
||||
| **[VoxCPM-ONNX](https://github.com/bluryar/VoxCPM-ONNX)** | ONNX export for CPU inference |
|
||||
| **[VoxCPMANE](https://github.com/0seba/VoxCPMANE)** | Apple Neural Engine backend |
|
||||
|
||||
39
README_zh.md
39
README_zh.md
@@ -73,6 +73,7 @@ VoxCPM 是一个**无离散音频分词器**（Tokenizer-Free）的语音合成
|
||||
- [命令行使用](#命令行使用)
|
||||
- [Web Demo](#web-demo)
|
||||
- [生产部署](#-生产部署nano-vllm)
|
||||
- [端侧推理（llama.cpp-omni）](#-端侧推理llamacpp-omni)
|
||||
- [模型与版本](#-模型与版本)
|
||||
- [性能评测](#-性能评测)
|
||||
- [微调](#%EF%B8%8F-微调)
|
||||
@@ -295,6 +296,43 @@ curl http://localhost:8000/v1/audio/speech \
|
||||
|
||||
> 基于上游 vLLM 调度器构建，开箱即用支持批量并发、流式分块输出和多 GPU 部署。完整示例见 [VoxCPM2 部署样例](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm2)。
|
||||
|
||||
### 📱 端侧推理（llama.cpp-omni）
|
||||
|
||||
如需在端侧/消费级硬件上无 Python 运行，使用 **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** — 基于 llama.cpp 的高性能 C++ 推理引擎，原生支持 VoxCPM2 GGUF，可在 **CPU / Metal / CUDA / Vulkan** 上运行。
|
||||
|
||||
**1. 下载 GGUF 权重**：从 [HF下载](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) | [ModelScope](https://modelscope.cn/models/DennisHuang/VoxCPM2-GGUF)，需要一个 **BaseLM**（F16 或 Q8_0）+ **Acoustic** 文件。Q8_0 体积减半，质量损失可忽略。
|
||||
|
||||
**2. 编译**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tc-mb/llama.cpp-omni.git && cd llama.cpp-omni
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --target voxcpm2-cli -j
|
||||
```
|
||||
|
||||
> CMake 会自动检测并启用 Metal（macOS）或 CUDA（Linux + NVIDIA GPU）。
|
||||
|
||||
**3. 运行**
|
||||
|
||||
```bash
|
||||
# 基础 TTS
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "你好，我是通过 llama.cpp-omni 运行的 VoxCPM2。" \
|
||||
-o output.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
|
||||
# 声音克隆（参考音频）
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "克隆的声音。" -r speaker.wav -o clone.wav \
|
||||
VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
|
||||
# 精准克隆（参考音频 + 转写文本）
|
||||
./build/bin/voxcpm2-cli \
|
||||
-t "目标文本。" --prompt-wav speaker.wav --prompt-text "参考音频的转写文本" \
|
||||
-o clone.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf
|
||||
```
|
||||
|
||||
> **在 Apple M4 Pro / Metal 上 RTF ~1.76（Q8_0）。** 主要参数：`--cfg`（引导尺度）、`--timesteps`（CFM 步数）、`--seed`、`--temperature`、`--stream`。详见 [llama.cpp-omni 仓库](https://github.com/tc-mb/llama.cpp-omni) 和 [GGUF 权重页面](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF)。
|
||||
|
||||
> **完整参数说明、多场景示例与声音克隆技巧 →** [快速开始指南](https://voxcpm.readthedocs.io/zh-cn/latest/quickstart.html) | [使用指南](https://voxcpm.readthedocs.io/zh-cn/latest/usage_guide.html) | [Cookbook](https://voxcpm.readthedocs.io/zh-cn/latest/cookbook.html)
|
||||
|
||||
---
|
||||
@@ -556,6 +594,7 @@ python lora_ft_webui.py # 然后打开 http://localhost:7860
|
||||
|---|---|
|
||||
| [**Nano-vLLM**](https://github.com/a710128/nanovllm-voxcpm) | 高吞吐快速 GPU 推理引擎 |
|
||||
| [**vLLM-Omni**](https://github.com/vllm-project/vllm-omni) | 官方 vLLM 全模态服务（原生支持 VoxCPM2）— PagedAttention、OpenAI 兼容 API |
|
||||
| [**llama.cpp-omni**](https://github.com/tc-mb/llama.cpp-omni) | 全双工全模态推理引擎 — VoxCPM2 GGUF，支持 CPU / Metal / CUDA / Vulkan |
|
||||
| [**VoxCPM.cpp**](https://github.com/bluryar/VoxCPM.cpp) | GGML/GGUF：CPU、CUDA、Vulkan 推理 |
|
||||
| [**VoxCPM-ONNX**](https://github.com/bluryar/VoxCPM-ONNX) | ONNX 导出，支持 CPU 推理 |
|
||||
| [**VoxCPMANE**](https://github.com/0seba/VoxCPMANE) | Apple Neural Engine 后端 |
|
||||
|
||||
Reference in New Issue
Block a user