feat: 新增 training-videos 素材流水线

AI 跟练的素材生产工具,产物写入 TUICallKit-Vue3/training:
- 教练动作帧:AI 设定稿 → 切帧对齐、锁脸、插帧 → 导出小程序帧与 COS 高清帧
- 示范视频:Remotion 渲染四个动作的示范视频
- 背景音乐:程序合成的原创循环音乐(npm run bgm)
This commit is contained in:
2026-10-31 10:33:25 +08:00
parent 75f4b74b3d
commit 3cd3553af2
136 changed files with 5894 additions and 0 deletions
+5
View File
@@ -0,0 +1,5 @@
node_modules
out
__pycache__
work
.venv
+83
View File
@@ -0,0 +1,83 @@
# training-videos
AI 跟练的素材流水线:教练“小甄”动作帧(AI 生成连续动作设定稿 → 切帧对齐 → 导出到小程序)、Remotion 示范视频、程序合成的背景音乐。
小程序侧代码在 `TUICallKit-Vue3/training/`(页面 `pages/pose-hub`、`pages/pose-coach`,逻辑在 `pose/`)。
## 教练动作帧
每一帧都是 AI 直接画的原图,不做合成补帧(光流、骨骼变形合成的中间帧会在四肢边缘留下残影)。
流畅度来自三点:
- **一个动作画在一张设定稿里**:同一次渲染,前后帧的脸、衣服、光照完全一致,播放时不会“闪”
- **帧密**:深蹲 24、侧平举 24、高抬腿每侧 13、开合跳每侧 10(动作过程约每秒 20 帧以上)。先按实测幅度挑掉倒退和重复的帧,
再用 RIFE 在相邻两帧之间插中间帧(有残影的自动跳过),最后按幅度均匀挑出目标帧数
- **按幅度播放**:`coachSequences.ts` 记录每帧实测的动作幅度,播放器按幅度查帧,帧间距不均匀时速度仍然均匀
- **锁脸**(`coach_facelock.py`):每帧的五官都是单独画的,连续播放时脸会“一闪一闪”;切帧后所有帧统一换成第一帧的脸,
并按头部对齐整帧(头不左右晃),相邻帧头部的像素差从 7~12 降到 2~4
左右交替的动作(开合跳、高抬腿)只画左侧,右侧由镜像换边生成:身体水平翻转,头部(脸 + 马尾)保留原图方向,
左右两侧出自同一次渲染,换边时不会跳。
| 步骤 | 命令 | 产物 |
|---|---|---|
| 1. 生成设定稿 | `npm run coach:sheets`(可只生成部分:`-- squat jack`) | `work/sheets/<动作>.png`(透明背景,约 1672×941) |
| 2. 采用 | `npm run coach:adopt -- <动作>` | `assets/sheets/<动作>.webp`(无损,入库) |
| 3. 切帧对齐 | `npm run coach:frames`(可只处理部分:`-- jack`) | `work/frames/master/*.png`、`work/frames/sequences.json`、`work/preview/contact_*.jpg`(检查图) |
| 4. 导出 | `npm run coach:export` | 小程序帧 `TUICallKit-Vue3/training/static/coach/*.png`(600px 轻微锐化,约 1.6MB)、COS 高清帧 `out/coach/*.webp`、视频帧 `public/coach/hd/*.webp`、帧清单 `TUICallKit-Vue3/training/pose/coachSequences.ts` |
- 依赖:本机 `codex` CLI(已登录、开启 `image_generation`);`cwebp`;Python 依赖见 `scripts/requirements.txt`
(`python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt`,之后先 `source .venv/bin/activate` 再跑 npm 命令)
- 模型:关键点用 torchvision Keypoint R-CNN,超分用 Real-ESRGAN x4plus,插帧用 rife-ncnn-vulkan(v4.6),首次运行自动下载
- 包体:训练分包上限 2MB,教练帧约占 1.6MB;加帧或提高导出尺寸前先算预算(`coach_frames.py` 的 `frames`、`coach_export.py` 的 `MINI_SIZE`)
- 角色参考图 `assets/coach-reference.png`;提示词在 `scripts/prompts/<动作>.txt`
- 姿势参考:深蹲、高抬腿用起止关键姿势 `assets/sheets/refs/*_keys.jpg`;侧平举、开合跳用 `scripts/coach_pose_guide.py`
程序画的逐帧人偶(被拍者左侧肢体橙色、右侧蓝色)。只靠文字描述时,模型常把开头几帧画得跳变、左右脚画反
- Codex 只能出约 1.5MP 的图,人物 300~450px 高:切帧时先 4 倍超分再缩到主帧尺寸(站姿身高 720px)
- 设定稿里手臂偶尔碰到隔壁或上一行的人物,切帧会按格子拆开;脚踝以下出现的肤色像素(别人的指尖)会被去掉
- 生成结果有随机性:先看 `work/preview/contact_*.jpg`,不满意就重新生成那一张;`coach_frames.py` 的 `MOVES` 可指定取哪些人物
- 手势帧(idle / wave / cheer / thumbs)和错误示范帧(wrong_*)不随动作帧重新生成,导出时沿用 `public/coach/hd` 里已有的
- 没有 `work/` 时,可用 `python3 scripts/coach_export.py --from-hd` 从高清帧重新导出小程序帧
## 高清帧(COS)
小程序分包上限 2MB,包里的教练帧只能做到 600px;手机上教练显示得大,清晰度靠 COS 上的高清帧:
1. `npm run coach:export` 后,把 `out/coach/` 整个目录(约 92 张 WebP、3.2MB)上传到 COS,公有读,
例如 `https://<bucket>.cos.<region>.myqcloud.com/training/coach/`
2. 把这个目录地址(末尾不带斜杠)填到 `TUICallKit-Vue3/training/pose/exercises.ts` 的 `COACH_CDN_BASE`
- 高清帧是 1000px 画布、只裁出人物所在区域(整张画布解码后 4MB/帧,一个动作几十帧手机吃不消)
- 播放时先用包内帧,高清帧在后台全部加载完再无缝切过去;没配置、网络差或任何一张加载失败时一直用包内帧
- 地址带内容版本号(`coachSequences.ts` 的 `COACH_HD.version`),重新导出后覆盖上传同名文件即可,缓存会自动刷新
- 开发者工具里 `COACH_CDN_BASE` 留空时读本机 `npm run serve`(`/coach/*.webp`)
## 背景音乐
跟练时循环播放的舒缓背景音乐由 `npm run bgm`(`scripts/bgm_synth.py`,依赖 numpy、ffmpeg)用程序合成:
旋律、和声、音色全部是代码生成的原创内容,不含任何采样,没有版权问题。
1. 生成 `out/music/calm.mp3`(约 1 分 47 秒、1.6MB,128kbps),上传到 COS,例如 `.../yundong/music/calm.mp3`,公有读
2. 地址填到 `TUICallKit-Vue3/training/pose/exercises.ts` 的 `BGM_URL`;以后换音乐时换个文件名,避免缓存
- 编曲:D 大调、72 拍/分,铺底长音 + 电钢琴分解和弦 + 颤音琴旋律 + 低音,32 小节首尾相接。
混响、延迟按循环计算,结尾的余音绕回开头;循环点落在音乐最轻的地方,播放器循环时有几十毫秒空隙也听不出来
- 固定随机种子,重复生成结果完全一致;想换风格改脚本里的和弦(`CHORDS` / `PROGRESSION`)、节奏型(`SPARSE` 等)、旋律(`MELODY`)
- `work/music/spectrogram.png` 是频谱图,生成时会打印响度、频谱重心、循环点前后的音量,用来检查段落结构和高低频平衡
- 小程序里的播放规则:倒计时开始淡入,暂停 / 结束淡出,切到后台立即停;教练说口令、纠错时自动压低,报数不压;
右上角 ♪ 可以关掉(记在本地);地址打不开时不显示开关、不影响训练
- 开发者工具里 `BGM_URL` 留空时读本机 `npm run serve`(`/music/calm.mp3`)
## 示范视频
```bash
npm install
npm run studio # 预览
npm run render # 渲染 out/{squat,march,jack,raise}.mp4,约 1.5MB/段
npm run serve # 开发者工具联调:http://127.0.0.1:5179/squat.mp4
```
上线前把 `out/*.mp4` 上传到 COS,填入 `TUICallKit-Vue3/training/pose/exercises.ts` 对应动作的 `videoUrl`;
未填写时开发环境自动使用本机 `npm run serve`,正式环境隐藏“完整示范”按钮。
`coach-showcase` 合成(`npx remotion render src/index.ts coach-showcase out/coach-showcase.mp4`)把四个动作并排循环,用于检查动作是否连贯。
Binary file not shown.

After

Width:  |  Height:  |  Size: 819 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 659 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 942 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 648 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 928 KiB

+3880
View File
File diff suppressed because it is too large Load Diff
+26
View File
@@ -0,0 +1,26 @@
{
"name": "training-videos",
"private": true,
"description": "AI 跟练示范视频与封面(Remotion)",
"scripts": {
"studio": "remotion studio",
"coach:sheets": "zsh scripts/gen-sheets.sh",
"coach:adopt": "python3 scripts/coach_adopt.py",
"coach:frames": "python3 scripts/coach_frames.py",
"coach:export": "python3 scripts/coach_export.py",
"bgm": "python3 scripts/bgm_synth.py",
"render": "node scripts/render-all.mjs",
"serve": "node scripts/serve.mjs",
"render:one": "remotion render"
},
"dependencies": {
"@remotion/cli": "4.0.529",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"remotion": "4.0.529"
},
"devDependencies": {
"@types/react": "^18.3.12",
"typescript": "^5.6.3"
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 53 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Some files were not shown because too many files have changed in this diff Show More