feat: 新增 training-videos 素材流水线
AI 跟练的素材生产工具,产物写入 TUICallKit-Vue3/training: - 教练动作帧:AI 设定稿 → 切帧对齐、锁脸、插帧 → 导出小程序帧与 COS 高清帧 - 示范视频:Remotion 渲染四个动作的示范视频 - 背景音乐:程序合成的原创循环音乐(npm run bgm)
@@ -0,0 +1,5 @@
|
||||
node_modules
|
||||
out
|
||||
__pycache__
|
||||
work
|
||||
.venv
|
||||
@@ -0,0 +1,83 @@
|
||||
# training-videos
|
||||
|
||||
AI 跟练的素材流水线:教练“小甄”动作帧(AI 生成连续动作设定稿 → 切帧对齐 → 导出到小程序)、Remotion 示范视频、程序合成的背景音乐。
|
||||
|
||||
小程序侧代码在 `TUICallKit-Vue3/training/`(页面 `pages/pose-hub`、`pages/pose-coach`,逻辑在 `pose/`)。
|
||||
|
||||
## 教练动作帧
|
||||
|
||||
每一帧都是 AI 直接画的原图,不做合成补帧(光流、骨骼变形合成的中间帧会在四肢边缘留下残影)。
|
||||
流畅度来自三点:
|
||||
|
||||
- **一个动作画在一张设定稿里**:同一次渲染,前后帧的脸、衣服、光照完全一致,播放时不会“闪”
|
||||
- **帧密**:深蹲 24、侧平举 24、高抬腿每侧 13、开合跳每侧 10(动作过程约每秒 20 帧以上)。先按实测幅度挑掉倒退和重复的帧,
|
||||
再用 RIFE 在相邻两帧之间插中间帧(有残影的自动跳过),最后按幅度均匀挑出目标帧数
|
||||
- **按幅度播放**:`coachSequences.ts` 记录每帧实测的动作幅度,播放器按幅度查帧,帧间距不均匀时速度仍然均匀
|
||||
- **锁脸**(`coach_facelock.py`):每帧的五官都是单独画的,连续播放时脸会“一闪一闪”;切帧后所有帧统一换成第一帧的脸,
|
||||
并按头部对齐整帧(头不左右晃),相邻帧头部的像素差从 7~12 降到 2~4
|
||||
|
||||
左右交替的动作(开合跳、高抬腿)只画左侧,右侧由镜像换边生成:身体水平翻转,头部(脸 + 马尾)保留原图方向,
|
||||
左右两侧出自同一次渲染,换边时不会跳。
|
||||
|
||||
| 步骤 | 命令 | 产物 |
|
||||
|---|---|---|
|
||||
| 1. 生成设定稿 | `npm run coach:sheets`(可只生成部分:`-- squat jack`) | `work/sheets/<动作>.png`(透明背景,约 1672×941) |
|
||||
| 2. 采用 | `npm run coach:adopt -- <动作>` | `assets/sheets/<动作>.webp`(无损,入库) |
|
||||
| 3. 切帧对齐 | `npm run coach:frames`(可只处理部分:`-- jack`) | `work/frames/master/*.png`、`work/frames/sequences.json`、`work/preview/contact_*.jpg`(检查图) |
|
||||
| 4. 导出 | `npm run coach:export` | 小程序帧 `TUICallKit-Vue3/training/static/coach/*.png`(600px 轻微锐化,约 1.6MB)、COS 高清帧 `out/coach/*.webp`、视频帧 `public/coach/hd/*.webp`、帧清单 `TUICallKit-Vue3/training/pose/coachSequences.ts` |
|
||||
|
||||
- 依赖:本机 `codex` CLI(已登录、开启 `image_generation`);`cwebp`;Python 依赖见 `scripts/requirements.txt`
|
||||
(`python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt`,之后先 `source .venv/bin/activate` 再跑 npm 命令)
|
||||
- 模型:关键点用 torchvision Keypoint R-CNN,超分用 Real-ESRGAN x4plus,插帧用 rife-ncnn-vulkan(v4.6),首次运行自动下载
|
||||
- 包体:训练分包上限 2MB,教练帧约占 1.6MB;加帧或提高导出尺寸前先算预算(`coach_frames.py` 的 `frames`、`coach_export.py` 的 `MINI_SIZE`)
|
||||
- 角色参考图 `assets/coach-reference.png`;提示词在 `scripts/prompts/<动作>.txt`
|
||||
- 姿势参考:深蹲、高抬腿用起止关键姿势 `assets/sheets/refs/*_keys.jpg`;侧平举、开合跳用 `scripts/coach_pose_guide.py`
|
||||
程序画的逐帧人偶(被拍者左侧肢体橙色、右侧蓝色)。只靠文字描述时,模型常把开头几帧画得跳变、左右脚画反
|
||||
- Codex 只能出约 1.5MP 的图,人物 300~450px 高:切帧时先 4 倍超分再缩到主帧尺寸(站姿身高 720px)
|
||||
- 设定稿里手臂偶尔碰到隔壁或上一行的人物,切帧会按格子拆开;脚踝以下出现的肤色像素(别人的指尖)会被去掉
|
||||
- 生成结果有随机性:先看 `work/preview/contact_*.jpg`,不满意就重新生成那一张;`coach_frames.py` 的 `MOVES` 可指定取哪些人物
|
||||
- 手势帧(idle / wave / cheer / thumbs)和错误示范帧(wrong_*)不随动作帧重新生成,导出时沿用 `public/coach/hd` 里已有的
|
||||
- 没有 `work/` 时,可用 `python3 scripts/coach_export.py --from-hd` 从高清帧重新导出小程序帧
|
||||
|
||||
## 高清帧(COS)
|
||||
|
||||
小程序分包上限 2MB,包里的教练帧只能做到 600px;手机上教练显示得大,清晰度靠 COS 上的高清帧:
|
||||
|
||||
1. `npm run coach:export` 后,把 `out/coach/` 整个目录(约 92 张 WebP、3.2MB)上传到 COS,公有读,
|
||||
例如 `https://<bucket>.cos.<region>.myqcloud.com/training/coach/`
|
||||
2. 把这个目录地址(末尾不带斜杠)填到 `TUICallKit-Vue3/training/pose/exercises.ts` 的 `COACH_CDN_BASE`
|
||||
|
||||
- 高清帧是 1000px 画布、只裁出人物所在区域(整张画布解码后 4MB/帧,一个动作几十帧手机吃不消)
|
||||
- 播放时先用包内帧,高清帧在后台全部加载完再无缝切过去;没配置、网络差或任何一张加载失败时一直用包内帧
|
||||
- 地址带内容版本号(`coachSequences.ts` 的 `COACH_HD.version`),重新导出后覆盖上传同名文件即可,缓存会自动刷新
|
||||
- 开发者工具里 `COACH_CDN_BASE` 留空时读本机 `npm run serve`(`/coach/*.webp`)
|
||||
|
||||
## 背景音乐
|
||||
|
||||
跟练时循环播放的舒缓背景音乐由 `npm run bgm`(`scripts/bgm_synth.py`,依赖 numpy、ffmpeg)用程序合成:
|
||||
旋律、和声、音色全部是代码生成的原创内容,不含任何采样,没有版权问题。
|
||||
|
||||
1. 生成 `out/music/calm.mp3`(约 1 分 47 秒、1.6MB,128kbps),上传到 COS,例如 `.../yundong/music/calm.mp3`,公有读
|
||||
2. 地址填到 `TUICallKit-Vue3/training/pose/exercises.ts` 的 `BGM_URL`;以后换音乐时换个文件名,避免缓存
|
||||
|
||||
- 编曲:D 大调、72 拍/分,铺底长音 + 电钢琴分解和弦 + 颤音琴旋律 + 低音,32 小节首尾相接。
|
||||
混响、延迟按循环计算,结尾的余音绕回开头;循环点落在音乐最轻的地方,播放器循环时有几十毫秒空隙也听不出来
|
||||
- 固定随机种子,重复生成结果完全一致;想换风格改脚本里的和弦(`CHORDS` / `PROGRESSION`)、节奏型(`SPARSE` 等)、旋律(`MELODY`)
|
||||
- `work/music/spectrogram.png` 是频谱图,生成时会打印响度、频谱重心、循环点前后的音量,用来检查段落结构和高低频平衡
|
||||
- 小程序里的播放规则:倒计时开始淡入,暂停 / 结束淡出,切到后台立即停;教练说口令、纠错时自动压低,报数不压;
|
||||
右上角 ♪ 可以关掉(记在本地);地址打不开时不显示开关、不影响训练
|
||||
- 开发者工具里 `BGM_URL` 留空时读本机 `npm run serve`(`/music/calm.mp3`)
|
||||
|
||||
## 示范视频
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run studio # 预览
|
||||
npm run render # 渲染 out/{squat,march,jack,raise}.mp4,约 1.5MB/段
|
||||
npm run serve # 开发者工具联调:http://127.0.0.1:5179/squat.mp4
|
||||
```
|
||||
|
||||
上线前把 `out/*.mp4` 上传到 COS,填入 `TUICallKit-Vue3/training/pose/exercises.ts` 对应动作的 `videoUrl`;
|
||||
未填写时开发环境自动使用本机 `npm run serve`,正式环境隐藏“完整示范”按钮。
|
||||
|
||||
`coach-showcase` 合成(`npx remotion render src/index.ts coach-showcase out/coach-showcase.mp4`)把四个动作并排循环,用于检查动作是否连贯。
|
||||
|
After Width: | Height: | Size: 819 KiB |
|
After Width: | Height: | Size: 659 KiB |
|
After Width: | Height: | Size: 942 KiB |
|
After Width: | Height: | Size: 648 KiB |
|
After Width: | Height: | Size: 60 KiB |
|
After Width: | Height: | Size: 69 KiB |
|
After Width: | Height: | Size: 928 KiB |
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "training-videos",
|
||||
"private": true,
|
||||
"description": "AI 跟练示范视频与封面(Remotion)",
|
||||
"scripts": {
|
||||
"studio": "remotion studio",
|
||||
"coach:sheets": "zsh scripts/gen-sheets.sh",
|
||||
"coach:adopt": "python3 scripts/coach_adopt.py",
|
||||
"coach:frames": "python3 scripts/coach_frames.py",
|
||||
"coach:export": "python3 scripts/coach_export.py",
|
||||
"bgm": "python3 scripts/bgm_synth.py",
|
||||
"render": "node scripts/render-all.mjs",
|
||||
"serve": "node scripts/serve.mjs",
|
||||
"render:one": "remotion render"
|
||||
},
|
||||
"dependencies": {
|
||||
"@remotion/cli": "4.0.529",
|
||||
"react": "^18.3.1",
|
||||
"react-dom": "^18.3.1",
|
||||
"remotion": "4.0.529"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/react": "^18.3.12",
|
||||
"typescript": "^5.6.3"
|
||||
}
|
||||
}
|
||||
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 46 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 47 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 53 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 53 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 40 KiB |
|
After Width: | Height: | Size: 27 KiB |
|
After Width: | Height: | Size: 39 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 39 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 37 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 41 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 40 KiB |
|
After Width: | Height: | Size: 27 KiB |
|
After Width: | Height: | Size: 39 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 37 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 53 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 52 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 47 KiB |
|
After Width: | Height: | Size: 31 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 48 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 31 KiB |
|
After Width: | Height: | Size: 31 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 44 KiB |
|
After Width: | Height: | Size: 31 KiB |
|
After Width: | Height: | Size: 29 KiB |
|
After Width: | Height: | Size: 40 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 36 KiB |
|
After Width: | Height: | Size: 25 KiB |
|
After Width: | Height: | Size: 34 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 35 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 23 KiB |
|
After Width: | Height: | Size: 32 KiB |
|
After Width: | Height: | Size: 22 KiB |