diff --git a/README.md b/README.md index bdd3c62..707980c 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,29 @@ - 固定数据目录、一键启动、升级备份与真人实拍、仙侠风格预设。 - 跨集结尾上下文、分镜起止状态、前镜头尾帧接续与持久化依赖队列。 - 中英文视频提示词同步保存，MiniMax H3 提交前使用英文制作描述，保留中文对白与素材引用。 +- 场景固定设定与场景参考图自动随每个分镜请求提交，覆盖单次、批量、换卡重做和高清制作。 +- LTX-2.3 / Wan2.2 先用全部绑定的人物、场景、道具图合成完整分镜首帧，再将单张首帧交给视频模型；超过 6 个素材分轮融合，不静默丢图。 +- 豆包英文补译关闭深度思考，拆分镜保存时最多同时补译 3 条，全部通过对白与素材标记校验后才保存。 + +## 场景一致性 + +视频制作的「场景固定设定」位于镜头衔接上方。同一场景 ID 的分镜（包括跨集复用）共用空间、固定陈设、时间和光源；没有单独固定设定时直接采用已有场景描述。编辑保存用于后续生成，不改写原分镜或旧视频。灯光开关、移动道具等剧情变化应记录在镜头起止状态中，换机位保持物理空间关系而非屏幕左右不变。 + +后端从绑定场景读取环境，单独加入每个视频/分镜图片请求，避免只写人物表演而遗漏背景。普通参考模式补带场景图并保留现有图片编号；尾帧接续和 VDN 高清沿用视频参考帧，同时保留环境文字。MiniMax H3 的环境描述翻译为英文，台词保持原文。缺失/失效的场景、超出模型参考图数量会在换卡取消旧任务之前报错；任务记录保存当时场景设定快照。 + +场景图应与文字设定一致。文字与参考图可减少漂移，但不保证生成画面逐像素一致；连续动作可使用已有「尾帧接续」。角色三视图、道具白底图不加入场景约束。没有场景图时页面明确提示仅有文字约束。 + +回归验证：`integration-work/test_scene_continuity.py` 使用独立 SQLite 和本机模拟模型服务，覆盖实际请求体、参考编号、换卡、跨集、转场、尾帧、高清、图片与数据持久化；设置 `SCENE_UI_TEST=1` 可同时验证页面保存和刷新。需先构建桌面后端及前端，并具备现有 Electron 测试运行环境。 + +## LTX / Wan 分镜首帧 + +视频制作选择 LTX-2.3、LTX-2.3-Text-Preserve 或 Wan2.2-I2V 后，点击生成会先使用顶部选定的 Qwen Image 2.1 图片配置制作完整画面。绑定的场景、人物、道具全部参与：6 个以内一次提交；更多素材以「上一轮合成图 + 最多 5 个新增素材」继续融合，保留每轮结果与素材清单。缺少参考图片会明确报错，需先补图；不再只选第一张场景图。 + +图片显卡跟随顶部图片配置，视频显卡跟随当前镜头设置。首帧准备完成后才提交视频；开启尾帧接续时，前镜头末帧也加入融合。右侧「分镜首帧」显示最终画面、素材清单和图片模型；输入未变时重试视频复用首帧，也可点击「重做首帧并生成视频」。修改分镜或绑定素材后重新合成，生成中修改会阻止旧首帧提交视频。 + +准备首帧期间可取消父视频任务，同时取消其当前图片任务；重启继续跟踪已提交图片，不重复生成。首帧失败不会提交视频。中间图与最终图进入任务历史，不覆盖人物原图、场景原图或旧视频。全参考图流程减少身份与环境漂移，生成结果仍需检查；跨多个融合轮次可能出现误差。 + +回归脚本：`integration-work/test_composed_first_frame.py`，设置 `COMPOSE_UI_TEST=1` 同时检查真实界面。使用隔离数据库和本机模拟服务，覆盖 15 张素材、GPU 路由、尾帧衔接、失败、取消、重启与避免重复提交，不消耗真实模型算力。 ## 英文视频提示词 @@ -26,7 +49,7 @@ MiniMax-H3 系列（含加速版、VDN 高清）提交前自动补齐或更新 - **独立镜头 / 转场**：不依赖前段，适合换场景、时间跳跃或开场。 - **状态承接**：将前镜头结束状态与本镜头起止状态加入生成约束；允许换角度，仍使用本镜头的资产参考图。 -- **尾帧接续**：同场景连续动作。等待前段完成，从所选视频结束前 0.05–3 秒提取画面作为单张首帧，再向选定 GPU 提交。支持当前本地 MiniMax-H3、MiniMax-H3-Hybrid、Wan2.2-I2V；不与多张参考图或参考音色混用。 +- **尾帧接续**：同场景连续动作。等待前段完成，从所选视频结束前 0.05–3 秒提取画面，再向选定 GPU 提交。当前本地 MiniMax-H3、MiniMax-H3-Hybrid 直接使用单张尾帧；LTX-2.3 和 Wan2.2-I2V 将尾帧与全部绑定素材融合后使用新首帧。不与参考音色混用。 来源默认为按集号、镜号排序的前一镜，也可选择本项目更早的镜头。不能引用未来、自身、其他项目或已删除的镜头。第一镜、转场需用独立模式。拆分镜工具会读取上一集结尾剧本及最后三个分镜；起止状态是文字设定，不代表系统已经识别或审核了成片画面。 @@ -36,12 +59,28 @@ MiniMax-H3 系列（含加速版、VDN 高清）提交前自动补齐或更新 ## 视频进度与预计耗时 +分镜拆分按每批最多 4 个段落校验中文、英文、时间轴和素材绑定，全部通过后一次性替换旧分镜。取消、超时、重启或校验失败均保留旧分镜；拆分期间修改剧本或原分镜，会阻止旧结果覆盖新编辑。 + +没有上传音色不影响拆分：生成草稿中误加的角色音色引用会改回普通人物引用；真实音色和对白原文保留。素材引用中的多余类别前缀会统一为实际资产名称。对白提到“血红文字”等内容不再误判为画面画字；实际要求画字仍会退回修正。连续三轮没有保存进展会停止并显示具体错误，诊断信息保存在本地任务状态中。 + +拆分任务按原文场次逐场运行，为每段原文分配来源编号，当前场次覆盖完成后才进入下一场。每场核对来源编号、按原文顺序保留的对白、最少镜头数和时长；仅出现结尾台词不能判定整集完成。预计镜头数量是节奏参考，不是固定上限；不能为了达到数量添加剧情。 + +长台词按原文切成小段，并给出约 4.5 汉字/秒的发声时长参考。校验具体对白时间段，不允许用几秒承载几十字台词；小幅时间分配错误会同步修正中英文时间轴，超过单镜头 15 秒容量时必须继续细分。全部场次通过校验才替换原分镜。来源编号和对白校验不等于人工审片，表演、调度及动作覆盖仍需检查。旧版本过度压缩的结果会提示重新细分，不自动改写已有镜头或视频。 + 视频预览下方、任务列表和高清制作页会显示排队、模型加载、采样、解码与编码阶段，以及已提交时长。采样步数来自对应 ComfyUI 任务的实时消息；积累足够的步速记录后显示当前采样阶段的剩余时间区间，后续解码、编码和保存另计。 整段视频耗时只参考同显卡、同工作流及规格的成功记录，不包含排队与下载；样本不足时不报估算。采样完成不代表成片完成，连接中断或数据过期会隐藏旧估算。进度读取不会取消、重新提交或改换显卡。 服务器的进度扩展源文件为 `integration-work/comfy_progress.py`，由 `install_comfy_progress.py` 备份后嵌入现有适配器；无需重启 ComfyUI。图片、视频及私有配置仍不进入 Git。 +## 下载后自动清理 ComfyUI 视频 + +本地视频任务（包括高清版）成功下载后，先完整解码校验、写入磁盘并保存到项目，才发送带文件长度和 SHA-256 的清理确认。服务器再次核对实际交付的视频内容，从该任务的历史记录中定位唯一输出，仅删除对应视频文件和该条 ComfyUI 历史记录。火宝中的本地成片、海报、历史版本、人物和场景参考图保持可用。 + +清理失败单独重试，不把已完成的视频改成失败；重启后继续确认，任务列表显示等待清理或已清理。未收到本地下载确认、正在运行、被其他任务引用、文件校验不符或路径不明确的任务不删除；既有已完成任务不批量追溯清理。 + +服务端运行 `comfy_video_cleanup.py` 用户服务，只消费两个私有桥接器保存的确认记录，没有新增网络监听，不重启 ComfyUI。先记录精确文件标识再删除，异常退出后可继续完成同一条记录。`test_video_cleanup_delivery.py`、`test_video_cleanup_api.py` 和 Linux 隔离测试 `test_comfy_video_cleanup.py` 分别验证本地下载、确认接口及删除范围。 + ## 取消任务 打开右上角「任务」，在对应图片或视频记录上点击「取消任务」。只停止这一条任务，不影响其他镜头、其他显卡任务或已有素材。等待前镜头的任务也可以直接取消。 diff --git a/huobao-source/backend/src/agents/index.ts b/huobao-source/backend/src/agents/index.ts index 5c7aeeb..df9246f 100644 --- a/huobao-source/backend/src/agents/index.ts +++ b/huobao-source/backend/src/agents/index.ts @@ -19,6 +19,7 @@ import { loadAgentPromptFile, loadBasePromptFile } from './prompts.js' import { buildLanguageDirective } from './language.js' import { getContentLanguageFromRC } from './context.js' import { getVoiceReferenceInstructions } from './voice-reference-instructions.js' +import { sceneContinuityInstructions } from '../services/scene-continuity.js' // Default prompts (used when workspace/prompts/.md 文件缺失时兜底) export const DEFAULT_PROMPTS: Record = { @@ -372,7 +373,14 @@ function buildInstructions(type: string) { const bilingualInstructions = ['storyboard_breaker', 'prompt_generator'].includes(type) ? ` 双语视频提示词（优先于旧的单字段保存规则）：每个分镜同步产出 video_prompt（中文供编辑）和 video_prompt_en（完整英文供视频模型）。英文版逐项忠实翻译画面、人物动作、运镜、光线、声音与否定约束，时间段、顺序和时长与中文完全一致；不得增减剧情。所有引号内对白、旁白逐字保留中文原文，不翻译、不音译；对白原文前紧接 says: 或 Dialogue:（不要写 says in Mandarin Chinese:，以兼容语音解析），表演说明写在该标记之前。@角色名/@场景名/@道具名/#音色名原样保留，不改名，不加新对白或字幕。英文仅用于制作描述，中文台词仍使用中文发声。不要把说明朗读为台词。 save_storyboards 每条同时提供两种提示词。单独生成/更新视频提示词时 update_storyboard 只传 storyboard_id、video_prompt、video_prompt_en 三个字段，其他字段不动。` : '' - return [baseInstructions, skillInstructions, voiceInstructions, languageDirective, continuityInstructions, bilingualInstructions] + const storyboardJobInstructions = type === 'storyboard_breaker' && requestContext?.get('storyboardJobId' as never) ? ` +本次使用带进度的整集拆分任务，以下保存规则覆盖旧提示词中的批量保存说明：每批最多 4 个分镜，顺序调用 save_storyboards，禁止并行保存。只有第一批带 replace_existing=true，后续省略。所有批次先暂存校验，全部完成后系统统一替换原分镜；不要调用 update_storyboard 修改原分镜。读取上下文后直接生成第一批，不要输出规划文字；继续到剧本最后一句，不要把“已暂存”误当成整集已完成。 +原文覆盖与合理时长同时满足才完成：系统按场次单独调用模型，每轮只拆 readStoryboardContext 返回的 script/source_units，禁止跳到下一场或整集结尾。每个分镜填写 source_ids。必须覆盖当前场次全部原文段落、全部对白，达到 current_section.min_seconds 和 min_shots；达到结尾台词本身不代表完成。时长不足应细分原有动作和对白，不能新增剧情或续写结尾。 +对白按约 4.5 汉字/秒安排到具体时间段，镜头 8–15 秒。系统已将长对白切成保留原文的小段，source_units 的 speech_seconds 是这小段至少需要的发声秒数；一个镜头优先只放一小段对白，不要把多段长台词重新合并。给对白时间段至少留出 speech_seconds，剩余时间安排动作。不能用 3–4 秒承载几十字长台词，不能只延长总 duration 而不调整对白所在时间段。一个时间段可以长于 3 秒；保证时间轴连续即可。原文必须逐字保留，不能增删台词。返回 section_complete 后立即结束当前轮，下一场由系统安排，不得继续调用工具。 +对白或来源未通过时先查看 coverage/current_section_coverage。repair_shot_numbers 列出的已有镜头必须原位修改并重存，不能把同一句话反复追加为新镜头。对白用“@角色名 + 表演动作 + 说：「原文」”的明确格式；怒吼、惨叫等表演不应改变对白原文。上一批没有减少缺失内容时立即修正，不要靠增加镜头数量假装完成。 +素材标记直接使用 @ 加素材列表中的完整 name/location，不额外加“场景”“角色”等类别前缀。英文中所有引号内原文（含对白、拟声、称谓）与素材标记完整保留；不得漏译画面说明。` : '' + return [baseInstructions, skillInstructions, voiceInstructions, languageDirective, continuityInstructions, bilingualInstructions, storyboardJobInstructions, + ['storyboard_breaker', 'prompt_generator'].includes(type) ? sceneContinuityInstructions : ''] .filter(Boolean) .join('\n\n') } diff --git a/huobao-source/backend/src/agents/tools/storyboard-tools.ts b/huobao-source/backend/src/agents/tools/storyboard-tools.ts index cc50d5f..89f6de0 100644 --- a/huobao-source/backend/src/agents/tools/storyboard-tools.ts +++ b/huobao-source/backend/src/agents/tools/storyboard-tools.ts @@ -12,9 +12,12 @@ import { logTaskProgress, logTaskSuccess } from '../../utils/task-logger.js' import { getDramaId, getEpisodeId } from '../context.js' import { previousEpisodeContext } from '../../services/continuity.js' import { resolveVoiceMentions } from '../../services/character-voice-references.js' +import { normalizeDraftVoices, normalizeDraftAssets } from '../../utils/storyboard-generated-prompt.js' +import { sectionBatchProblems, sectionCoverage, fitDraftDialogueTiming } from '../../utils/storyboard-coverage.js' import { isScriptStale, targetEpisodeSeconds } from '../../utils/script-source.js' import { storyboardSegmentProblems } from '../../utils/storyboard-checks.js' -import { bilingualFields, englishPromptStatus } from '../../services/english-video-prompt.js' +import { bilingualFields, bilingualBatchFields, englishPromptStatus, promptAssetNames } from '../../services/english-video-prompt.js' +import { assertStoryboardJob, assertStoryboardSourceUnchanged, getStoryboardJob, rejectStoryboardBatch, stageStoryboardBatch, updateStoryboardJob } from '../../services/storyboard-job-state.js' async function validateVideoPromptVoices(prompt: string | undefined, dramaId: number, characterIds: number[] = []) { if (!prompt?.includes('#')) return @@ -73,7 +76,7 @@ async function getEpisodePropIds(episodeId: number) { return new Set(links.map(link => link.propId)) } -async function validateStoryboardBindings(episodeId: number, dramaId: number, sceneId: number | null | undefined, characterIds: number[] | undefined, propIds?: number[] | undefined) { +async function validateStoryboardBindings(episodeId: number, dramaId: number, sceneId: number | null | undefined, characterIds: number[] | undefined, propIds?: number[] | undefined, dryRun = false) { const episodeSceneIds = await getEpisodeSceneIds(episodeId) const episodeCharacterIds = await getEpisodeCharacterIds(episodeId) const episodePropIds = await getEpisodePropIds(episodeId) @@ -84,7 +87,7 @@ async function validateStoryboardBindings(episodeId: number, dramaId: number, sc if (!scene || scene.dramaId !== dramaId || scene.deletedAt) { throw new Error(`scene_id ${sceneId} 不属于当前项目`) } - await db.insert(schema.episodeScenes).values({ episodeId, sceneId, createdAt: now() }) + if (!dryRun) await db.insert(schema.episodeScenes).values({ episodeId, sceneId, createdAt: now() }) } const uniqueCharacterIds = [...new Set((characterIds || []).filter(Boolean))] @@ -94,7 +97,7 @@ async function validateStoryboardBindings(episodeId: number, dramaId: number, sc if (!character || character.dramaId !== dramaId || character.deletedAt) { throw new Error(`character_id ${characterId} 不属于当前项目`) } - await db.insert(schema.episodeCharacters).values({ episodeId, characterId, createdAt: now() }) + if (!dryRun) await db.insert(schema.episodeCharacters).values({ episodeId, characterId, createdAt: now() }) } const uniquePropIds = [...new Set((propIds || []).filter(Boolean))] @@ -104,11 +107,12 @@ async function validateStoryboardBindings(episodeId: number, dramaId: number, sc if (!prop || prop.dramaId !== dramaId || prop.deletedAt) { throw new Error(`prop_id ${propId} 不属于当前项目`) } - await db.insert(schema.episodeProps).values({ episodeId, propId, createdAt: now() }) + if (!dryRun) await db.insert(schema.episodeProps).values({ episodeId, propId, createdAt: now() }) } } type ToolContext = ToolExecutionContext | undefined +const storyboardJobId = (context: ToolContext) => context?.requestContext?.get('storyboardJobId' as never) as string|undefined function requireIds(context: ToolContext): { episodeId: number; dramaId: number } | { error: string } { const episodeId = getEpisodeId(context?.requestContext) @@ -125,6 +129,8 @@ const readStoryboardContext = createTool({ const ids = requireIds(context) if ('error' in ids) return ids const { episodeId, dramaId } = ids + const jobId=storyboardJobId(context) + if (jobId) assertStoryboardJob(episodeId,jobId) const [ep] = await db.select().from(schema.episodes) .where(eq(schema.episodes.id, episodeId)) if (!ep) return { error: 'Episode not found' } @@ -134,6 +140,8 @@ const readStoryboardContext = createTool({ } const script = ep.scriptContent || ep.content if (!script) return { error: 'Episode has no script' } + const splitJob=jobId?assertStoryboardJob(episodeId,jobId):null + const sectionIndex=splitJob?.sectionIndex||0,section=splitJob?.sections?.[sectionIndex] const charLinks = await db.select().from(schema.episodeCharacters) .where(eq(schema.episodeCharacters.episodeId, episodeId)) @@ -153,7 +161,7 @@ const readStoryboardContext = createTool({ const prps = await db.select().from(schema.props) .where(eq(schema.props.dramaId, dramaId)) const existingStoryboards = await db.select().from(schema.storyboards) - .where(eq(schema.storyboards.episodeId, episodeId)) + .where(and(eq(schema.storyboards.episodeId, episodeId), isNull(schema.storyboards.deletedAt))) const characters = chars .filter(c => !c.deletedAt) @@ -180,6 +188,7 @@ const readStoryboardContext = createTool({ time: s.time, prompt: s.prompt || '', lighting: s.lighting || '', + environment_lock: s.continuityPrompt || s.prompt || '', image_url: s.imageUrl || '', storyboard_count: s.storyboardCount || 0, })) @@ -230,16 +239,24 @@ const readStoryboardContext = createTool({ episode_number: ep.episodeNumber, description: ep.description || '', }, - script, + script:section?.script||script, + source_units:section?.units, + current_section:section?{id:section.id,index:sectionIndex+1,total:splitJob?.sectionCount,min_seconds:section.minSeconds,min_shots:section.minShots,target_seconds:section.targetSeconds}:undefined, + next_shot_number:splitJob?(splitJob.drafts.at(-1)?.shot_number||0)+1:1, + current_section_drafts:splitJob?.drafts.filter(s=>s.source_section===sectionIndex), + current_section_coverage:section&&splitJob?sectionCoverage(section,splitJob.drafts.filter(s=>s.source_section===sectionIndex)):undefined, + previous_section_end:splitJob?.drafts.filter(s=>s.source_section({description:s.description,continuity_end:s.continuity_end})), + completion_rule: '只拆当前场次，覆盖 source_units 的全部段落与对白；每个分镜填写 source_ids。达到本场 min_seconds/min_shots 且对白全部出现才完成，禁止跳场或续写剧情。', // 总量锚定的确定性数值（剧本去空白字数 ÷ 500 字/分钟），save_storyboards 会按它回报进度 - target_total_seconds: targetEpisodeSeconds(script), - target_segment_count: Math.max(1, Math.round(targetEpisodeSeconds(script) / 12)), + target_total_seconds: splitJob?.targetSeconds||targetEpisodeSeconds(script), + target_segment_count: splitJob?.target||Math.max(1, Math.round(targetEpisodeSeconds(script) / 12)), characters, scenes, props, previous_episode: await previousEpisodeContext(episodeId), - existing_storyboards: existingStoryboardPayload, + existing_storyboards: jobId ? [] : existingStoryboardPayload, } + if (jobId) updateStoryboardJob(episodeId,jobId,{phase:`正在拆分第 ${sectionIndex+1}/${splitJob?.sectionCount||1} 个场次（每批最多 4 个）`}) logTaskSuccess('StoryboardTool', 'read-context', { episodeId, dramaId, @@ -255,6 +272,7 @@ const readStoryboardContext = createTool({ const storyboardFields = z.object({ shot_number: z.number(), + source_ids:z.array(z.string()).optional().describe('Exact source_units IDs covered by this shot in the current section. A long dialogue may span several shots with the same source ID. Do not claim IDs from another section.'), continuity_mode: z.enum(['none', 'context', 'first_frame']).optional().describe('none: independent/new scene; context: inherit state across a cut; first_frame: same-scene continuous action using previous video tail; automatic predecessor is previous chronological shot'), continuity_start: z.string().max(1200).optional().describe('Opening state: positions, facing, costumes, injuries, held objects/hands, lighting, ongoing action; unknowns explicitly marked'), continuity_end: z.string().max(1200).optional().describe('Planned ending state for the next shot; not an observation of generated video'), @@ -280,7 +298,7 @@ const storyboardFields = z.object({ const saveStoryboards = createTool({ id: 'save_storyboards', - description: 'Save storyboards for this episode. Call in batches of at most 8 storyboards: the first batch must set replace_existing: true (clears all old storyboards for the episode, then writes), every following batch omits replace_existing (appends). Rows are upserted by shot_number, so overlapping batches and retries never create duplicates.', + description: 'Save storyboards for this episode. Call in batches of at most 4 storyboards: the first batch sets replace_existing: true, every following batch omits replace_existing and appends sequential shot_numbers. During a tracked job, batches are staged and only replace existing storyboards when the entire screenplay is complete. Validation errors must be corrected on the current batch before proceeding.', inputSchema: z.object({ replace_existing: z.boolean().optional(), storyboards: z.array(storyboardFields), @@ -289,6 +307,34 @@ const saveStoryboards = createTool({ const ids = requireIds(context) if ('error' in ids) return ids const { episodeId, dramaId } = ids + const jobId=storyboardJobId(context) + const committing=context?.requestContext?.get('storyboardCommit' as never)===true + if (jobId) { + const job=assertStoryboardJob(episodeId,jobId) + const requestedSection=context?.requestContext?.get('storyboardSectionIndex' as never) + if(!committing && requestedSection!==undefined && requestedSection!==job.sectionIndex) return {section_complete:true,message:'本轮场次已完成，请立即结束，由系统启动下一场。'} + if(!committing && job.sectionIndex===job.sectionCount) return {script_complete:true,message:'所有原文场次已完成，不能追加后续剧情。'} + storyboards=[...storyboards].sort((a,b)=>a.shot_number-b.shot_number) + if (!committing && storyboards.length>4) { + const error='每批最多保存 4 个分镜，请拆小批次以减少等待，保留全部剧情。' + rejectStoryboardBatch(episodeId,jobId,error);return {error} + } + updateStoryboardJob(episodeId,jobId,{phase:committing?'正在保存完整分镜':'正在校验中文分镜和英文提示词'}) + } + try { + if(jobId&&!committing) { + storyboards=storyboards.map(fitDraftDialogueTiming) + const job=assertStoryboardJob(episodeId,jobId),section=job.sections?.[job.sectionIndex||0] + if(!section)throw new Error('当前场次已完成，请结束本轮') + const problems=sectionBatchProblems(section,storyboards) + const previous=job.drafts.filter(s=>s.source_section<(job.sectionIndex||0)).at(-1)?.shot_number||0 + if(storyboards.some(s=>s.shot_number<=previous))problems.push('不能覆盖已完成场次的分镜，请从当前场次继续编号') + const numbers=[...new Set([...job.drafts.map(s=>s.shot_number),...storyboards.map(s=>s.shot_number)])].sort((a,b)=>a-b) + const gap=numbers.findIndex((n,i)=>n!==i+1) + if(gap>=0)problems.push(`分镜编号缺少 ${gap+1}。上批未保存的分镜不能跳过，请从 ${gap+1} 开始重新提交，source_ids 仍使用当前场次原文编号。`) + if(new Set(storyboards.map(s=>s.shot_number)).size!==storyboards.length)problems.push('本批 shot_number 不能重复') + if(problems.length)throw new Error(problems.join('\n')) + } const ts = now() logTaskProgress('StoryboardTool', 'save-begin', { episodeId, @@ -297,7 +343,15 @@ const saveStoryboards = createTool({ count: storyboards.length, shotNumbers: storyboards.map(sb => sb.shot_number).join(','), }) - // Check all voice references before replace_existing can remove the previous storyboards. + // Missing voice uploads must not prevent writing new storyboard drafts. + // Normalize model-invented #markers on both languages, never actual video submissions. + const voiceCharacters = (await db.select().from(schema.characters).where(eq(schema.characters.dramaId, dramaId))).filter(c=>!c.deletedAt) + const assetNames=await promptAssetNames(dramaId) + storyboards = storyboards.map(sb=>({...sb, + video_prompt: normalizeDraftAssets(normalizeDraftVoices(sb.video_prompt, voiceCharacters, sb.character_ids || []),assetNames), + video_prompt_en: normalizeDraftAssets(normalizeDraftVoices(sb.video_prompt_en, voiceCharacters, sb.character_ids || []),assetNames), + })) + // Check all remaining voice references before any old storyboard is touched. for (const sb of storyboards) { await validateVideoPromptVoices(sb.video_prompt, dramaId, sb.character_ids) } @@ -308,73 +362,62 @@ const saveStoryboards = createTool({ storyboardSegmentProblems(sb, dramaCharacters).map(p => `段落 ${sb.shot_number}：${p}`)) if (segmentProblems.length) { logTaskProgress('StoryboardTool', 'save-rejected', { episodeId, problems: segmentProblems.length }) + if (jobId) rejectStoryboardBatch(episodeId,jobId,segmentProblems.join('\n')) return { error: `本批未保存，请逐条修正后用同样的 shot_number 重新调用 save_storyboards：\n${segmentProblems.join('\n')}` } } - const english = new Map>>() - for (const sb of storyboards) { - if (sb.video_prompt?.trim()) english.set(sb.shot_number, await bilingualFields(sb.video_prompt, sb.video_prompt_en, dramaId, - { configId: context?.requestContext?.get('textConfigId' as never) as number | undefined })) + for (const sb of storyboards) await validateStoryboardBindings(episodeId,dramaId,sb.scene_id,sb.character_ids,sb.prop_ids,true) + const english = await bilingualBatchFields(storyboards, dramaId, { + configId: context?.requestContext?.get('textConfigId' as never) as number | undefined, + model: context?.requestContext?.get('modelOverride' as never) as string | undefined, + }) + if (jobId) { + const job=assertStoryboardJob(episodeId,jobId) + if (!committing) return stageStoryboardBatch(episodeId,jobId,storyboards.map(sb=>({...sb,video_prompt_en:english.get(sb.shot_number)?.videoPromptEn||sb.video_prompt_en})),replace_existing===true) + if(job.sections?.some((section,index)=>!sectionCoverage(section,job.drafts.filter(s=>s.source_section===index)).ready))throw new Error('仍有场次原文、对白或时长未完整覆盖，未替换原分镜') + assertStoryboardSourceUnchanged(job) } - if (replace_existing === true) { - const existingStoryboardRows = await db.select().from(schema.storyboards) - .where(eq(schema.storyboards.episodeId, episodeId)) - for (const storyboardId of existingStoryboardRows.map(sb => sb.id)) { - await db.delete(schema.storyboardCharacters) - .where(eq(schema.storyboardCharacters.storyboardId, storyboardId)) - await db.delete(schema.storyboardProps) - .where(eq(schema.storyboardProps.storyboardId, storyboardId)) + // Validate the entire batch before touching existing rows; the final swap is atomic. + db.transaction(tx => { + if (jobId) assertStoryboardSourceUnchanged(assertStoryboardJob(episodeId,jobId)) + if (replace_existing === true) { + const previous=tx.select().from(schema.storyboards).where(and(eq(schema.storyboards.episodeId,episodeId),isNull(schema.storyboards.deletedAt))).all() + for (const row of previous) { + tx.delete(schema.storyboardCharacters).where(eq(schema.storyboardCharacters.storyboardId,row.id)).run() + tx.delete(schema.storyboardProps).where(eq(schema.storyboardProps.storyboardId,row.id)).run() + } + tx.delete(schema.storyboards).where(and(eq(schema.storyboards.episodeId,episodeId),isNull(schema.storyboards.deletedAt))).run() } - await db.delete(schema.storyboards).where(eq(schema.storyboards.episodeId, episodeId)) - } - - // shot_number → id 索引（含本次调用内新增的行），保证按分镜号幂等 upsert - const existingRows = await db.select().from(schema.storyboards) - .where(eq(schema.storyboards.episodeId, episodeId)) - const shotToId = new Map( - existingRows.filter(sb => !sb.deletedAt).map(sb => [sb.storyboardNumber, sb.id]), - ) - - for (const sb of storyboards) { - await validateStoryboardBindings(episodeId, dramaId, sb.scene_id, sb.character_ids, sb.prop_ids) - const existingId = shotToId.get(sb.shot_number) - if (existingId !== undefined) { - await db.update(schema.storyboards).set({ - title: sb.title, shotType: sb.shot_type, - angle: sb.angle, movement: sb.movement, - location: sb.location, time: sb.time, - description: sb.description, result: sb.result, - atmosphere: sb.atmosphere, imagePrompt: sb.image_prompt, - videoPrompt: sb.video_prompt, bgmPrompt: sb.bgm_prompt, - ...english.get(sb.shot_number), - soundEffect: sb.sound_effect, - continuityMode: sb.continuity_mode, continuityStart: sb.continuity_start, continuityEnd: sb.continuity_end, - sceneId: sb.scene_id, duration: sb.duration || 10, - updatedAt: ts, - }).where(eq(schema.storyboards.id, existingId)) - await syncStoryboardCharacters(existingId, sb.character_ids || []) - await syncStoryboardProps(existingId, sb.prop_ids || []) - } else { - const res = await db.insert(schema.storyboards).values({ - episodeId, - storyboardNumber: sb.shot_number, - title: sb.title, shotType: sb.shot_type, - angle: sb.angle, movement: sb.movement, - location: sb.location, time: sb.time, - description: sb.description, result: sb.result, - atmosphere: sb.atmosphere, imagePrompt: sb.image_prompt, - videoPrompt: sb.video_prompt, bgmPrompt: sb.bgm_prompt, - ...english.get(sb.shot_number), - soundEffect: sb.sound_effect, - continuityMode: sb.continuity_mode, continuityStart: sb.continuity_start, continuityEnd: sb.continuity_end, - sceneId: sb.scene_id, duration: sb.duration || 10, - createdAt: ts, updatedAt: ts, - }) - const newId = getInsertId(res) - shotToId.set(sb.shot_number, newId) - await syncStoryboardCharacters(newId, sb.character_ids || []) - await syncStoryboardProps(newId, sb.prop_ids || []) + const rows=tx.select().from(schema.storyboards).where(eq(schema.storyboards.episodeId,episodeId)).all() + const shotToId=new Map(rows.filter(s=>!s.deletedAt).map(s=>[s.storyboardNumber,s.id])) + for (const sb of storyboards) { + if (sb.scene_id && !tx.select().from(schema.episodeScenes).where(and(eq(schema.episodeScenes.episodeId,episodeId),eq(schema.episodeScenes.sceneId,sb.scene_id))).get()) + tx.insert(schema.episodeScenes).values({episodeId,sceneId:sb.scene_id,createdAt:ts}).run() + for (const characterId of new Set(sb.character_ids||[])) { + if (!tx.select().from(schema.episodeCharacters).where(and(eq(schema.episodeCharacters.episodeId,episodeId),eq(schema.episodeCharacters.characterId,characterId))).get()) + tx.insert(schema.episodeCharacters).values({episodeId,characterId,createdAt:ts}).run() + } + for (const propId of new Set(sb.prop_ids||[])) { + if (!tx.select().from(schema.episodeProps).where(and(eq(schema.episodeProps.episodeId,episodeId),eq(schema.episodeProps.propId,propId))).get()) + tx.insert(schema.episodeProps).values({episodeId,propId,createdAt:ts}).run() + } + const fields={title:sb.title,shotType:sb.shot_type,angle:sb.angle,movement:sb.movement, + location:sb.location,time:sb.time,description:sb.description,result:sb.result, + atmosphere:sb.atmosphere,imagePrompt:sb.image_prompt,videoPrompt:sb.video_prompt, + bgmPrompt:sb.bgm_prompt,...english.get(sb.shot_number),soundEffect:sb.sound_effect, + continuityMode:sb.continuity_mode,continuityStart:sb.continuity_start,continuityEnd:sb.continuity_end, + sceneId:sb.scene_id,duration:sb.duration||10,updatedAt:ts} + let storyboardId=shotToId.get(sb.shot_number) + if (storyboardId!==undefined) tx.update(schema.storyboards).set(fields).where(eq(schema.storyboards.id,storyboardId)).run() + else { + storyboardId=getInsertId(tx.insert(schema.storyboards).values({...fields,episodeId,storyboardNumber:sb.shot_number,createdAt:ts}).run()) + shotToId.set(sb.shot_number,storyboardId) + } + tx.delete(schema.storyboardCharacters).where(eq(schema.storyboardCharacters.storyboardId,storyboardId)).run() + tx.delete(schema.storyboardProps).where(eq(schema.storyboardProps.storyboardId,storyboardId)).run() + for (const characterId of new Set(sb.character_ids||[])) tx.insert(schema.storyboardCharacters).values({storyboardId,characterId}).run() + for (const propId of new Set(sb.prop_ids||[])) tx.insert(schema.storyboardProps).values({storyboardId,propId}).run() } - } + }) // 整集时长 = 当前全部存活分镜时长之和（分批保存时不能再按单批累加） const allRows = await db.select().from(schema.storyboards) @@ -399,6 +442,12 @@ const saveStoryboards = createTool({ ? `已保存 ${totalDuration} 秒，目标约 ${target} 秒（剧本字数 ÷ 500 字/分钟）。剧本还没拆完，继续按顺序保存后续段落，直到剧本最后一句。` : `已保存 ${totalDuration} 秒，目标约 ${target} 秒。` return { message: `Saved ${storyboards.length} storyboards. ${progress}`, count: storyboards.length, total_duration: totalDuration, target_total_seconds: target } + } catch(error) { + if (!jobId || committing) throw error + const message=error instanceof Error?error.message:'分镜保存失败' + if (jobId && !committing && getStoryboardJob(episodeId)?.status==='running') rejectStoryboardBatch(episodeId,jobId,message) + return {error:message} + } }, }) @@ -430,6 +479,11 @@ const updateStoryboard = createTool({ const ids = requireIds(context) if ('error' in ids) return ids const { episodeId, dramaId } = ids + const jobId = storyboardJobId(context) + if (jobId) { + assertStoryboardJob(episodeId, jobId) + return { error: '整集拆分期间只允许通过 save_storyboards 暂存新分镜，不能修改原分镜。' } + } const [storyboard] = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, storyboard_id)) if (!storyboard || storyboard.episodeId !== episodeId || storyboard.deletedAt) return { error: `Storyboard ${storyboard_id} not found in this episode` } @@ -487,7 +541,8 @@ const updateStoryboard = createTool({ if ('duration' in fields) updates.duration = fields.duration if ('video_prompt' in fields || 'video_prompt_en' in fields) Object.assign(updates, await bilingualFields(fields.video_prompt ?? storyboard.videoPrompt ?? '', fields.video_prompt_en, dramaId, - { configId: context?.requestContext?.get('textConfigId' as never) as number | undefined })) + { configId: context?.requestContext?.get('textConfigId' as never) as number | undefined, + model: context?.requestContext?.get('modelOverride' as never) as string | undefined })) await db.update(schema.storyboards).set(updates).where(eq(schema.storyboards.id, storyboard_id)) if ('character_ids' in fields) await syncStoryboardCharacters(storyboard_id, fields.character_ids || []) if ('prop_ids' in fields) await syncStoryboardProps(storyboard_id, fields.prop_ids || []) diff --git a/huobao-source/backend/src/agents/voice-reference-instructions.ts b/huobao-source/backend/src/agents/voice-reference-instructions.ts index 54be122..12e3caa 100644 --- a/huobao-source/backend/src/agents/voice-reference-instructions.ts +++ b/huobao-source/backend/src/agents/voice-reference-instructions.ts @@ -6,7 +6,7 @@ const instructions: Record = { - read_storyboard_context 的角色提供 has_voice 与 voice_reference；已有分镜提供 available_voice_references。只有 has_voice 为 true 的角色才有可用音色。 - 为某个分镜生成或重写 video_prompt 时，只使用该分镜 available_voice_references 中的音色；首次拆分时，只使用本段 character_ids 对应角色的 voice_reference。 - 先从 description 的台词确定说话角色。画面动作使用 @角色名；有台词且有可用音色时，在对应对白的说话人位置直接使用 #角色名，紧接语气、发声动作与台词。例如：0-3秒：@林小雨抬头，#林小雨轻声说：「早上好。」 -- 音色引用必须与对白写在同一个时间段内，不新增独立的「音色：」信息头。每次对白都明确标出自己的 #说话人；多人对话逐句对应，不能把所有音色集中列在开头。 +- 有可用音色时，音色引用与对白写在同一个时间段内，不新增独立的「音色：」信息头；没有可用音色时直接写 @角色名说：「原文台词」，无需上传音色也能拆分镜。多人对话逐句对应。 - 直接复制 voice_reference 的完整值（如 #林小雨），禁止改名、缩写、翻译或追加“音色”后缀。@角色名引用形象，#角色名引用音色，两者分别放在画面动作与发声位置。 - 没有上传音色的角色、未绑定角色、已移除音色和不说话的角色，不添加 # 引用。旁白只有明确对应某个已绑定、已有音色的角色时才可引用，不借用其他角色的音色。 - 音频仅用于参考音色，台词内容与说话时机遵循 description；不要为了使用音色新增台词、人物或镜头。 diff --git a/huobao-source/backend/src/db/schema.ts b/huobao-source/backend/src/db/schema.ts index 68f4e49..f0f34fe 100644 --- a/huobao-source/backend/src/db/schema.ts +++ b/huobao-source/backend/src/db/schema.ts @@ -99,6 +99,7 @@ export const scenes = sqliteTable('scenes', { time: text('time').notNull(), prompt: text('prompt').notNull(), lighting: text('lighting'), + continuityPrompt: text('continuity_prompt'), finalPrompt: text('final_prompt'), storyboardCount: integer('storyboard_count').default(1), imageUrl: text('image_url'), diff --git a/huobao-source/backend/src/db/sqlite-schema.ts b/huobao-source/backend/src/db/sqlite-schema.ts index d132834..4490eb9 100644 --- a/huobao-source/backend/src/db/sqlite-schema.ts +++ b/huobao-source/backend/src/db/sqlite-schema.ts @@ -403,6 +403,7 @@ export function initSqliteSchema(sqlite: Database.Database) { // Reference inputs are independent of generated outputs; preserve all existing rows. const sceneColumns = new Set((sqlite.prepare('PRAGMA table_info(scenes)').all() as { name: string }[]).map(c => c.name)) if (!sceneColumns.has('reference_images')) sqlite.exec('ALTER TABLE scenes ADD COLUMN reference_images TEXT') + if (!sceneColumns.has('continuity_prompt')) sqlite.exec('ALTER TABLE scenes ADD COLUMN continuity_prompt TEXT') const upgradeSeed = sqlite.prepare(UPGRADE_SQL) const removeSeed = sqlite.prepare(REMOVE_SQL) for (const s of stylePresetSeeds) { diff --git a/huobao-source/backend/src/index.ts b/huobao-source/backend/src/index.ts index 2c40539..f2265de 100644 --- a/huobao-source/backend/src/index.ts +++ b/huobao-source/backend/src/index.ts @@ -26,6 +26,7 @@ import storage from './routes/storage.js' import serverUpdate from './routes/serverUpdate.js' import { requestLogger, errorHandler } from './middleware/logger.js' import { resumeInterruptedTasks } from './services/generation.js' +import { resumeVideoCleanups } from './services/video-cleanup.js' import { DATA_ROOT } from './utils/paths.js' const __dirname = path.dirname(fileURLToPath(import.meta.url)) @@ -89,5 +90,6 @@ const port = Number(process.env.PORT || 5679) console.log(`🚀 Huobao Drama TS server on http://localhost:${port}`) await resumeInterruptedTasks() +resumeVideoCleanups() serve({ fetch: app.fetch, port }) diff --git a/huobao-source/backend/src/routes/agent.ts b/huobao-source/backend/src/routes/agent.ts index e7b301d..8d3c7fa 100644 --- a/huobao-source/backend/src/routes/agent.ts +++ b/huobao-source/backend/src/routes/agent.ts @@ -7,9 +7,23 @@ import { buildAgentRequestContext } from '../agents/context.js' import { mastra } from '../mastra/index.js' import { success, badRequest } from '../utils/response.js' import { logTaskError, logTaskPayload, logTaskProgress, logTaskStart, logTaskSuccess } from '../utils/task-logger.js' +import { startStoryboardJob, cancelStoryboardJob } from '../services/storyboard-jobs.js' +import { getStoryboardJob, publicStoryboardJob } from '../services/storyboard-job-state.js' const app = new Hono() +app.post('/storyboard_breaker/jobs',async c=>{ + const body=await c.req.json() + if (!Number.isInteger(body.episode_id)||!Number.isInteger(body.drama_id)||typeof body.message!=='string'||!body.message.trim()) return badRequest(c,'需要有效的项目、集数和拆分要求') + if(body.resume_from!==undefined&&(typeof body.resume_from!=='string'||!body.resume_from))return badRequest(c,'需要有效的暂存任务编号') + try { return success(c,startStoryboardJob(body)) } catch(error:any) { return badRequest(c,error.message) } +}) +app.get('/storyboard_breaker/jobs/:episodeId',c=>success(c,publicStoryboardJob(getStoryboardJob(Number(c.req.param('episodeId')))))) +app.post('/storyboard_breaker/jobs/:episodeId/cancel',async c=>{ + const body=await c.req.json() + try { return success(c,cancelStoryboardJob(Number(c.req.param('episodeId')),body.id)) } catch(error:any) { return badRequest(c,error.message) } +}) + // Mastra v1.17 的 ToolCallChunk / ToolResultChunk 结构： // { type: 'tool-call', payload: { toolCallId, toolName, args } } // { type: 'tool-result', payload: { toolCallId, toolName, result, isError } } diff --git a/huobao-source/backend/src/routes/episodes.ts b/huobao-source/backend/src/routes/episodes.ts index 9979704..39cf0c7 100644 --- a/huobao-source/backend/src/routes/episodes.ts +++ b/huobao-source/backend/src/routes/episodes.ts @@ -179,7 +179,7 @@ app.get('/:id/video-prompts-status', async (c) => { app.get('/:episode_id/storyboards', async (c) => { const episodeId = Number(c.req.param('episode_id')) const rows = await db.select().from(schema.storyboards) - .where(eq(schema.storyboards.episodeId, episodeId)) + .where(and(eq(schema.storyboards.episodeId, episodeId), isNull(schema.storyboards.deletedAt))) .orderBy(schema.storyboards.storyboardNumber) const links = await db.select().from(schema.storyboardCharacters) @@ -262,7 +262,10 @@ app.get('/:id/generation-tasks', async (c) => { .slice(0, 20) return success(c, { - tasks: toSnakeCaseArray(tasks.map(task => ({ ...task, routing: taskRouting(task), canCancel: canCancelGenerationTask(task) }))), + tasks: toSnakeCaseArray(tasks.map(task => ({ ...task, + storyboardNumber: sbs.find(sb => sb.id === task.storyboardId)?.storyboardNumber, + storyboardDeleted: !!sbs.find(sb => sb.id === task.storyboardId)?.deletedAt, + routing: taskRouting(task), canCancel: canCancelGenerationTask(task) }))), merges: toSnakeCaseArray(merges), }) }) @@ -274,7 +277,7 @@ app.get('/:id/pipeline-status', async (c) => { const chars = await db.select().from(schema.characters).where(eq(schema.characters.dramaId, ep.dramaId)) const scenes = await db.select().from(schema.scenes).where(eq(schema.scenes.dramaId, ep.dramaId)) - const sbs = await db.select().from(schema.storyboards).where(eq(schema.storyboards.episodeId, episodeId)) + const sbs = await db.select().from(schema.storyboards).where(and(eq(schema.storyboards.episodeId, episodeId), isNull(schema.storyboards.deletedAt))) const merges = await db.select().from(schema.videoMerges).where(eq(schema.videoMerges.episodeId, episodeId)) const sbsWithImage = sbs.filter(s => s.composedImage) diff --git a/huobao-source/backend/src/routes/hd.ts b/huobao-source/backend/src/routes/hd.ts index 9f3703a..3b79632 100644 --- a/huobao-source/backend/src/routes/hd.ts +++ b/huobao-source/backend/src/routes/hd.ts @@ -1,5 +1,5 @@ import { Hono } from 'hono' -import { and, asc, eq } from 'drizzle-orm' +import { and, asc, eq, isNull } from 'drizzle-orm' import fs from 'node:fs' import path from 'node:path' import { randomUUID } from 'node:crypto' @@ -31,7 +31,7 @@ async function versions(id: number) { } async function context(id: number) { const [sb] = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, id)) - if (!sb) throw new Error('分镜不存在') + if (!sb || sb.deletedAt) throw new Error('分镜不存在或已删除') const tasks = await versions(id) const original = tasks.map(t => params(t.params).originalVideoUrl).find(Boolean) || sb.videoUrl return { sb, tasks, original } @@ -40,7 +40,7 @@ async function context(id: number) { app.get('/episodes/:id', async c => { const id = Number(c.req.param('id')) if (!Number.isSafeInteger(id) || id <= 0) return badRequest(c, '集 ID 无效') - const boards = await db.select().from(schema.storyboards).where(eq(schema.storyboards.episodeId, id)) + const boards = await db.select().from(schema.storyboards).where(and(eq(schema.storyboards.episodeId, id), isNull(schema.storyboards.deletedAt))) const configs = (await db.select().from(schema.aiServiceConfigs)).filter(row => row.isActive && row.serviceType === 'video' && row.provider === 'localvideo' && row.model?.includes(MODEL)).sort((a, b) => (b.priority || 0) - (a.priority || 0)) return success(c, { configured: configs.length > 0, diff --git a/huobao-source/backend/src/routes/scenes.ts b/huobao-source/backend/src/routes/scenes.ts index c3d39ae..318e4a5 100644 --- a/huobao-source/backend/src/routes/scenes.ts +++ b/huobao-source/backend/src/routes/scenes.ts @@ -17,6 +17,7 @@ app.post('/', async (c) => { const body = await c.req.json() if (!body.drama_id) return badRequest(c, 'drama_id 必填') if (!body.location?.trim()) return badRequest(c, '地点必填') + if (body.continuity_prompt !== undefined && (typeof body.continuity_prompt !== 'string' || body.continuity_prompt.length > 4000)) return badRequest(c, '场景固定设定须为 4000 字以内的文本') const ts = now() const res = await db.insert(schema.scenes).values({ dramaId: body.drama_id, @@ -25,6 +26,7 @@ app.post('/', async (c) => { time: body.time || '', prompt: body.prompt || body.description || body.location, lighting: body.lighting || '', + continuityPrompt: body.continuity_prompt?.trim() || null, createdAt: ts, updatedAt: ts, }) @@ -51,6 +53,10 @@ app.put('/:id', async (c) => { if (body.prompt !== undefined) updates.prompt = body.prompt if (body.description !== undefined) updates.prompt = body.description if (body.lighting !== undefined) updates.lighting = body.lighting + if (body.continuity_prompt !== undefined) { + if (typeof body.continuity_prompt !== 'string' || body.continuity_prompt.length > 4000) return badRequest(c, '场景固定设定须为 4000 字以内的文本') + updates.continuityPrompt = body.continuity_prompt.trim() || null + } // 用户上传场景图：直接写入图片地址与本地路径 const uploadedImage = body.image_url ?? body.imageUrl if (uploadedImage !== undefined) { diff --git a/huobao-source/backend/src/routes/storyboards.ts b/huobao-source/backend/src/routes/storyboards.ts index 3799469..54d6763 100644 --- a/huobao-source/backend/src/routes/storyboards.ts +++ b/huobao-source/backend/src/routes/storyboards.ts @@ -1,5 +1,5 @@ import { Hono } from 'hono' -import { eq } from 'drizzle-orm' +import { and, desc, eq, isNull } from 'drizzle-orm' import { db, getInsertId, schema } from '../db/index.js' import { success, created, now, badRequest } from '../utils/response.js' import { toSnakeCase } from '../utils/transform.js' @@ -7,8 +7,26 @@ import { logTaskPayload, logTaskStart, logTaskSuccess } from '../utils/task-logg import { continuityDetails, continuityUpdate, extractContinuityFrame, resolveContinuity, sourceReadiness } from '../services/continuity.js' import { ensureStoryboardEnglish, englishSourceHash, promptAssetNames, validateEnglishPrompt, withEnglishStatus } from '../services/english-video-prompt.js' +import { resolveSceneLock } from '../services/scene-continuity.js' +import { deleteStoryboards } from '../services/storyboard-delete.js' const app = new Hono() +app.post('/batch-delete', async c => { + try { + const body = await c.req.json() + return success(c, deleteStoryboards(body.episode_id, body.storyboard_ids)) + } catch (error: any) { return badRequest(c, error.message) } +}) +app.get('/:id/environment', async c => { + try { + const id = Number(c.req.param('id')) + const current = await resolveSceneLock(id) + const latest = await db.select().from(schema.sysTask).where(and(eq(schema.sysTask.storyboardId, id), eq(schema.sysTask.type, 'video'))).orderBy(desc(schema.sysTask.id)).get() + let used: any = null + try { used = JSON.parse(latest?.params || '{}').sceneLock || null } catch { /* legacy task */ } + return success(c, { current, used, taskId: latest?.id, stale: !!latest && used?.hash !== current.hash }) + } catch (error: any) { return badRequest(c, error.message) } +}) app.post('/:id/english-prompt', async c => { try { const body = await c.req.json().catch(() => ({})) @@ -151,7 +169,7 @@ app.put('/:id', async (c) => { const id = Number(c.req.param('id')) const body = await c.req.json() const [storyboard] = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, id)) - if (!storyboard) return badRequest(c, '镜头不存在') + if (!storyboard || storyboard.deletedAt) return badRequest(c, '镜头不存在或已删除') logTaskStart('StoryboardAPI', 'update', { storyboardId: id, episodeId: storyboard.episodeId, @@ -190,7 +208,7 @@ app.put('/:id', async (c) => { 'prop_ids' in body ? body.prop_ids : await getStoryboardPropIds(id), ) - await db.update(schema.storyboards).set(updates).where(eq(schema.storyboards.id, id)) + await db.update(schema.storyboards).set(updates).where(and(eq(schema.storyboards.id, id), isNull(schema.storyboards.deletedAt))) if ('character_ids' in body) await syncStoryboardCharacters(id, body.character_ids || []) if ('prop_ids' in body) await syncStoryboardProps(id, body.prop_ids || []) logTaskSuccess('StoryboardAPI', 'update', { @@ -205,12 +223,13 @@ app.put('/:id', async (c) => { // DELETE /storyboards/:id app.delete('/:id', async (c) => { const id = Number(c.req.param('id')) - logTaskStart('StoryboardAPI', 'delete', { storyboardId: id }) - await db.delete(schema.storyboardCharacters).where(eq(schema.storyboardCharacters.storyboardId, id)) - await db.delete(schema.storyboardProps).where(eq(schema.storyboardProps.storyboardId, id)) - await db.delete(schema.storyboards).where(eq(schema.storyboards.id, id)) - logTaskSuccess('StoryboardAPI', 'delete', { storyboardId: id }) - return success(c) + try { + const shot = db.select().from(schema.storyboards).where(eq(schema.storyboards.id, id)).get() + if (!shot) return badRequest(c, '分镜不存在') + const result = deleteStoryboards(shot.episodeId, [id]) + logTaskSuccess('StoryboardAPI', 'delete', { storyboardId: id }) + return success(c, result) + } catch (error: any) { return badRequest(c, error.message) } }) export default app diff --git a/huobao-source/backend/src/routes/tasks.ts b/huobao-source/backend/src/routes/tasks.ts index d08770b..c37c690 100644 --- a/huobao-source/backend/src/routes/tasks.ts +++ b/huobao-source/backend/src/routes/tasks.ts @@ -6,6 +6,7 @@ import { generateImage, generateVideo, recoverGenerationTask, cancelWaitingVideo import { safeTaskError } from '../services/video-download.js' import { taskRouting } from '../services/worker-routing.js' import { getActiveConfig, getConfigById } from '../services/ai.js' +import { needsComposedFirstFrame } from '../services/first-frame-composition.js' import { getDramaStylePrompt } from '../services/style-preset.js' import { resolveStoryboardVoiceReferences } from '../services/character-voice-references.js' import { logTaskError, logTaskPayload, logTaskStart, logTaskSuccess } from '../utils/task-logger.js' @@ -124,7 +125,8 @@ function validateVideoRequest(body: any, provider?: string): string | null { const total = imgs + vids + auds + Number(first) + Number(last) + Number(file) + Number(link) if (total > 20) return 'Wan 3.0 input.media 最多 20 项' } else { - if (imgs > 9 || vids > 3 || auds > 3) return '参考素材超限：图片≤9、视频≤3、音频≤3' + const composing = body.storyboard_id && needsComposedFirstFrame(body.model) && !first && !body.image_url + if ((!composing && imgs > 9) || vids > 3 || auds > 3) return '参考素材超限：图片≤9、视频≤3、音频≤3' if (auds > 0 && imgs + vids === 0) return '参考音频需要至少 1 个参考图片或视频' } @@ -163,6 +165,7 @@ app.post('/', async (c) => { let storyboardDramaId: number | undefined if (body.storyboard_id) { const [sb] = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, Number(body.storyboard_id))) + if (!sb || sb.deletedAt) return badRequest(c, '分镜不存在或已删除') if (sb) { const [ep] = await db.select().from(schema.episodes).where(eq(schema.episodes.id, sb.episodeId)) const locked = type === 'image' ? ep?.imageConfigId : ep?.videoConfigId @@ -176,6 +179,7 @@ app.post('/', async (c) => { const effectiveConfig = configId ? (await getConfigById(configId)) ?? await getActiveConfig('video') : await getActiveConfig('video') + videoBody.model ||= effectiveConfig?.model const validationError = validateVideoRequest(videoBody, effectiveConfig?.provider) if (validationError) return badRequest(c, validationError) const voices = await resolveStoryboardVoiceReferences(String(videoBody.prompt || ''), Number(body.storyboard_id) || undefined, videoBody.reference_audio_urls) @@ -217,6 +221,9 @@ app.post('/', async (c) => { configId, }) : await generateVideo({ + frameImageConfigId: body.frame_image_config_id, + frameImageModel: body.frame_image_model, + regenerateFirstFrame: body.regenerate_first_frame === true, gpu: body.gpu, replaceTaskId: body.replace_task_id, forceNew: body.force_new === true, diff --git a/huobao-source/backend/src/services/adapters/local-video.ts b/huobao-source/backend/src/services/adapters/local-video.ts index ca663e0..4944ad2 100644 --- a/huobao-source/backend/src/services/adapters/local-video.ts +++ b/huobao-source/backend/src/services/adapters/local-video.ts @@ -12,6 +12,14 @@ export class LocalVideoAdapter extends MiniMaxVideoAdapter { if (record.referenceFileUrl || record.referenceLinkUrl) throw new Error('本地视频暂不支持 file / link 参考素材') const request = super.buildGenerateRequest(config, { ...record, model: 'MiniMax-H3' }) const body = request.body as Record + const content = body.content as Array<{ type: string; role?: string }> + const images = content.filter(item => item.type === 'image_url') + const singleImage = ['LTX-2.3', 'LTX-2.3-Text-Preserve', 'Wan2.2-I2V'].includes(model) + const limit = singleImage ? 1 : 9 + if (images.length > limit) throw new Error(`${model} 最多支持 ${limit} 张参考图，当前 ${images.length} 张；请选择单张参考图或改用 MiniMax-H3`) + if (['Wan2.2-I2V', 'MiniMax-H3-R2V-4Step', 'MiniMax-H3-VDN-H3'].includes(model) && !images.length) throw new Error(`${model} 需要至少 1 张参考图`) + if (content.some(item => !['text', 'image_url'].includes(item.type) || item.role === 'last_frame')) throw new Error('本地视频工作流暂不支持参考视频、音色或尾帧，请移除后提交') + if (images.some(item => item.role === 'first_frame') && images.length !== 1) throw new Error('首帧模式只能提交 1 张图片，不能混用多张参考图') const resolution = String(record.resolution || '768P').toUpperCase() if (!['480P', '720P', '768P', '1080P', '2K', '4K'].includes(resolution)) throw new Error('不支持的本地视频分辨率') if (resolution === '4K' && model !== 'MiniMax-H3-VDN-H3') throw new Error('4K 需要 VDN-H3 高清制作模型') diff --git a/huobao-source/backend/src/services/continuity.ts b/huobao-source/backend/src/services/continuity.ts index a22dd3e..d3353b8 100644 --- a/huobao-source/backend/src/services/continuity.ts +++ b/huobao-source/backend/src/services/continuity.ts @@ -84,8 +84,8 @@ export function validateContinuityModel(plan: ContinuityPlan, config: AIConfig, let local = false try { const u = new URL(config.baseUrl); local = u.hostname === '192.168.110.111' && ['18792', '18794'].includes(u.port) } catch {} // These local workflows implement genuine image-to-video; VDN/R2V-only workflows do not. - if (!local || !['MiniMax-H3', 'MiniMax-H3-Hybrid', 'Wan2.2-I2V'].includes(model || config.model || '')) { - throw new Error('尾帧接续请选择本地 MiniMax-H3、MiniMax-H3-Hybrid 或 Wan2.2-I2V；当前模型可使用状态承接') + if (!local || !['MiniMax-H3', 'MiniMax-H3-Hybrid', 'Wan2.2-I2V', 'LTX-2.3', 'LTX-2.3-Text-Preserve'].includes(model || config.model || '')) { + throw new Error('尾帧接续请选择本地 MiniMax-H3、MiniMax-H3-Hybrid、LTX-2.3 或 Wan2.2-I2V；当前模型可使用状态承接') } } diff --git a/huobao-source/backend/src/services/english-video-prompt.ts b/huobao-source/backend/src/services/english-video-prompt.ts index 7b00155..8fb102f 100644 --- a/huobao-source/backend/src/services/english-video-prompt.ts +++ b/huobao-source/backend/src/services/english-video-prompt.ts @@ -4,6 +4,7 @@ import { and, eq } from 'drizzle-orm' import { db, schema } from '../db/index.js' import { getConfigById, getTextConfig, getTextProviderBaseUrl } from './ai.js' import { now } from '../utils/response.js' +import { isSpeechPrefix } from '../utils/spoken-literals.js' type Shot = typeof schema.storyboards.$inferSelect type TextOptions = { configId?: number; model?: string } @@ -49,7 +50,7 @@ export function validateEnglishPrompt(source: string, candidate: string, names: // Broadcasts/off-screen speech also need an explicit cue understood by H3's // audio parser. Add only a label to speech already present in the source. for (const match of source.matchAll(/([^。\n！？]{0,90})[:：]\s*(「[^」]*」|“[^”]*”|"[^"\n]*")/g)) { - if (!/(说|旁白|台词|广播|响起|喊|问|答|低语|低声|画外|念|嘀咕|声音|语音|系统音|提示音|播报|开口|心声|says|dialogue|voice|narrat)/i.test(match[1])) continue + if (!isSpeechPrefix(match[1]+':')) continue const literal = match[2], before = candidate candidate = candidate.replaceAll(literal, (value, offset) => /(?:says|Dialogue):\s*$/i.test(before.slice(0,offset)) ? value : 'Dialogue: ' + value) } @@ -74,6 +75,9 @@ export async function translateVisualPrompt(source: string, names: string[] = [] } else { url = `${base}/chat/completions`; headers = { Authorization: `Bearer ${config.apiKey}`, 'Content-Type': 'application/json' } body = { model: opts.model || config.model, temperature: 0.1, max_tokens: 5000, messages: [{ role: 'system', content: instruction }, { role: 'user', content: protectedPrompt.text }] } + // This direct fetch bypasses the Agent's thinking-off wrapper. Doubao + // otherwise defaults to deep reasoning for each individual translation. + if (config.provider === 'volcengine' && /^doubao-seed-/.test(opts.model || config.model)) body.thinking = { type: 'disabled' } } const response = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body), signal: AbortSignal.timeout(90_000) }) if (!response.ok) throw new Error(`英文提示词生成失败（文本服务 HTTP ${response.status}），未提交视频，请稍后重试`) @@ -103,6 +107,24 @@ export async function bilingualFields(source: string, candidate?: string, dramaI return { videoPromptEn: text, videoPromptEnSourceHash: englishSourceHash(source) } } +/** Validate supplied EN immediately; at most three independent fallback translations + * run at a time. All finish before the caller replaces or writes any storyboards. */ +export async function bilingualBatchFields(items: { shot_number: number; video_prompt?: string; video_prompt_en?: string }[], dramaId: number, opts: TextOptions = {}) { + const results: Array<[number, Awaited>]> = [] + let next = 0, failure: unknown + const worker = async () => { + while (failure === undefined && next < items.length) { + const index = next++, item = items[index] + if (!item.video_prompt?.trim()) continue + try { results[index] = [item.shot_number, await bilingualFields(item.video_prompt, item.video_prompt_en, dramaId, opts)] } + catch (error) { failure = error } + } + } + await Promise.all(Array.from({ length: Math.min(3, items.length) }, worker)) + if (failure !== undefined) throw failure + return new Map(results.filter(Boolean)) +} + const englishJobs = new Map>() export async function ensureStoryboardEnglish(id: number, force = false, opts: TextOptions = {}): Promise { if (englishJobs.has(id)) return englishJobs.get(id)! @@ -130,11 +152,12 @@ export async function englishSubmissionPrompt(input: string, storyboardId?: numb const ep = await db.select().from(schema.episodes).where(eq(schema.episodes.id,shot.episodeId)).get() dramaId = ep?.dramaId const original = shot.videoPrompt?.trim() - const normalized = input.replace(/@图片\d+/g,'@') + const names = await promptAssetNames(dramaId) + // Non-greedy index matching preserves numbered place names such as 404宿舍. + const normalized = names.length ? input.replace(new RegExp(`@图片\\d+?(?=(?:${names.map(escape).join('|')}))`, 'g'), '@') : input if (original && normalized.includes(original)) { const english = await ensureStoryboardEnglish(shot.id) prompt = normalized.replace(original,english.videoPromptEn!) - const names = await promptAssetNames(dramaId) for (const name of names) { const match = input.match(new RegExp('@图片(\\d+)' + escape(name))) if (match) prompt = prompt.replace(new RegExp('@' + escape(name), 'g'), `@图片${match[1]}${name}`) diff --git a/huobao-source/backend/src/services/ffmpeg-merge.ts b/huobao-source/backend/src/services/ffmpeg-merge.ts index a98a038..3d54de5 100644 --- a/huobao-source/backend/src/services/ffmpeg-merge.ts +++ b/huobao-source/backend/src/services/ffmpeg-merge.ts @@ -5,7 +5,7 @@ import fs from 'fs' import path from 'path' import { v4 as uuid } from 'uuid' import { db, getInsertId, schema } from '../db/index.js' -import { eq } from 'drizzle-orm' +import { and, eq, isNull } from 'drizzle-orm' import { now } from '../utils/response.js' import { logTaskError, logTaskStart, logTaskSuccess } from '../utils/task-logger.js' import { extractVideoPoster } from '../utils/video-poster.js' @@ -25,7 +25,7 @@ function toAbsPath(relativePath: string): string { */ export async function mergeEpisodeVideos(episodeId: number, dramaId: number, storyboardIds?: number[]): Promise { let storyboards = await db.select().from(schema.storyboards) - .where(eq(schema.storyboards.episodeId, episodeId)) + .where(and(eq(schema.storyboards.episodeId, episodeId), isNull(schema.storyboards.deletedAt))) .orderBy(schema.storyboards.storyboardNumber) if (storyboardIds?.length) { diff --git a/huobao-source/backend/src/services/first-frame-composition.ts b/huobao-source/backend/src/services/first-frame-composition.ts new file mode 100644 index 0000000..5e9bfe4 --- /dev/null +++ b/huobao-source/backend/src/services/first-frame-composition.ts @@ -0,0 +1,100 @@ +/** Durable all-reference preparation for single-image video workflows. */ +import { createHash } from 'node:crypto' +import { eq } from 'drizzle-orm' +import { db, schema } from '../db/index.js' +import { getActiveConfig, getConfigById } from './ai.js' +import { resolveSceneLock, type SceneLock } from './scene-continuity.js' +import type { AIConfig } from './adapters/types.js' + +export const needsComposedFirstFrame = (model?: string | null) => ['LTX-2.3', 'LTX-2.3-Text-Preserve', 'Wan2.2-I2V'].includes(model || '') +export interface FrameAsset { kind: 'scene' | 'character' | 'prop'; id: number; name: string; url: string; description: string } +export interface FrameComposition { + version: 1; sourceHash: string; assets: FrameAsset[]; scene: SceneLock + shot: { description: string; videoPrompt: string; start: string; end: string } + sourcePrompt: string; size: string + imageConfigId: number; imageProvider: string; imageBaseUrl: string; imageModel: string; imageGpu: string + rounds: number; round: number; childTaskIds: number[]; images: string[] + frameUrl?: string; reusedFromTaskId?: number; continuityFrameUrl?: string +} + +async function shotInputs(id: number) { + const shot = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, id)).get() + if (!shot || shot.deletedAt) throw new Error('首帧合成：分镜不存在或已删除') + const episode = await db.select().from(schema.episodes).where(eq(schema.episodes.id, shot.episodeId)).get() + if (!episode || episode.deletedAt) throw new Error('首帧合成：当前集不存在') + const scene = await resolveSceneLock(id) + const assets: FrameAsset[] = [{ kind: 'scene', id: scene.sceneId, name: scene.location, url: scene.imageUrl, description: scene.environment }] + const characters = await db.select().from(schema.storyboardCharacters).where(eq(schema.storyboardCharacters.storyboardId, id)) + for (const link of characters.sort((a,b) => a.characterId-b.characterId)) { + const item = await db.select().from(schema.characters).where(eq(schema.characters.id, link.characterId)).get() + if (!item || item.deletedAt || item.dramaId !== episode.dramaId) throw new Error('首帧合成：绑定角色已失效，请重新绑定') + assets.push({ kind:'character', id:item.id, name:item.name, url:item.imageUrl || item.localPath || '', description:[item.appearance,item.description].filter(Boolean).join('；') }) + } + const props = await db.select().from(schema.storyboardProps).where(eq(schema.storyboardProps.storyboardId, id)) + for (const link of props.sort((a,b) => a.propId-b.propId)) { + const item = await db.select().from(schema.props).where(eq(schema.props.id, link.propId)).get() + if (!item || item.deletedAt || item.dramaId !== episode.dramaId) throw new Error('首帧合成：绑定道具已失效，请重新绑定') + assets.push({ kind:'prop', id:item.id, name:item.name, url:item.imageUrl || item.localPath || '', description:item.description || '' }) + } + const missing = assets.filter(a => !a.url) + if (missing.length) throw new Error(`首帧合成：${missing.map(a=>a.name).join('、')}尚无参考图，请先生成或上传；未忽略任何绑定素材`) + return { episode, scene, assets, shot:{ description:shot.description || '', videoPrompt:shot.videoPrompt || '', start:shot.continuityStart || '', end:shot.continuityEnd || '' } } +} +function inputHash(value: Pick) { + return createHash('sha256').update(JSON.stringify({ assets:value.assets, scene:value.scene.hash, shot:value.shot })).digest('hex') +} +export async function assertFrameInputsCurrent(id: number, plan: FrameComposition) { + if (inputHash(await shotInputs(id)) !== plan.sourceHash) throw new Error('首帧合成：分镜、绑定素材或场景设定已变化，请重新生成；旧首帧保留，未提交视频') +} +export function frameImageSize(ratio = '16:9') { + const sizes: Record = { '16:9':'1280x720', '9:16':'720x1280', '1:1':'1024x1024', '4:3':'1152x864', '3:4':'864x1152', '21:9':'1344x576' } + if (!sizes[ratio]) throw new Error('首帧合成：不支持的画面比例') + return sizes[ratio] +} +/** First request carries six originals. Later requests carry the prior full scene + five new originals. */ +export function frameBatches(assets: T[], continuation = false): T[][] { + const batches: T[][] = [] + for (let i=0; i { + const config = await getConfigById(plan.imageConfigId) + if (!config || config.baseUrl !== plan.imageBaseUrl || config.provider !== plan.imageProvider) throw new Error('首帧合成：原图片服务配置已停用或变更，请重新选择后生成') + return { ...config, model:plan.imageModel, gpu:plan.imageGpu as AIConfig['gpu'] } +} +export function withoutImageIndexes(text: string, assets: FrameAsset[]) { + const escape = (s:string) => s.replace(/[.*+?^${}()|[\]\\]/g,'\\$&') + const names = [...new Set(assets.map(a=>a.name))].sort((a,b)=>b.length-a.length) + return text.replace(new RegExp(`@图片\\d+?(?=(?:${names.map(escape).join('|')}))`,'g'),'@') +} +export function compositionRound(plan: FrameComposition, round: number) { + const batches = frameBatches(plan.assets, !!plan.continuityFrameUrl) + const batch = batches[round] + if (!batch) throw new Error('首帧合成：轮次记录无效') + const previous = round ? plan.images[round-1] : plan.continuityFrameUrl + const references = [...(previous ? [previous] : []), ...batch.map(a=>a.url)] + const list = batch.map((asset,index) => `图片${index+1+(previous?1:0)}：${asset.kind==='scene'?'场景环境':asset.kind==='character'?'人物身份与服装':'道具外形'}「${asset.name}」；${asset.description}`).join('\n') + const initialShot = plan.shot.description.split(/【镜头\s*2】/)[0] + const prompt = `生成一张完整的电影分镜首帧，只有一个连续画面，禁止拼图、分屏、三视图、人物设定板、文字标签和字幕。\n`+ + `全部绑定素材：${plan.assets.map(a=>a.name).join('、')}。严格区分人物身份、服装、道具与场景；人物参考图中的白底或背景不替换场景。参考图只定义素材，不强行让后续才登场的人物提前进入首帧。\n`+ + (previous ? `图片1是${round?'上轮融合的完整首帧':'前镜头的最后一帧'}：保留已有的人脸、服装、场景布局、道具位置和机位，只按本轮新参考图补充或校正相应素材，不能删除或替换其他已确定元素。\n` : '')+ + `本轮图像顺序与用途：\n${list}\n`+ + `固定场景：${plan.scene.location}；${plan.scene.environment}\n时间：${plan.scene.time}；光源：${plan.scene.lighting}\n`+ + `首个镜头描述：${initialShot}\n开始状态：${plan.shot.start || '采用第一个镜头/第一个时间段的起始状态'}\n`+ + `下面是整段视频的风格和动作上下文，仅用于确定首帧。只画动作开始前或开始瞬间，不把多个时间段同时画进一张图，不提前画结束状态，不把对白画成文字：\n${withoutImageIndexes(plan.sourcePrompt,plan.assets)}` + if (prompt.length > 32000) throw new Error('首帧合成：提示词过长，请精简分镜描述后重试') + return { references, prompt, assets:batch, rounds:batches.length } +} diff --git a/huobao-source/backend/src/services/generation.ts b/huobao-source/backend/src/services/generation.ts index 23b6da4..39c2404 100644 --- a/huobao-source/backend/src/services/generation.ts +++ b/huobao-source/backend/src/services/generation.ts @@ -7,6 +7,7 @@ import { and, desc, eq, gt, ne, or, isNull } from 'drizzle-orm' import fs from 'node:fs' import { getAbsolutePath } from '../utils/storage.js' import { downloadVideo, safeTaskError } from './video-download.js' +import { supportsVideoCleanup, videoOutputIndex, videoReceipt, verifyRetainedVideo, pendingVideoCleanup, cleanupSavedVideo } from './video-cleanup.js' import { prepareReferenceAudio, resolveReferenceAudioUrls } from './reference-audio.js' import { getActiveConfig, getConfigById } from './ai.js' import { parseWorkerRouting, type WorkerRouting } from './worker-routing.js' @@ -22,6 +23,8 @@ import { logTaskError, logTaskPayload, logTaskProgress, logTaskStart, logTaskSuc import { continuityPrompt, extractContinuityFrame, resolveContinuity, sourceReadiness, validateContinuityModel, type ContinuityPlan } from './continuity.js' import { englishSubmissionPrompt, usesEnglishVideoPrompt } from './english-video-prompt.js' +import { resolveSceneLock, assertSceneLockCurrent, addSceneReference, sceneLockPrompt, stripSceneLock } from './scene-continuity.js' +import { needsComposedFirstFrame, prepareFrameComposition, assertFrameInputsCurrent, frameImageConfig, frameBatches, compositionRound, withoutImageIndexes, type FrameComposition } from './first-frame-composition.js' type TaskType = 'image' | 'video' @@ -51,6 +54,9 @@ interface GenerateImageParams { } interface GenerateVideoParams { + frameImageConfigId?: number + frameImageModel?: string + regenerateFirstFrame?: boolean gpu?: AIConfig['gpu'] replaceTaskId?: number variant?: 'hd' @@ -86,6 +92,13 @@ export async function generateImage(params: GenerateImageParams): Promisea.url)) + const firstBatch = compositionRound(composition,0) + getImageAdapter(prepared.config.provider).buildGenerateRequest(prepared.config,{id:0,model:composition.imageModel,prompt:firstBatch.prompt,size:composition.size, + referenceImages:JSON.stringify(await normalizeReferenceImages(firstBatch.references,true))}) + if (continuity?.mode !== 'first_frame' && !params.regenerateFirstFrame) { + const previous = await db.select().from(schema.sysTask).where(and(eq(schema.sysTask.type,'video'),eq(schema.sysTask.storyboardId,storyboardId))).orderBy(desc(schema.sysTask.id)) + for (const task of previous) { + const old = parseTaskParams(task.params).firstFrameComposition as FrameComposition | undefined + if (old?.frameUrl && old.sourceHash===composition.sourceHash && old.sourcePrompt===composition.sourcePrompt && old.size===composition.size + && old.imageConfigId===composition.imageConfigId && old.imageModel===composition.imageModel && old.imageGpu===composition.imageGpu + && old.imageProvider===composition.imageProvider && old.imageBaseUrl===composition.imageBaseUrl + && fs.existsSync(getAbsolutePath(old.frameUrl))) { + composition = { ...composition, frameUrl:old.frameUrl, images:old.images, childTaskIds:old.childTaskIds, round:old.rounds, rounds:old.rounds, reusedFromTaskId:task.id }; break + } + } + } + params.prompt = withoutImageIndexes(params.prompt,composition.assets) + params.referenceImageUrls = [] + } + // Frame workflows already carry the environment in their single visual input. + const sceneFrame = !!composition || continuity?.mode === 'first_frame' || params.variant === 'hd' || !!(params.firstFrameUrl || params.imageUrl) + let sceneReferenceIndex: number | null = null + if (sceneLock && !sceneFrame && !params.referenceFileUrl && !params.referenceLinkUrl) { + const limit = /^(?:LTX-2\.3|Wan2\.2-I2V)/.test(params.model || config.model || '') ? 1 : config.provider === 'aliyun' ? 10 : 9 + const references = addSceneReference(sceneLock, params.referenceImageUrls, limit) + params.referenceImageUrls = references.urls + sceneReferenceIndex = references.index + } if (continuity) validateContinuityModel(continuity, config, params.model) if (continuity?.mode === 'first_frame' && params.referenceAudioUrls?.length) throw new Error('当前尾帧接续工作流不支持音色参考，请改为状态承接') const previewParams = { ...params, prompt: continuity ? continuityPrompt(params.prompt, continuity) : params.prompt } const english = usesEnglishVideoPrompt(params.model || config.model) // Translate and validate BEFORE cancelling an old job or admitting a new one. if (english) previewParams.prompt = await englishSubmissionPrompt(previewParams.prompt, storyboardId, params.dramaId) - if (continuity?.mode === 'first_frame') { + // Append after translation so the deterministic layout rules cannot disappear in an LLM rewrite. + if (sceneLock) previewParams.prompt = await sceneLockPrompt(previewParams.prompt, sceneLock, { english, referenceIndex: sceneReferenceIndex, frame: sceneFrame }) + if (continuity?.mode === 'first_frame' || composition) { // A validation-only placeholder; never submitted as a generation input. previewParams.firstFrameUrl = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aN1kAAAAASUVORK5CYII=' previewParams.imageUrl = undefined; previewParams.lastFrameUrl = undefined @@ -169,6 +221,7 @@ export async function generateVideo(params: GenerateVideoParams): Promise, ): Promise { const ts = now() - const res = await db.insert(schema.sysTask).values({ + const res = db.transaction(tx => { + if (fields.storyboardId) { + const shot = tx.select().from(schema.storyboards).where(eq(schema.storyboards.id, fields.storyboardId)).get() + if (!shot || shot.deletedAt) throw new Error('分镜不存在或已删除，未提交生成任务') + } + return tx.insert(schema.sysTask).values({ type, ...fields, provider: config.provider, configId: config.id, configBaseUrl: config.baseUrl, - stage: (params.continuity as ContinuityPlan | undefined)?.mode === 'first_frame' ? 'waiting_predecessor' : 'submitting', + stage: (params.continuity as ContinuityPlan | undefined)?.mode === 'first_frame' ? 'waiting_predecessor' : params.firstFrameComposition ? 'preparing_first_frame' : 'submitting', params: JSON.stringify({ ...params, gpu: config.gpu || 'auto' }), status: 'processing', createdAt: ts, updatedAt: ts, + }).run() }) const id = getInsertId(res) @@ -354,6 +424,97 @@ function parseTaskParams(raw: string | null | undefined): Record { } } +async function settleCompositionCancellation(id: number): Promise<'cancelled'|'pending'> { + if (cancellingTasks.has(id)) return 'pending' + cancellingTasks.add(id) + try { + const parent = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (!parent || parent.status !== 'processing') return 'cancelled' + const params = parseTaskParams(parent.params), plan = params.firstFrameComposition as FrameComposition + const childId = plan.childTaskIds.at(-1) + const child = childId ? await db.select().from(schema.sysTask).where(eq(schema.sysTask.id,childId)).get() : null + if (child?.status === 'processing') { + if (!canCancelGenerationTask(child)) return 'pending' + const result = await cancelGenerationTask(child.id) + if (result.outcome === 'pending') return 'pending' + } + await markTaskCancelled(id,params.cancellation?.reason) + return 'cancelled' + } finally { cancellingTasks.delete(id) } +} + +async function prepareComposedFirstFrame(id: number): Promise { + while (true) { + const parent = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (!parent || parent.status !== 'processing') return false + const params = parseTaskParams(parent.params), plan = params.firstFrameComposition as FrameComposition | undefined + if (!plan || !parent.storyboardId) return true + if (params.cancellation) { + if (await settleCompositionCancellation(id) === 'cancelled') return false + await new Promise(r=>setTimeout(r,1000)); continue + } + if (plan.frameUrl) { + await assertFrameInputsCurrent(parent.storyboardId,plan) + if (!fs.existsSync(getAbsolutePath(plan.frameUrl))) throw new Error('首帧合成：已保存的首帧文件不存在，请重做首帧') + params.firstFrameUrl=plan.frameUrl; params.imageUrl=undefined; params.lastFrameUrl=undefined + params.referenceImageUrls=[]; params.referenceVideoUrls=[]; params.referenceAudioUrls=[] + params.referenceMode='first_frame' + const claimed = db.transaction(tx=>{ + const current = tx.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (!current || current.status!=='processing' || parseTaskParams(current.params).cancellation) return false + tx.update(schema.sysTask).set({params:JSON.stringify(params),stage:'first_frame_ready',updatedAt:now()}).where(eq(schema.sysTask.id,id)).run() + tx.update(schema.storyboards).set({firstFrameImage:plan.frameUrl,updatedAt:now()}).where(eq(schema.storyboards.id,parent.storyboardId!)).run() + return true + }) + return claimed + } + const round = plan.round + const childId = plan.childTaskIds[round] + if (childId) { + const child = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id,childId)).get() + if (!child) throw new Error('首帧合成：关联图片任务已删除，未提交视频，请重做首帧') + if (child.status==='failed') { + if (child.errorCode==='user_cancelled') { await markTaskCancelled(id); return false } + throw new Error(`首帧合成：第 ${round+1}/${plan.rounds} 轮失败，未提交视频。${safeTaskError(child.errorMsg || '请检查图片任务后重试')}`) + } + if (child.status==='completed') { + if (!child.localPath) throw new Error('首帧合成：图片尚未保存到本地，未提交视频') + await assertFrameInputsCurrent(parent.storyboardId,plan) + plan.images[round]=child.localPath; plan.round=round+1 + if (plan.round>=plan.rounds) plan.frameUrl=child.localPath + // Preserve a cancellation that arrived while checking image/scene state. + db.transaction(tx=>{ + const current = tx.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (current?.status==='processing' && !parseTaskParams(current.params).cancellation) { + tx.update(schema.sysTask).set({params:JSON.stringify(params),updatedAt:now()}).where(eq(schema.sysTask.id,id)).run() + } + }) + continue + } + await new Promise(r=>setTimeout(r,1000)); continue + } + await assertFrameInputsCurrent(parent.storyboardId,plan) + const config = await frameImageConfig(plan) + const batch = compositionRound(plan,round) + const prepared = await normalizeReferenceImages(batch.references,true) + getImageAdapter(config.provider).buildGenerateRequest(config,{id:0,model:plan.imageModel,prompt:batch.prompt,size:plan.size,referenceImages:JSON.stringify(prepared)}) + const child = db.transaction(tx=>{ + const current = tx.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (!current || current.status!=='processing' || parseTaskParams(current.params).cancellation) return 0 + const inserted=tx.insert(schema.sysTask).values({type:'image',storyboardId:parent.storyboardId,dramaId:parent.dramaId, + provider:config.provider,configId:config.id,configBaseUrl:config.baseUrl,model:plan.imageModel,prompt:batch.prompt, + stage:'prepared',status:'processing',createdAt:now(),updatedAt:now(),params:JSON.stringify({size:plan.size, + referenceImages:batch.references,frameType:'video_first_frame_stage',compositionParentTaskId:id, + compositionRound:round+1,compositionRounds:batch.rounds,referenceAssets:batch.assets,gpu:plan.imageGpu})}).run() + const imageId=getInsertId(inserted) + plan.childTaskIds[round]=imageId; plan.rounds=batch.rounds + tx.update(schema.sysTask).set({params:JSON.stringify(params),stage:'preparing_first_frame',updatedAt:now()}).where(eq(schema.sysTask.id,id)).run() + return imageId + }) + if (child) runBackground(child,()=>processTask(child,config)) + } +} + async function prepareContinuityTask(record: SysTaskRecord): Promise { const params = parseTaskParams(record.params) const plan = params.continuity as ContinuityPlan | undefined @@ -361,8 +522,9 @@ async function prepareContinuityTask(record: SysTaskRecord): Promise { while (true) { const current = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id, record.id)).get() if (!current || current.status !== 'processing') return false + if (parseTaskParams(current.params).cancellation) return false const fresh = (await resolveContinuity(record.storyboardId)).plan - if (JSON.stringify(fresh) !== JSON.stringify(plan)) throw new Error('镜头衔接设置或前镜头结束状态已变化，请重新提交此镜头') + if (JSON.stringify(fresh) !== JSON.stringify(Object.fromEntries(Object.keys(fresh).map(key=>[key,plan[key as keyof ContinuityPlan]])))) throw new Error('镜头衔接设置或前镜头结束状态已变化，请重新提交此镜头') const source = plan.sourceId ? await sourceReadiness(plan.sourceId) : null if (plan.mode === 'first_frame' && source?.state !== 'ready') { await db.update(schema.sysTask).set({ stage: source?.state || 'waiting_predecessor', updatedAt: now() }) @@ -375,6 +537,10 @@ async function prepareContinuityTask(record: SysTaskRecord): Promise { const after = await sourceReadiness(plan.sourceId!) if (after.state !== 'ready' || after.video !== source.video || after.taskId !== source.taskId) continue params.firstFrameUrl = frameUrl; params.imageUrl = undefined; params.lastFrameUrl = undefined + if (params.firstFrameComposition) { + params.firstFrameComposition.continuityFrameUrl = frameUrl + params.firstFrameComposition.rounds = frameBatches(params.firstFrameComposition.assets,true).length + } params.referenceImageUrls = []; params.referenceVideoUrls = []; params.referenceAudioUrls = [] params.referenceFileUrl = undefined; params.referenceLinkUrl = undefined params.referenceMode = 'first_frame' @@ -383,9 +549,13 @@ async function prepareContinuityTask(record: SysTaskRecord): Promise { params.continuity = { ...plan, sourceVideo: source?.video || '', sourceTaskId: source?.taskId ?? null } } const prompt = params.promptIncludesContinuity ? record.prompt || '' : continuityPrompt(record.prompt || '', plan) - const claimed = await db.update(schema.sysTask).set({ params: JSON.stringify(params), prompt, stage: 'submitting', updatedAt: now() }) - .where(and(eq(schema.sysTask.id, record.id), eq(schema.sysTask.status, 'processing'), isNull(schema.sysTask.taskId))).returning({ id: schema.sysTask.id }) - return claimed.length > 0 + return db.transaction(tx=>{ + const latest=tx.select().from(schema.sysTask).where(eq(schema.sysTask.id,record.id)).get() + if (!latest || latest.status!=='processing' || latest.taskId || parseTaskParams(latest.params).cancellation) return false + tx.update(schema.sysTask).set({params:JSON.stringify(params),prompt,stage:params.firstFrameComposition?'preparing_first_frame':'submitting',updatedAt:now()}) + .where(eq(schema.sysTask.id,record.id)).run() + return true + }) } } @@ -393,7 +563,14 @@ async function processTask(id: number, config: AIConfig) { try { let [record] = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id, id)) if (!record || record.status !== 'processing') return - if (record.type === 'video' && !await prepareContinuityTask(record)) return + if (record.type === 'video') { + if (parseTaskParams(record.params).cancellation && parseTaskParams(record.params).firstFrameComposition) { + while (await settleCompositionCancellation(id)==='pending') await new Promise(r=>setTimeout(r,1000)) + return + } + if (!await prepareContinuityTask(record)) return + if (!await prepareComposedFirstFrame(id)) return + } ;[record] = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id, id)) const type = record.type as TaskType const label = taskLabel(type) @@ -410,7 +587,7 @@ async function processTask(id: number, config: AIConfig) { if (type === 'image') { const adapter = getImageAdapter(config.provider) - const resolvedReferenceImages = await normalizeReferenceImages(params.referenceImages) + const resolvedReferenceImages = await normalizeReferenceImages(params.referenceImages,!!params.compositionParentTaskId) ;({ url, method, headers, body } = adapter.buildGenerateRequest(config, { id: record.id, model: record.model, @@ -456,12 +633,18 @@ async function processTask(id: number, config: AIConfig) { const beforeSend = await db.select().from(schema.sysTask).where(eq(schema.sysTask.id, id)).get() if (!beforeSend || beforeSend.status !== 'processing') return + if (parseTaskParams(beforeSend.params).cancellation) return if (params.continuity?.mode === 'first_frame') { const source = await sourceReadiness(params.continuity.sourceId) if (source.state !== 'ready' || source.video !== params.continuity.sourceVideo || source.taskId !== params.continuity.sourceTaskId) { throw new Error('前镜头版本已变化，请重新提交此镜头；尚未发送到显卡') } } + if (type === 'video' && params.firstFrameComposition && record.storyboardId) await assertFrameInputsCurrent(record.storyboardId,params.firstFrameComposition) + // Only this state may have sent an unacknowledged request; never auto-resubmit it on restart. + const claimed = await db.update(schema.sysTask).set({ stage:'submitting', updatedAt:now() }) + .where(and(eq(schema.sysTask.id,id),eq(schema.sysTask.status,'processing'),ne(schema.sysTask.stage,'cancelling'))).returning({id:schema.sysTask.id}) + if (!claimed.length) return const resp = await fetch(url, { method, headers, @@ -712,6 +895,9 @@ async function handleImageCompleteBase64(record: SysTaskRecord, base64Data: stri // 图片完成后回写业务表：分镜(按 frameType)、角色、场景、道具 async function writeBackImageAssets(record: SysTaskRecord, localPath: string) { const params = parseTaskParams(record.params) + // Intermediate frames belong to this video attempt. Only the finished parent + // may publish its final frame; never overwrite character/scene source artwork. + if (params.compositionParentTaskId) return // A late result is kept in task history, never substituted for a newer attempt. const newer = await db.select().from(schema.sysTask).where(and(eq(schema.sysTask.type, 'image'), gt(schema.sysTask.id, record.id), or( eq(schema.sysTask.characterId, record.characterId || -1), eq(schema.sysTask.sceneId, record.sceneId || -1), @@ -747,6 +933,13 @@ async function handleVideoComplete(record: SysTaskRecord, videoUrl: string, dura try { const localPath = record.localPath && fs.existsSync(getAbsolutePath(record.localPath)) ? record.localPath : await downloadVideo(videoUrl, record.id) + const autoCleanup=supportsVideoCleanup(current) + // Keep the original downloaded-byte receipt even if native HD padding is cropped below. + const originalReceipt=autoCleanup ? (record.localPath && params.downloadReceipt ? params.downloadReceipt : await videoReceipt(localPath)) : null + if (originalReceipt) { + params.downloadReceipt=originalReceipt + await db.update(schema.sysTask).set({params:JSON.stringify(params)}).where(eq(schema.sysTask.id,record.id)) + } if (params.variant === 'hd') { const readDimensions = () => new Promise<{ width?: number; height?: number }>((resolve, reject) => { ffmpeg.ffprobe(getAbsolutePath(localPath), (error, meta) => { @@ -775,6 +968,10 @@ async function handleVideoComplete(record: SysTaskRecord, videoUrl: string, dura } await db.update(schema.sysTask).set({ localPath, stage: 'saving', updatedAt: now() }).where(eq(schema.sysTask.id, record.id)) await extractVideoPoster(localPath) + if (autoCleanup && originalReceipt) { + const retained=await verifyRetainedVideo(localPath) + params.remoteCleanup=pendingVideoCleanup(originalReceipt as {bytes:number;sha256:string},retained,videoOutputIndex(videoUrl,current.configBaseUrl!,current.taskId!)) + } // 完成状态与业务回写原子提交；恢复旧任务不能覆盖用户后来生成的视频。 db.transaction(tx => { const newer = record.storyboardId ? tx.select({ id: schema.sysTask.id }).from(schema.sysTask) @@ -782,9 +979,11 @@ async function handleVideoComplete(record: SysTaskRecord, videoUrl: string, dura if (record.storyboardId && !newer && parseTaskParams(record.params).variant !== 'hd') tx.update(schema.storyboards) .set({ videoUrl: localPath, duration: duration || undefined, updatedAt: now() }) .where(eq(schema.storyboards.id, record.storyboardId)).run() - tx.update(schema.sysTask).set({ localPath, status: 'completed', stage: 'completed', errorCode: null, + tx.update(schema.sysTask).set({ localPath, params:JSON.stringify(params), status: 'completed', stage: 'completed', errorCode: null, errorMsg: null, completedAt: now(), updatedAt: now() }).where(eq(schema.sysTask.id, record.id)).run() }) + // Cleanup has its own retry state; it must never invalidate a saved video. + if (params.remoteCleanup) void cleanupSavedVideo(record.id).catch(()=>{}) } catch (error) { await failTask(record.id, safeTaskError(error), 'delivery_failed') } @@ -861,6 +1060,14 @@ export async function resumeInterruptedTasks() { const tasks = candidates.filter(task => task.status === 'processing' || (!task.errorCode && /^Timeout: (polling attempts exhausted|Polling exceeded 10 minutes)/.test(task.errorMsg || '') && isLocalQueuedTask({ provider: task.provider || '', baseUrl: task.configBaseUrl || '' }, task.taskId))) for (const task of tasks) { + const params = parseTaskParams(task.params) + if (task.status === 'processing' && !task.taskId && ( + task.type === 'video' && params.firstFrameComposition && ['preparing_first_frame','first_frame_ready','cancelling','waiting_predecessor','blocked_predecessor'].includes(task.stage || '') + || task.type === 'image' && params.compositionParentTaskId && task.stage === 'prepared')) { + try { const config = await recoveryConfig(task); runBackground(task.id,()=>processTask(task.id,{...config,gpu:params.gpu || config.gpu})) } + catch (error) { await failTask(task.id,safeTaskError(error),'recovery_failed') } + continue + } if (task.type === 'video' && !task.taskId && task.status === 'processing' && ['waiting_predecessor', 'blocked_predecessor'].includes(task.stage || '') && parseTaskParams(task.params).continuity?.mode === 'first_frame') { try { const config = await recoveryConfig(task) @@ -879,7 +1086,7 @@ export async function resumeInterruptedTasks() { // ─── 参考素材归一化 ─────────────────────────────────────────────── -async function normalizeReferenceImages(refs: string[] | null | undefined): Promise { +async function normalizeReferenceImages(refs: string[] | null | undefined, preserveOrder = false): Promise { if (!Array.isArray(refs) || !refs.length) return [] const deduped = Array.from( @@ -890,7 +1097,7 @@ async function normalizeReferenceImages(refs: string[] | null | undefined): Prom ), ) - const normalized = await Promise.all(deduped.map(async (value) => { + const normalized = await Promise.all((preserveOrder ? refs : deduped).map(async (value) => { if (value.startsWith('data:image/')) return value if (value.startsWith('static/') || value.startsWith('/static/')) { const localPath = value.startsWith('/static/') ? value.slice(1) : value diff --git a/huobao-source/backend/src/services/scene-continuity.ts b/huobao-source/backend/src/services/scene-continuity.ts new file mode 100644 index 0000000..ef7d046 --- /dev/null +++ b/huobao-source/backend/src/services/scene-continuity.ts @@ -0,0 +1,93 @@ +/** Shared environment constraints, independent of character identity and shot-to-shot motion. */ +import { createHash } from 'node:crypto' +import { eq } from 'drizzle-orm' +import { db, schema } from '../db/index.js' +import { translateVisualPrompt } from './english-video-prompt.js' + +export interface SceneLock { + version: 1 + sceneId: number + location: string + time: string + environment: string + lighting: string + imageUrl: string + hash: string + source: 'fixed' | 'scene' +} + +export function sceneLockFromScene(scene: typeof schema.scenes.$inferSelect): SceneLock { + // finalPrompt is deliberately excluded: empty-set image prompts prohibit people + // and prescribe a fixed camera, which would contradict a performance shot. + const fields = { version: 1 as const, sceneId: scene.id, location: scene.location.trim(), + time: scene.time?.trim() || '', environment: (scene.continuityPrompt || scene.prompt || '').trim(), + lighting: scene.lighting?.trim() || '', imageUrl: (scene.imageUrl || scene.localPath || '').trim(), + source: scene.continuityPrompt?.trim() ? 'fixed' as const : 'scene' as const } + return { ...fields, hash: createHash('sha256').update(JSON.stringify(fields)).digest('hex') } +} + +export async function resolveSceneLock(storyboardId: number): Promise { + const shot = await db.select().from(schema.storyboards).where(eq(schema.storyboards.id, storyboardId)).get() + if (!shot || shot.deletedAt) throw new Error('分镜不存在') + if (!shot.sceneId) throw new Error('请先为此分镜绑定场景，避免模型随机重画背景') + const episode = await db.select().from(schema.episodes).where(eq(schema.episodes.id, shot.episodeId)).get() + const scene = await db.select().from(schema.scenes).where(eq(schema.scenes.id, shot.sceneId)).get() + if (!episode || episode.deletedAt || !scene || scene.deletedAt || scene.dramaId !== episode.dramaId) { + throw new Error('分镜绑定的场景已失效，请重新绑定当前项目的场景') + } + const lock = sceneLockFromScene(scene) + if (!lock.environment || lock.environment === lock.location && !lock.lighting && !lock.imageUrl) { + throw new Error('场景只有名称，请先补充空间、布景与灯光描述，或上传场景参考图') + } + return lock +} + +export async function assertSceneLockCurrent(id: number, lock: SceneLock) { + if ((await resolveSceneLock(id)).hash !== lock.hash) throw new Error('准备期间场景设定或绑定已变化，请重新提交；未替换原任务') +} + +const referenceKey = (url: string) => url.replace(/^https?:\/\/(?:localhost|127\.0\.0\.1)(?::\d+)?(?=\/static\/)/i, '').replace(/^\/(?=static\/)/, '') +export function addSceneReference(lock: SceneLock, references: string[] = [], limit = 9) { + // Append only. Reordering/deduplicating the existing list corrupts @图片N mappings. + const urls = [...references] + if (urls.length > limit) throw new Error(`所选模型最多支持 ${limit} 张参考图，当前 ${urls.length} 张；请减少参考图或选择多图模型`) + let index = lock.imageUrl ? urls.findIndex(url => referenceKey(url) === referenceKey(lock.imageUrl)) : -1 + if (lock.imageUrl && index < 0) { + if (urls.length >= limit) throw new Error(`场景参考图尚未包含，所选模型最多支持 ${limit} 张参考图；请减少其他参考图或使用包含场景的首帧`) + index = urls.length + urls.push(lock.imageUrl) + } + return { urls, index: index < 0 ? null : index + 1 } +} + +const translations = new Map>() +async function englishEnvironment(lock: SceneLock) { + const source = [`Set: @${lock.location}`, `Time: ${lock.time || 'as established in the scene reference'}`, + `Environment: ${lock.environment}`, `Lighting: ${lock.lighting || 'preserve the established scene lighting'}`].join('\n') + if (!translations.has(lock.hash)) { + const job = translateVisualPrompt(source, [lock.location]) + translations.set(lock.hash, job) + void job.catch(() => { if (translations.get(lock.hash) === job) translations.delete(lock.hash) }) + if (translations.size > 100) translations.delete(translations.keys().next().value!) + } + return translations.get(lock.hash)! +} + +export const stripSceneLock = (prompt: string) => prompt.replace(/\s*\[HB_SCENE_LOCK_V1\][\s\S]*?\[\/HB_SCENE_LOCK_V1\]/g, '').trim() + +export async function sceneLockPrompt(prompt: string, lock: SceneLock, options: { english?: boolean; referenceIndex?: number | null; frame?: boolean } = {}) { + const environment = options.english ? await englishEnvironment(lock) + : [`场景：@${lock.location}`, `时间：${lock.time}`, `固定环境：${lock.environment}`, `光源与色调：${lock.lighting}`].filter(line => !line.endsWith('：')).join('\n') + const reference = options.frame + ? 'Use the supplied first frame / original-video frame as the visual environment anchor; continue its physical set.' + : options.referenceIndex + ? `Reference image ${options.referenceIndex} defines the ENVIRONMENT, not character identity. Preserve its architecture, materials, furniture and relative object positions. Character reference backgrounds must NOT replace this set.` + : 'No scene image is supplied. Use the written environment consistently; do not invent a different room.' + return `${stripSceneLock(prompt)}\n\n[HB_SCENE_LOCK_V1]\nEnvironment continuity — visual directions only, never dialogue, narration or on-screen text.\n${environment}\n${reference}\nApply this environment to EVERY shot and time segment, including close-ups and reverse angles. Keep the same room geometry, doors, windows, wall materials, fixed furnishings, key prop positions and light sources. A tighter crop or changed viewpoint does not change the physical set; do not force off-screen objects into frame. Use this bound scene rather than the previous scene when cutting to a new location. Explicit scripted actions may move props or change lights; preserve the resulting state, do not reset it. Keep the scripted characters and actions; an empty scene reference does not mean removing the actors.\n[/HB_SCENE_LOCK_V1]` +} + +export const sceneContinuityInstructions = ` +场景一致性：read_storyboard_context 中 scenes 的 environment_lock 是本场景共用的固定设定（来源是 continuity_prompt，未设置时取 prompt）；lighting 固定光源与色调。每个分镜必须绑定正确 scene_id；没有对应场景时说明需要先补充场景，不要拿人物参考背景或上一场景凑数。 +生成 description、video_prompt 和 video_prompt_en 时，每条分镜都写入当前场景的空间结构、门窗/墙面、关键陈设相对位置、时间与光源等已知信息，不能仅在标题/信息头写场景名，也不能只写人物表演。每个子镜头/时间段写出该景别可见的背景锚点；特写只描述合理可见部分，保持物理位置而非所有视角的屏幕左右相同。不得臆造未提供的房间布局、移动固定家具或用氛围词替换已有光源。角色三视图/LoRA 只用于人物身份，场景图用于环境。 +同一 scene_id（包括跨集复用）共用固定设定；不同 scene_id 表示新场景，不把旧房间复制过去。道具移动、灯光开关和破损等剧情变化记入 continuity_start/continuity_end 并承接，不随切镜复位。不要将空场景生图的“无人/固定机位”规则带入有人物表演的视频。已有明确地点冲突应报告修正绑定，不能擅自更改剧情。 +画面英文版包含同样的环境约束，中文对白逐字保持。环境约束不作为旁白、对白或字幕。` diff --git a/huobao-source/backend/src/services/storyboard-delete.ts b/huobao-source/backend/src/services/storyboard-delete.ts new file mode 100644 index 0000000..83e8923 --- /dev/null +++ b/huobao-source/backend/src/services/storyboard-delete.ts @@ -0,0 +1,35 @@ +import { and, eq, inArray, isNull } from 'drizzle-orm' +import { db, schema } from '../db/index.js' +import { now } from '../utils/response.js' +import { getStoryboardJob } from './storyboard-job-state.js' + +/** Remove shots from production, keeping task history, bindings and media. */ +export function deleteStoryboards(episodeId: number, input: unknown) { + if (!Number.isSafeInteger(episodeId) || episodeId <= 0 || !Array.isArray(input) || !input.length || input.length > 500 || + input.some(id => !Number.isSafeInteger(id) || id <= 0)) throw new Error('请选择当前集要删除的分镜') + const ids = [...new Set(input as number[])] + if (getStoryboardJob(episodeId)?.status === 'running') throw new Error('正在拆分分镜，请先取消拆分或等待完成后再删除') + return db.transaction(tx => { + const rows = tx.select().from(schema.storyboards).where(inArray(schema.storyboards.id, ids)).all() + if (rows.length !== ids.length || rows.some(row => row.episodeId !== episodeId)) throw new Error('所选分镜不存在或不属于当前集，未删除任何分镜') + const active = rows.filter(row => !row.deletedAt) + const running = tx.select().from(schema.sysTask).where(inArray(schema.sysTask.status, ['processing', 'pending', 'queued', 'running'])).all() + for (const task of running) { + let sourceId: number | undefined + try { sourceId = JSON.parse(task.params || '{}').continuity?.sourceId } catch { /* legacy task */ } + if ((task.storyboardId && ids.includes(task.storyboardId)) || (sourceId && ids.includes(sourceId))) { + throw new Error(`任务 #${task.id} 正在生成或依赖所选分镜，请先取消该任务或等待完成；未删除任何分镜`) + } + } + const ts = now() + if (active.length) { + tx.update(schema.storyboards).set({ deletedAt: ts, updatedAt: ts }) + .where(inArray(schema.storyboards.id, active.map(row => row.id))).run() + const remaining = tx.select().from(schema.storyboards) + .where(and(eq(schema.storyboards.episodeId, episodeId), isNull(schema.storyboards.deletedAt))).all() + tx.update(schema.episodes).set({ duration: Math.ceil(remaining.reduce((sum, row) => sum + (row.duration || 0), 0) / 60), updatedAt: ts }) + .where(eq(schema.episodes.id, episodeId)).run() + } + return { deleted_ids: active.map(row => row.id), count: active.length } + }) +} diff --git a/huobao-source/backend/src/services/storyboard-job-state.ts b/huobao-source/backend/src/services/storyboard-job-state.ts new file mode 100644 index 0000000..a7b360b --- /dev/null +++ b/huobao-source/backend/src/services/storyboard-job-state.ts @@ -0,0 +1,133 @@ +import { createHash, randomUUID } from 'node:crypto' +import { eq } from 'drizzle-orm' +import { db, schema } from '../db/index.js' +import { buildStoryboardPlan, sectionCoverage, coverageAdvanced, resumableDraftPrefix, type StoryboardSection } from '../utils/storyboard-coverage.js' + +export interface StoryboardJob { + id: string; episodeId: number; dramaId: number + status: 'running'|'done'|'error'|'cancelled'; phase: string + startedAt: string; finishedAt?: string; error?: string + completed: number; target: number; targetSeconds: number; seconds: number + steps: number; rejections: number; lastValidation?: string + model?: string; responseModel?: string; configId?: number + stepTrace?: Array<{step:number;tools:string[];finish:string;error?:string;completed:number}> + endingCue?:string; narrativeEndShot?:number + sections?:StoryboardSection[]; sectionIndex?:number; sectionCount?:number + resumedFrom?:string + drafts: any[]; sourceHash: string; rowsHash: string +} +const jobs = new Map() +const controllers = new Map() +const key = (id: number) => `storyboard_job:${id}` +const hash = (v: unknown) => createHash('sha256').update(JSON.stringify(v)).digest('hex') +export function storyboardSource(episodeId: number) { + const ep = db.select().from(schema.episodes).where(eq(schema.episodes.id,episodeId)).get() + const rows = db.select().from(schema.storyboards).where(eq(schema.storyboards.episodeId,episodeId)).all() + const ids = new Set(rows.map(r=>r.id)) + const characters = db.select().from(schema.storyboardCharacters).all().filter(r=>ids.has(r.storyboardId)) + const props = db.select().from(schema.storyboardProps).all().filter(r=>ids.has(r.storyboardId)) + return { ep, sourceHash: hash([ep?.content,ep?.scriptContent,ep?.scriptSourceHash]), rowsHash: hash([rows,characters,props]) } +} +export function persistStoryboardJob(job: StoryboardJob) { + db.insert(schema.appSettings).values({key:key(job.episodeId),value:JSON.stringify(job),updatedAt:new Date().toISOString()}) + .onConflictDoUpdate({target:schema.appSettings.key,set:{value:JSON.stringify(job),updatedAt:new Date().toISOString()}}).run() + jobs.set(job.episodeId,job) +} +export function getStoryboardJob(episodeId: number): StoryboardJob|null { + let job=jobs.get(episodeId) + if (!job) { + const row=db.select().from(schema.appSettings).where(eq(schema.appSettings.key,key(episodeId))).get() + if (!row) return null + try { job=JSON.parse(row.value) } catch { return null } + if (!job) return null + if (job.status==='running') { + job.status='error';job.phase='拆分已中断';job.error='程序重启中断了拆分，原分镜保留，请重新拆分。';job.finishedAt=new Date().toISOString() + persistStoryboardJob(job) + } else jobs.set(episodeId,job) + } + return job +} +export function publicStoryboardJob(job: StoryboardJob|null) { + if (!job) return null + const {drafts,sourceHash,rowsHash,rejections,stepTrace,endingCue,sections,...view}=job + if(job.status==='done'&&!sections&&job.targetSeconds&&job.secondssum+(s.duration||0),0) + job.resumedFrom=previous.id;job.phase=`已保留 ${job.completed} 个分镜，从第 ${job.sectionIndex+1} 场继续拆分` + } + job.targetSeconds=job.sections.reduce((sum,s)=>sum+s.targetSeconds,0);job.target=Math.round(job.targetSeconds/12) + if(previous) db.insert(schema.appSettings).values({key:`storyboard_job_archive:${previous.id}`,value:JSON.stringify(previous),updatedAt:new Date().toISOString()}).onConflictDoNothing().run() + const controller=new AbortController();controllers.set(job.id,controller);persistStoryboardJob(job) + return {job,created:true,controller} +} +export function assertStoryboardJob(episodeId: number, id: string) { + const job=jobs.get(episodeId) + if (!job || job.id!==id || job.status!=='running' || controllers.get(id)?.signal.aborted) throw new Error('本次拆分已结束或取消，迟到的结果不会覆盖分镜') + return job +} +export function updateStoryboardJob(episodeId: number, id: string, changes: Partial) { + const job=assertStoryboardJob(episodeId,id);Object.assign(job,changes);persistStoryboardJob(job);return job +} +export function finishStoryboardJob(job: StoryboardJob, status: StoryboardJob['status'], message?: string) { + if (job.status!=='running') return + job.status=status;job.finishedAt=new Date().toISOString();job.phase=status==='done'?'分镜拆分完成':status==='cancelled'?'已取消拆分':'拆分未完成' + if (message) job.error=message.slice(0,800) + if (status!=='done') controllers.get(job.id)?.abort(new Error(message||'已取消拆分')) + persistStoryboardJob(job);controllers.delete(job.id) +} +export function rejectStoryboardBatch(episodeId:number,id:string,message:string) { + const job=assertStoryboardJob(episodeId,id) + job.rejections++;job.lastValidation=message.slice(0,800);job.phase='正在修正分镜校验问题';persistStoryboardJob(job) + if (job.rejections>=3) { finishStoryboardJob(job,'error',`同一批分镜连续三次校验未通过：${message}。原分镜已保留。`);throw new Error(job.error) } +} +export function stageStoryboardBatch(episodeId:number,id:string,items:any[],replace:boolean) { + const job=assertStoryboardJob(episodeId,id) + // A model may repeat replace_existing at the start of each scene. Tracked + // calls always upsert private drafts; replacement happens only at commit. + const byNumber=new Map(job.drafts.map(s=>[s.shot_number,s])) + for (const item of items) byNumber.set(item.shot_number,{...item,source_section:job.sectionIndex||0}) + const candidate=[...byNumber.values()].sort((a,b)=>a.shot_number-b.shot_number) + const index=job.sectionIndex||0,section=job.sections?.[index] + const current=job.drafts.filter(s=>s.source_section===index) + const coverage=section?sectionCoverage(section,candidate.filter(s=>s.source_section===index)):null + const previousCoverage=section?sectionCoverage(section,current):null + if(coverage&&previousCoverage&&!coverageAdvanced(previousCoverage,coverage)) { + const missingIds=new Set(previousCoverage.missingDialogue.map(p=>p.id)) + const repair=current.filter(s=>s.source_ids?.some((id:string)=>missingIds.has(id))).map(s=>s.shot_number) + throw new Error(`本批未减少缺失原文或对白，已拦截重复分镜。请修改已有镜头 ${repair.join('、')||current.map(s=>s.shot_number).join('、')}，不要追加重复内容；使用明确的“角色说：「原文」”格式。`) + } + job.drafts=candidate;job.completed=job.drafts.length;job.seconds=job.drafts.reduce((sum,s)=>sum+(s.duration||10),0) + if(coverage?.ready) job.sectionIndex=index+1 + if(job.sections?.length && job.sectionIndex===job.sections.length) job.narrativeEndShot=job.drafts.at(-1)?.shot_number + job.rejections=0;delete job.lastValidation;job.phase=`已暂存 ${job.completed} 个分镜 · 已完成 ${job.sectionIndex||0}/${job.sectionCount||1} 个场次` + persistStoryboardJob(job) + const missingIds=new Set(coverage?.missingDialogue.map(p=>p.id)||[]) + const repair=job.drafts.filter(s=>s.source_section===index&&s.source_ids?.some((id:string)=>missingIds.has(id))).map(s=>s.shot_number) + return {count:items.length,total_duration:job.seconds,target_total_seconds:job.targetSeconds,section_complete:!!coverage?.ready,script_complete:!!job.narrativeEndShot,coverage,repair_shot_numbers:repair,message:`已暂存 ${job.completed} 个双语分镜，共 ${job.seconds} 秒。${coverage?.ready?'当前场次已完整覆盖，立即结束本轮，下一场由系统继续安排。':'当前场次尚未完成：缺失对白请修改 repair_shot_numbers 对应已有镜头，原位重存，不要追加重复剧情；缺失来源或时长不足才继续细分原有动作/对白。'}不要跳到后面的场次或结尾，不要压缩对白、删词或加速念完。后续每批最多 4 条并省略 replace_existing。`} +} +export function assertStoryboardSourceUnchanged(job: StoryboardJob) { + const current=storyboardSource(job.episodeId) + if (current.sourceHash!==job.sourceHash || current.rowsHash!==job.rowsHash) throw new Error('拆分期间剧本或分镜已修改，本次结果未覆盖新内容，请重新拆分。') +} diff --git a/huobao-source/backend/src/services/storyboard-jobs.ts b/huobao-source/backend/src/services/storyboard-jobs.ts new file mode 100644 index 0000000..24f4d22 --- /dev/null +++ b/huobao-source/backend/src/services/storyboard-jobs.ts @@ -0,0 +1,91 @@ +import { mastra } from '../mastra/index.js' +import { buildAgentRequestContext } from '../agents/context.js' +import { storyboardTools } from '../agents/tools/storyboard-tools.js' +import { isScriptStale } from '../utils/script-source.js' +import { sectionCoverage } from '../utils/storyboard-coverage.js' +import { assertStoryboardJob, createStoryboardJob, finishStoryboardJob, getStoryboardJob, persistStoryboardJob, publicStoryboardJob, storyboardSource } from './storyboard-job-state.js' + +export function startStoryboardJob(body: {episode_id:number;drama_id:number;message:string;model?:string;config_id?:number;resume_from?:string}) { + const source=storyboardSource(body.episode_id) + if (!source.ep || source.ep.dramaId!==body.drama_id) throw new Error('当前集不存在或不属于该项目') + if (isScriptStale(source.ep)) throw new Error('格式化剧本已过期，请先重新 AI 改写或跳过改写。') + if (!(source.ep.scriptContent||source.ep.content)?.trim()) throw new Error('请先保存剧本内容') + const {job,created,controller}=createStoryboardJob(body.episode_id,body.drama_id,body.resume_from) + if (!created) return publicStoryboardJob(job) + job.model=body.model;job.configId=body.config_id;persistStoryboardJob(job) + const requestContext=buildAgentRequestContext({episodeId:job.episodeId,dramaId:job.dramaId,modelOverride:body.model,textConfigId:body.config_id}) + requestContext.set('storyboardJobId' as never,job.id as never) + const instruction=body.message+`\n系统已按原文划分场次。本轮只拆 readStoryboardContext 返回的当前场次，不能自行跳场。每次 save_storyboards 最多 4 个分镜；每个分镜填写 source_ids，逐项覆盖 source_units。长对白按 4.5 汉字/秒安排时间，8–15 秒装不下就拆成多个镜头；一句可以跨镜头，原文不得删减。只有全任务第一批使用 replace_existing=true，后续按 next_shot_number 追加；如校验失败，修正当前批次，不要跳过。收到 section_complete 后结束本轮，下一场由系统调用。` + const timeout=Number(process.env.HUOBAO_STORYBOARD_TIMEOUT_MS||Math.max(900000,Math.min(3600000,(job.sectionCount||1)*180000))) + const timer=setTimeout(()=>finishStoryboardJob(job,'error',`拆分超过 ${Math.round(timeout/60000)} 分钟仍未完成，已停止等待，原分镜保留。请重试或切换文本模型。`),timeout) + timer.unref() + void (async()=>{ + let previousDrafts='',noProgress=0 + const cancelled=new Promise((_,reject)=>{ + controller.signal.addEventListener('abort',()=>reject(controller.signal.reason||new Error('已停止拆分')),{once:true}) + }) + try { + for(let sectionIndex=job.sectionIndex||0;sectionIndex<(job.sectionCount||0);sectionIndex++) { + noProgress=0 + requestContext.set('storyboardSectionIndex' as never,sectionIndex as never) + for(let attempt=0;attempt<2 && job.sectionIndex===sectionIndex;attempt++) { + assertStoryboardJob(job.episodeId,job.id) + const generate=mastra.getAgent('storyboard_breaker').generate([{role:'user',content:instruction}],{ + requestContext,abortSignal:controller.signal,toolCallConcurrency:1, + activeTools:['readStoryboardContext','saveStoryboards'], + // Mastra's maxSteps overrides stopWhen; keep both limits in one condition. + stopWhen:({steps}:{steps:unknown[]})=>(job.sectionIndex||0)>sectionIndex||steps.length>=12, + modelSettings:{maxRetries:0}, + onStepFinish:(step:any)=>{ + if (job.status!=='running') return + job.steps++ + const reason=step?.finishReason + const results=(step?.toolResults||[]).map((r:any)=>r.payload||r) + const names=(step?.toolCalls||[]).map((r:any)=>(r.payload||r).toolName).filter(Boolean) + const errors=results.map((r:any)=>{ + let value=r.result + if (typeof value==='string') { try { value=JSON.parse(value) } catch {} } + return value?.error || (r.isError ? (typeof value==='string'?value:value?.message||'分镜工具执行失败') : '') + }).filter(Boolean).map(String) + if (errors.length) { job.lastValidation=errors.join('\n').slice(0,800);job.phase='正在修正分镜保存问题' } + if (reason==='length') job.lastValidation='本批输出达到模型长度上限，请减少本批分镜数量。' + if (step.response?.modelId) job.responseModel=step.response.modelId + job.stepTrace=[...(job.stepTrace||[]),{step:job.steps,tools:names,finish:reason||'',error:errors.join('\n').slice(0,800)||undefined,completed:job.completed}].slice(-12) + const nextDrafts=JSON.stringify(job.drafts) + if (nextDrafts!==previousDrafts) noProgress=0 + else if (job.steps>1 && !names.includes('readStoryboardContext')) noProgress++ + previousDrafts=nextDrafts + persistStoryboardJob(job) + if (noProgress>=3) finishStoryboardJob(job,'error',`连续三轮未能保存新的分镜，已停止重复请求。${job.lastValidation||'模型没有调用保存工具，请重试或切换文本模型。'} 原分镜保留。`) + }, + }) + // Reject promptly even if a provider ignores AbortSignal. Tools recheck the + // job identity after awaited translation and before any commit. + await Promise.race([generate,cancelled]) + } + if(job.sectionIndex===sectionIndex) { + const coverage=sectionCoverage(job.sections![sectionIndex],job.drafts.filter(s=>s.source_section===sectionIndex)) + throw new Error(`第 ${sectionIndex+1} 场尚未完成：缺少 ${coverage.missing.length} 段原文、${coverage.missingDialogue.length} 处对白。本场已暂存 ${coverage.shots} 个镜头、${coverage.seconds} 秒。暂存结果已保留，可从未完成场次继续。`) + } + } + assertStoryboardJob(job.episodeId,job.id) + if (!job.drafts.length) throw new Error(job.lastValidation||'模型结束了回复，但没有生成可保存的分镜，请重试。') + if (job.drafts.some((s,i)=>s.shot_number!==i+1)) throw new Error('分镜编号不连续，未覆盖原分镜，请重新拆分。') + if(job.sectionIndex!==job.sectionCount) throw new Error('还有未拆分的场次，不能标记整集完成') + if (job.targetSeconds && job.seconds => { try { return JSON.parse(raw || '{}') } catch { return {} } } + +/** Only our authenticated local bridges implement a download acknowledgement. */ +export function supportsVideoCleanup(record: {provider?:string|null; configBaseUrl?:string|null; taskId?:string|null}) { + if (!['minimax','localvideo'].includes(record.provider || '') || !record.taskId) return false + try { + const url=new URL(record.configBaseUrl || '') + return url.protocol==='http:' && url.hostname==='192.168.110.111' && ['18792','18794'].includes(url.port) && !url.username && !url.password + } catch { return false } +} +export function videoOutputIndex(url: string, base: string, taskId: string) { + const parsed=new URL(url), origin=new URL(base) + const parts=parsed.pathname.split('/').filter(Boolean) + if (parsed.origin!==origin.origin || parts[0]!=='media' || decodeURIComponent(parts[1] || '')!==taskId || !/^\d+$/.test(parts[2] || '')) throw new Error('视频来源与原任务不匹配，未清理服务器文件') + return Number(parts[2]) +} +async function localVideoFile(relative:string) { + if (!/^static\/videos\/[^/\\]+\.(mp4|webm)$/i.test(relative)) throw new Error('本地视频路径无效，未清理服务器文件') + const root=await fs.promises.realpath(STORAGE_ROOT) + const file=await fs.promises.realpath(path.join(root,relative.slice(7))) + if (!file.startsWith(root+path.sep)) throw new Error('本地视频超出素材目录，未清理服务器文件') + return file +} +export async function videoReceipt(relative:string):Promise { + const file=await localVideoFile(relative), digest=createHash('sha256') + let bytes=0 + for await (const chunk of fs.createReadStream(file)) { digest.update(chunk); bytes+=chunk.length } + if (!bytes) throw new Error('本地视频为空，未清理服务器文件') + return {bytes,sha256:digest.digest('hex')} +} +/** Decode every video/audio packet, then flush and fingerprint the retained file. */ +export async function verifyRetainedVideo(relative:string):Promise { + const file=await localVideoFile(relative), suite=await checkFfmpegSuite() + if (!suite.ffmpeg || !suite.ffprobe) throw new Error('视频校验工具不可用，服务器原文件保持不变') + await new Promise((resolve,reject)=>ffmpeg.ffprobe(file,(error,metadata)=>{ + const video=metadata?.streams?.find(s=>s.codec_type==='video') + if (error || !video?.width || !video.height || !(Number(metadata.format.duration)>0)) reject(new Error('视频结构或时长无效，服务器原文件保持不变')) + else resolve() + })) + await new Promise((resolve,reject)=>{ + const command=ffmpeg(file).inputOptions(['-v error','-xerror']).outputOptions(['-map 0:v:0','-map 0:a?']).format('null').output('-') + const timer=setTimeout(()=>{command.kill('SIGKILL');reject(new Error('视频完整性校验超时，服务器原文件保持不变'))},180_000) + command.on('end',()=>{clearTimeout(timer);resolve()}).on('error',()=>{clearTimeout(timer);reject(new Error('视频解码校验失败，服务器原文件保持不变'))}).run() + }) + const handle=await fs.promises.open(file,'r+') + try { await handle.sync() } finally { await handle.close() } + return videoReceipt(relative) +} +export function pendingVideoCleanup(receipt:VideoReceipt,localReceipt:VideoReceipt,index:number):CleanupState { + return {version:1,status:'pending',receipt,localReceipt,index,attempts:0} +} +const active=new Set() +export async function cleanupSavedVideo(id:number) { + if (active.has(id)) return + active.add(id) + try { + const record=await db.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + const params=paramsOf(record?.params), state=params.remoteCleanup as CleanupState|undefined + if (!record || record.type!=='video' || record.status!=='completed' || !record.localPath || !state || state.status==='cleaned' || !supportsVideoCleanup(record)) return + if (state.nextAttemptAt && Date.parse(state.nextAttemptAt)>Date.now()) return + try { + const receipt=await videoReceipt(record.localPath) + if (receipt.sha256!==state.localReceipt.sha256 || receipt.bytes!==state.localReceipt.bytes) throw new Error('本地视频已改变，暂停发送新的清理确认') + const config=record.configId ? await getConfigById(record.configId) : null + if (!config || config.baseUrl!==record.configBaseUrl || config.provider!==record.provider) throw new Error('原服务配置不可用，等待恢复后清理') + const response=await fetch(`${config.baseUrl.replace(/\/$/,'')}/v2/video_generation/${encodeURIComponent(record.taskId!)}/cleanup`,{ + method:'POST',headers:{Authorization:`Bearer ${config.apiKey}`,'Content-Type':'application/json'}, + body:JSON.stringify({version:1,index:state.index,...state.receipt}),signal:AbortSignal.timeout(20_000)}) + if (!response.ok) throw new Error(`服务器清理确认暂不可用（HTTP ${response.status}）`) + const result=await response.json() as any + if (result.status==='cleaned' && result.sha256===state.receipt.sha256 && result.bytes===state.receipt.bytes && result.index===state.index) { + state.status='cleaned';state.completedAt=new Date().toISOString();delete state.lastError;delete state.nextAttemptAt + } else if (result.status==='pending') { + state.nextAttemptAt=new Date(Date.now()+60_000).toISOString() + if (result.retrying) state.lastError='服务器清理暂未成功，将自动重试' + else delete state.lastError + } + else throw new Error('服务器尚未确认清理，保留本地视频并等待重试') + } catch(error) { state.lastError=safeTaskError(error);state.nextAttemptAt=new Date(Date.now()+Math.min(15,2**Math.min(state.attempts,4))*60_000).toISOString() } + state.attempts++ + db.transaction(tx=>{ + const latest=tx.select().from(schema.sysTask).where(eq(schema.sysTask.id,id)).get() + if (latest?.status==='completed') tx.update(schema.sysTask).set({params:JSON.stringify({...paramsOf(latest.params),remoteCleanup:state})}).where(eq(schema.sysTask.id,id)).run() + }) + } finally { active.delete(id) } +} +let timer:ReturnType|undefined +export function resumeVideoCleanups() { + const scan=async()=>{ + const records=await db.select().from(schema.sysTask).where(and(eq(schema.sysTask.type,'video'),eq(schema.sysTask.status,'completed'))) + // Only acknowledged downloads participate. Never sweep old completed tasks. + for (const record of records) if (paramsOf(record.params).remoteCleanup?.status==='pending') await cleanupSavedVideo(record.id) + } + void scan().catch(()=>{}) + if (!timer) { timer=setInterval(()=>{void scan().catch(()=>{})},60_000);timer.unref() } +} diff --git a/huobao-source/backend/src/services/video-download.ts b/huobao-source/backend/src/services/video-download.ts index 7588e7a..416f763 100644 --- a/huobao-source/backend/src/services/video-download.ts +++ b/huobao-source/backend/src/services/video-download.ts @@ -52,6 +52,8 @@ export async function downloadVideo(url: string, taskId: number, options: { callback(null, chunk) } }) await pipeline(Readable.fromWeb(resp.body as any), validator, fs.createWriteStream(partial), { signal }) + const length=Number(resp.headers.get('content-length')) + if (length>0 && !resp.headers.get('content-encoding') && bytes!==length) throw new VideoDownloadError('download_incomplete','Video download is incomplete',true) const mp4 = head.length >= 12 && head.toString('ascii', 4, 8) === 'ftyp' const webm = head.length >= 4 && head.readUInt32BE(0) === 0x1a45dfa3 if (!mp4 && !webm) throw new VideoDownloadError('download_format', 'Response is not an MP4 or WebM video') diff --git a/huobao-source/backend/src/utils/spoken-literals.ts b/huobao-source/backend/src/utils/spoken-literals.ts new file mode 100644 index 0000000..44145fb --- /dev/null +++ b/huobao-source/backend/src/utils/spoken-literals.ts @@ -0,0 +1,36 @@ +/** One speech classifier for coverage, duration and screen-text validation. */ +export function isSpeechPrefix(prefix:string) { + const clause=prefix.slice(-160).split(/[。！？\n]/).at(-1)||'' + if (!/[:：]\s*$/.test(clause)) return false + // A named object displaying a quotation is not someone speaking it. + if (/(?:写着|写有|印着|刻着|字幕|文字|标语|门牌|屏幕|显示|字样)[^，,；;：:]{0,35}[:：]\s*$/.test(clause)) return false + return /(?:说|念|喊|吼|叫|问|答|低语|耳语|嘀咕|嘟囔|喃喃|咆哮|呵斥|哭诉|哀嚎|惊呼|叹道|旁白|台词|播报|广播|画外|声音|嗓音|says?|shouts?|screams?|whispers?|dialogue|narrat\w*)[^：:「」“”"]{0,100}[:：]\s*$/i.test(clause) + || /^\s*[@#][\p{L}\p{N}·]+(?:\s*[（(][^）)]*[）)])?\s*[:：]\s*$/u.test(clause) +} + +/** Outermost quotes only. Nested words inside a spoken line count once. */ +export function quotedLiterals(text:string):Array<{text:string;start:number;end:number}> { + const pairs:Record={'「':'」','『':'』','“':'”','"':'"'} + const result:Array<{text:string;start:number;end:number}>=[] + for(let i=0;iisSpeechPrefix(text.slice(0,q.start))).map(q=>q.text) +} + +export function withoutSpokenLiterals(text:string) { + for(const q of quotedLiterals(text).reverse()) { + if(isSpeechPrefix(text.slice(0,q.start)))text=text.slice(0,q.start)+text.slice(q.end) + } + return text +} diff --git a/huobao-source/backend/src/utils/storyboard-checks.ts b/huobao-source/backend/src/utils/storyboard-checks.ts index 9712d18..be2272b 100644 --- a/huobao-source/backend/src/utils/storyboard-checks.ts +++ b/huobao-source/backend/src/utils/storyboard-checks.ts @@ -1,3 +1,5 @@ +import { withoutSpokenLiterals } from './storyboard-generated-prompt.js' + /** * 分镜段落保存前的确定性检查（save_storyboards 调用）。 * 只查能机械判定的三类问题，查出来就把问题清单回给 Agent 让它改了重存： @@ -54,7 +56,7 @@ export function storyboardSegmentProblems(sb: SegmentLike, characters: Array<{ i } for (const [field, text] of [['video_prompt', prompt], ['image_prompt', sb.image_prompt || '']] as const) { - const cleaned = text.replace(NEGATED, '') + const cleaned = withoutSpokenLiterals(text).replace(NEGATED, '') const hit = cleaned.match(SCREEN_TEXT) if (hit) problems.push(`${field} 要求画面里出现文字（「${hit[0]}」）——规则字、系统提示、字幕一律后期叠加，画面只写人物反应，并注明画面中不出现任何文字`) } diff --git a/huobao-source/backend/src/utils/storyboard-coverage.ts b/huobao-source/backend/src/utils/storyboard-coverage.ts new file mode 100644 index 0000000..8a8b67c --- /dev/null +++ b/huobao-source/backend/src/utils/storyboard-coverage.ts @@ -0,0 +1,173 @@ +import { targetEpisodeSeconds } from './script-source.js' +import { normalizeNarrative } from './storyboard-script-ending.js' +import { spokenLiterals } from './spoken-literals.js' +export { spokenLiterals, quotedLiterals } from './spoken-literals.js' + +export interface SourceUnit { id:string; text:string; dialogue?:string; speech_seconds?:number } +export interface StoryboardSection { id:string; title:string; script:string; units:SourceUnit[]; targetSeconds:number; minSeconds:number; minShots:number } +type CoverageShot={shot_number:number;duration?:number;video_prompt?:string;description?:string;source_ids?:string[];source_section?:number} + +/** A deterministic source inventory. Scene headers are boundaries, never hints + * the model may skip. Unformatted scripts use bounded paragraph groups. */ +export function buildStoryboardPlan(script:string):StoryboardSection[] { + const blocks:Array<{title:string;lines:string[]}>=[] + let current={title:'正文',lines:[] as string[]} + for(const line of script.replace(/\r\n/g,'\n').split('\n')) { + if(/^\s*#{1,3}\s*(?:S\d+\b|场(?:次|景)?\s*[一二三四五六七八九十\d]+)/i.test(line)) { + if(current.lines.some(s=>s.trim())) blocks.push(current) + current={title:line.trim(),lines:[]} + } else current.lines.push(line) + } + if(current.lines.some(s=>s.trim())) blocks.push(current) + const bounded:Array<{title:string;text:string}>=[] + for(const block of blocks) { + const paragraphs=block.lines.join('\n').split(/\n\s*\n/).map(s=>s.trim()).filter(Boolean) + let part:string[]=[],size=0 + for(const paragraph of paragraphs) { + if(size>900) { bounded.push({title:block.title,text:part.join('\n\n')});part=[];size=0 } + part.push(paragraph);size+=paragraph.length + } + if(part.length) bounded.push({title:block.title,text:part.join('\n\n')}) + } + return bounded.map((block,index)=>{ + const id=`S${String(index+1).padStart(2,'0')}` + const units=block.text.split(/\n+/).map(s=>s.trim()).filter(Boolean).flatMap((text,i)=>{ + const spoken=text.match(/^[\p{L}\p{N}· ]{1,30}[:：]\s*(?:[（(][^）)]*[）)]\s*)?([^\n]+)$/u) + const unitId=`${id}P${String(i+1).padStart(2,'0')}` + if(!spoken)return [{id:unitId,text}] + const parts=splitSpeech(spoken[1]),prefix=text.slice(0,text.length-spoken[1].length) + return parts.map((part,p)=>({id:parts.length===1?unitId:`${unitId}.${p+1}`,text:prefix+part,dialogue:part,speech_seconds:Math.ceil(speechSeconds(part))})) + }) + const dialogueSeconds=units.reduce((sum,u)=>sum+(u.dialogue?speechSeconds(u.dialogue):0),0) + const targetSeconds=Math.max(targetEpisodeSeconds(block.text),Math.ceil(dialogueSeconds+units.filter(u=>!u.dialogue).length)) + const minSeconds=Math.max(Math.ceil(targetSeconds*.8),Math.ceil(dialogueSeconds)) + return {id,title:block.title,script:block.title+'\n\n'+units.map(u=>u.text).join('\n\n'),units,targetSeconds,minSeconds,minShots:Math.max(1,Math.ceil(minSeconds/12))} + }) +} + +// Chinese: 4.5 characters/sec; other languages: 2.5 words/sec. This is a +// configurable writing pace estimate, not a speech synthesizer measurement. +export function speechSeconds(text:string) { + const chinese=(text.match(/\p{Script=Han}/gu)||[]).length + const other=(text.replace(/\p{Script=Han}/gu,' ').match(/[\p{L}\p{N}]+/gu)||[]).length + return chinese/4.5+other/2.5 +} + +/** Split long original speech without rewriting a character. Prefer a nearby + * clause boundary; the concatenated parts equal the exact original string. */ +export function splitSpeech(text:string):string[] { + const parts:string[]=[] + let remaining=text + while(speechSeconds(remaining)>10) { + let end=0,boundary=0 + for(const char of remaining) { + if(speechSeconds(remaining.slice(0,end+char.length))>10)break + end+=char.length + if(/[，,。.!！?？；;\s]/.test(char))boundary=end + } + const cut=boundary>=end/2?boundary:Math.max(1,end) + parts.push(remaining.slice(0,cut));remaining=remaining.slice(cut) + } + if(remaining)parts.push(remaining) + return parts +} + +const timeMarkers=(prompt:string)=>[...prompt.matchAll(/^[ \t]*(\d+(?:\.\d+)?)\s*[-–—~～至到]\s*(\d+(?:\.\d+)?)\s*(?:秒|s(?![a-z])|sec)\s*[:：]/gim)] + +/** Correct small timing errors in generated drafts, in both languages together. + * Never remove speech/actions, change order, or exceed the shot length limit. */ +export function fitDraftDialogueTiming(shot:T):T { + const prompt=shot.video_prompt||'',markers=timeMarkers(prompt) + if(!markers.length)return shot + let originalCursor=0 + for(const m of markers) { if(Number(m[1])!==originalCursor||Number(m[2])<=originalCursor)return shot;originalCursor=Number(m[2]) } + if(originalCursor!==(shot.duration||0))return shot + const minimum=markers.map((m,i)=>{ + const part=prompt.slice(m.index!+m[0].length,markers[i+1]?.index??prompt.length) + return Math.max(1,Math.ceil(spokenLiterals(part).reduce((sum,s)=>sum+speechSeconds(s),0))) + }) + const needed=minimum.reduce((a,b)=>a+b,0) + if(needed>15||minimum.every((s,i)=>s<=Number(markers[i][2])-Number(markers[i][1])))return shot + const duration=Math.max(8,shot.duration||0,needed) + if(duration>15)return shot + const lengths=[...minimum];let remaining=duration-needed + for(let i=0;i0;i++) { + const extra=Math.min(remaining,Math.max(0,Number(markers[i][2])-Number(markers[i][1])-lengths[i]));lengths[i]+=extra;remaining-=extra + } + lengths[lengths.length-1]+=remaining + const replace=(text:string,english:boolean)=>{ + const found=timeMarkers(text) + if(found.length!==markers.length||found.some((m,i)=>Number(m[1])!==Number(markers[i][1])||Number(m[2])!==Number(markers[i][2])))return text + let cursor=0,output='',end=0 + found.forEach((m,i)=>{output+=text.slice(end,m.index);const next=cursor+lengths[i];output+=`${cursor}-${next}${english?'s:':'秒：'}`;end=m.index!+m[0].length;cursor=next}) + return output+text.slice(end) + } + return {...shot,duration,video_prompt:replace(prompt,false),video_prompt_en:shot.video_prompt_en?replace(shot.video_prompt_en,true):shot.video_prompt_en} +} + +export function dialogueTimingProblems(shot:CoverageShot):string[] { + const prompt=shot.video_prompt||'' + const markers=timeMarkers(prompt) + const problems:string[]=[] + if(!markers.length) return ['缺少从 0 秒开始的完整时间轴'] + for(let i=0;isum+speechSeconds(s),0) + const available=Number(marker[2])-Number(marker[1]) + if(speech>available+.25) problems.push(`${marker[1]}-${marker[2]} 秒的对白至少需要 ${Math.ceil(speech)} 秒，目前只有 ${available} 秒；拆开长对白，分配更长时间段或更多分镜，不要删词或加速念完`) + } + const duration=shot.duration||0 + if(duration<8||duration>15) problems.push(`段落时长 ${duration} 秒不在 8–15 秒范围内，请细分镜头`) + return problems +} + +export function sectionCoverage(section:StoryboardSection,shots:CoverageShot[]) { + const ids=new Set(shots.flatMap(s=>s.source_ids||[])) + const missing=section.units.filter(u=>!ids.has(u.id)).map(u=>u.id) + const speech=normalizeNarrative(shots.flatMap(s=>spokenLiterals(s.video_prompt||'')).join('')) + let speechCursor=0 + const missingDialogue=section.units.filter(u=>u.dialogue).flatMap(u=>{ + // Allow one original line to span multiple shots, while retaining every + // phrase. Punctuation, quoting and breathing marks may differ. + const phrases=u.dialogue!.split(/[，,。.!！?？；;：:\n]/).map(normalizeNarrative).filter(Boolean) + return phrases.filter(p=>{ + const found=speech.indexOf(p,speechCursor) + if(found<0)return true + speechCursor=found+p.length;return false + }).map(p=>({id:u.id,text:p})) + }) + const seconds=shots.reduce((sum,s)=>sum+(s.duration||0),0) + const ready=!missing.length&&!missingDialogue.length&&seconds>=section.minSeconds&&shots.length>=section.minShots + return {ready,missing,missingDialogue,seconds,minSeconds:section.minSeconds,shots:shots.length,minShots:section.minShots} +} + +/** More duplicate shots are not progress when the same words remain missing. */ +export function coverageAdvanced(before:ReturnType,after:ReturnType) { + return after.ready || after.missing.lengthMath.min(before.seconds,before.minSeconds) + || Math.min(after.shots,after.minShots)>Math.min(before.shots,before.minShots) +} + +/** Reuse only whole, validated scenes before the failed scene. Incomplete + * retries remain archived; never replay their duplicate shots into the result. */ +export function resumableDraftPrefix(sections:StoryboardSection[],drafts:any[],completedSections:number) { + const kept:any[]=[];let sectionIndex=0 + for(let index=0;indexs.source_section===index) + if(!sectionCoverage(sections[index],shots).ready || sectionBatchProblems(sections[index],shots).length + || shots.some((s,i)=>s.shot_number!==kept.length+i+1))break + kept.push(...shots);sectionIndex++ + } + return {drafts:kept,sectionIndex} +} + +export function sectionBatchProblems(section:StoryboardSection,items:CoverageShot[]) { + const allowed=new Set(section.units.map(u=>u.id)) + return items.flatMap(s=>{ + const issues=dialogueTimingProblems(s) + if(!s.source_ids?.length) issues.push('必须填写 source_ids，指明本镜头对应当前场次的哪些原文段落') + for(const id of s.source_ids||[]) if(!allowed.has(id))issues.push(`source_ids ${id} 不属于当前场次 ${section.id}，不能跳场或提前写结尾`) + return issues.map(p=>`段落 ${s.shot_number}：${p}`) + }) +} diff --git a/huobao-source/backend/src/utils/storyboard-generated-prompt.ts b/huobao-source/backend/src/utils/storyboard-generated-prompt.ts new file mode 100644 index 0000000..4a06da0 --- /dev/null +++ b/huobao-source/backend/src/utils/storyboard-generated-prompt.ts @@ -0,0 +1,28 @@ +const QUOTED = /「[^」]*」|『[^』]*』|“[^”]*”|"[^"\n]*"/g +export { withoutSpokenLiterals } from './spoken-literals.js' +const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + +/** Normalize invented voice references only in AI drafts, preserving quotes, + * real uploaded voices and unknown/unbound speakers for explicit validation. */ +export function normalizeDraftVoices(prompt: string | undefined, characters: Array<{id:number;name:string;voiceAudioUrl:string|null}>, boundIds: number[]) { + if (!prompt?.includes('#')) return prompt + const quoted: string[] = [] + let text = prompt.replace(QUOTED, value => `\u0000${quoted.push(value)-1}\u0000`) + const names = characters.map(c => c.name).filter(Boolean).sort((a,b) => b.length-a.length) + if (names.length) text = text.replace(new RegExp(`#(${names.map(escape).join('|')})(?:音色)?`, 'g'), (marker, name) => { + const character = characters.find(c => c.name === name) + return character && boundIds.includes(character.id) && !character.voiceAudioUrl?.trim() ? `@${name}` : marker + }) + return text.replace(/\u0000(\d+)\u0000/g, (_, i) => quoted[Number(i)]) +} + +/** Models sometimes invent category prefixes (@场景房间 / @角色小明). + * Canonicalize only exact existing asset names, outside literal speech. */ +export function normalizeDraftAssets(prompt: string | undefined, names: string[]) { + if (!prompt?.includes('@') || !names.length) return prompt + const quoted:string[]=[] + const text=prompt.replace(QUOTED,value=>`\u0000${quoted.push(value)-1}\u0000`) + const alternatives=[...names].sort((a,b)=>b.length-a.length).map(escape).join('|') + return text.replace(new RegExp(`@(?:场景|角色|人物|道具)(${alternatives})`,'g'),'@$1') + .replace(/\u0000(\d+)\u0000/g,(_,i)=>quoted[Number(i)]) +} diff --git a/huobao-source/backend/src/utils/storyboard-script-ending.ts b/huobao-source/backend/src/utils/storyboard-script-ending.ts new file mode 100644 index 0000000..62ea60d --- /dev/null +++ b/huobao-source/backend/src/utils/storyboard-script-ending.ts @@ -0,0 +1,17 @@ +export const normalizeNarrative = (value:string) => value.replace(/[^\p{L}\p{N}]/gu,'') + +/** Only use a distinctive, unique terminal dialogue as a mechanical stop cue. + * Short utterances and free-form action endings remain model-directed. */ +export function scriptEndingCue(script:string) { + const line=script.trim().split('\n').map(s=>s.trim()).filter(Boolean).at(-1)||'' + const match=line.match(/^[\p{L}\p{N}· ]{1,30}[:：]\s*(?:[（(][^）)]*[）)]\s*)?(.+)$/u) + if (!match) return '' + const cue=normalizeNarrative(match[1]) + const normalized=normalizeNarrative(script) + return cue.length>=16 && normalized.split(cue).length===2 ? cue : '' +} + +export function endingShotIndex(items:Array<{description?:string;video_prompt?:string}>,cue?:string) { + if(!cue) return -1 + return items.findIndex(s=>[s.description,s.video_prompt].some(p=>p&&normalizeNarrative(p).includes(cue))) +} diff --git a/huobao-source/backend/tests/local-video.test.ts b/huobao-source/backend/tests/local-video.test.ts new file mode 100644 index 0000000..34559c9 --- /dev/null +++ b/huobao-source/backend/tests/local-video.test.ts @@ -0,0 +1,23 @@ +import { test } from 'node:test' +import assert from 'node:assert/strict' +import { LocalVideoAdapter } from '../src/services/adapters/local-video.js' +import type { AIConfig, VideoGenerationRecord } from '../src/services/adapters/types.js' +const adapter = new LocalVideoAdapter() +const config = { provider:'localvideo', baseUrl:'http://fixture.invalid', apiKey:'fixture', model:'LTX-2.3', gpu:'gpu3' } as AIConfig +const base = { id:1, prompt:'A girl in a fixed dormitory.', duration:9, resolution:'720p', model:'LTX-2.3' } as VideoGenerationRecord +test('LTX rejects the screenshot failure before a provider job is admitted', () => { + assert.throws(() => adapter.buildGenerateRequest(config, {...base, referenceImageUrls:JSON.stringify(['scene','girl','prop'])}), /LTX-2.3 最多支持 1 张参考图/) + const request = adapter.buildGenerateRequest(config, {...base, referenceImageUrls:JSON.stringify(['scene'])}) + const body = request.body as any + assert.equal(body.model,'LTX-2.3') + assert.equal(body.gpu,'gpu3') + assert.equal(body.content.filter((x:any)=>x.type==='image_url').length,1) +}) +test('local model-specific references do not regress H3 or image-required models', () => { + assert.doesNotThrow(() => adapter.buildGenerateRequest(config, {...base, model:'MiniMax-H3', referenceImageUrls:JSON.stringify(['scene','girl','prop'])})) + assert.doesNotThrow(() => adapter.buildGenerateRequest(config, base)) + for (const model of ['Wan2.2-I2V','MiniMax-H3-R2V-4Step','MiniMax-H3-VDN-H3']) { + assert.throws(() => adapter.buildGenerateRequest(config, {...base,model}), /需要至少 1 张参考图/) + } + assert.throws(() => adapter.buildGenerateRequest(config, {...base,referenceAudioUrls:'["voice"]'}), /暂不支持参考视频、音色或尾帧/) +}) diff --git a/huobao-source/backend/tests/storyboard-coverage.test.ts b/huobao-source/backend/tests/storyboard-coverage.test.ts new file mode 100644 index 0000000..2e278df --- /dev/null +++ b/huobao-source/backend/tests/storyboard-coverage.test.ts @@ -0,0 +1,72 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import {buildStoryboardPlan,dialogueTimingProblems,sectionBatchProblems,sectionCoverage,spokenLiterals,splitSpeech,speechSeconds,fitDraftDialogueTiming} from '../src/utils/storyboard-coverage.js' + +const dialogue='请先检查每一扇窗户是否关好，再把桌上的图纸按照编号摆放整齐，确认材料数量以后我们才能开始修理屋顶。' +test('long dialogue cannot fit in a four-second timeline slot even with a long total duration',()=>{ + assert.ok(dialogueTimingProblems({shot_number:1,duration:15,video_prompt:`0-4秒：老师说：「${dialogue}」\n4-15秒：大家检查工具。`}).some(s=>s.includes('对白至少需要'))) + assert.deepEqual(dialogueTimingProblems({shot_number:1,duration:15,video_prompt:`0-14秒：老师说：「${dialogue}」\n14-15秒：大家点头。`}),[]) +}) +test('nested quotes counted once and offscreen speech counted as dialogue',()=>{ + assert.deepEqual(spokenLiterals('老师（画外）：「请回答『准备好了』，然后开门。」'),['请回答『准备好了』，然后开门。']) + assert.deepEqual(spokenLiterals('特写：门牌写着「图书室」。'),[]) +}) +test('plan retains each scene and paragraph in order',()=>{ + const plan=buildStoryboardPlan(`## S01 | 工作室 | 白天\n\n甲检查窗户。\n\n甲：（认真）${dialogue}\n\n## S02 | 屋顶 | 白天\n\n乙走上屋顶。\n\n乙：工具已经准备好了，我们现在可以开始修理了。`) + assert.equal(plan.length,2);assert.ok(plan[0].units.length>=3) + assert.equal(plan[0].units.map(u=>u.dialogue||'').join(''),dialogue) + assert.ok(plan[0].minSeconds>=Math.ceil(dialogue.replace(/[^\p{Script=Han}]/gu,'').length/4.5)) + const skipped=sectionBatchProblems(plan[0],[{shot_number:1,duration:12,video_prompt:'0-12秒：修理屋顶。',source_ids:['S02P01']}]) + assert.ok(skipped.some(s=>s.includes('不能跳场'))) +}) +test('ending sentence does not bypass missing source, speech or duration',()=>{ + const [section]=buildStoryboardPlan(`## S01 | 工作室 | 白天\n\n甲检查窗户。\n\n甲：${dialogue}\n\n乙：工具已经准备好了，我们现在可以开始修理了。`) + const rushed=[{shot_number:1,duration:12,source_ids:section.units.map(u=>u.id),video_prompt:'0-12秒：乙说：「工具已经准备好了，我们现在可以开始修理了。」'}] + const coverage=sectionCoverage(section,rushed) + assert.equal(coverage.ready,false);assert.ok(coverage.missingDialogue.length) +}) +test('dialogue can span shots while all source units and minimum duration remain required',()=>{ + const [section]=buildStoryboardPlan(`## S01 | 工作室 | 白天\n\n甲：${dialogue}`) + const [a,b]=dialogue.split('确认') + const shots=[{shot_number:1,duration:12,source_ids:section.units.map(u=>u.id),video_prompt:`0-12秒：甲说：「${a}」`},{shot_number:2,duration:12,source_ids:section.units.map(u=>u.id),video_prompt:`0-12秒：甲说：「确认${b}」`}] + assert.equal(sectionCoverage(section,shots).ready,true) +}) +test('adjacent lines without blank separators still inventory dialogue',()=>{ + const [section]=buildStoryboardPlan('大家来到工作室。\n老师：'+dialogue) + assert.ok(section.units.length>=3);assert.equal(section.units.map(u=>u.dialogue||'').join(''),dialogue) + assert.equal(sectionCoverage(section,[{shot_number:1,duration:15,source_ids:section.units.map(u=>u.id),video_prompt:'0-15秒：大家检查工具。'}]).ready,false) +}) + +test('long speech chunks preserve every character and fit their declared speaking budget',()=>{ + for(const text of [dialogue.repeat(3),'Please check every window and put the drawings in order before we begin repairing the roof. '.repeat(5),'没有标点的长台词'.repeat(20)]) { + const chunks=splitSpeech(text) + assert.ok(chunks.length>1);assert.equal(chunks.join(''),text) + assert.ok(chunks.every(c=>speechSeconds(c)<=10)) + } +}) + +test('retiming preserves all speech and actions with identical Chinese and English time ranges',()=>{ + const cn=`0-2秒：老师打开窗户。\n2-6秒：老师说：「${dialogue}」\n6-9秒：大家点头。` + const en=`0-2s: The teacher opens a window.\n2-6s: The teacher says: 「${dialogue}」\n6-9s: Everyone nods.` + const fixed=fitDraftDialogueTiming({shot_number:1,duration:9,video_prompt:cn,video_prompt_en:en}) + assert.deepEqual(dialogueTimingProblems(fixed),[]) + assert.ok(fixed.duration<=15&&fixed.duration>9) + const strip=(s:string)=>s.replace(/^\d+-\d+(?:秒：|s:)/gm,'') + assert.equal(strip(fixed.video_prompt),strip(cn));assert.equal(strip(fixed.video_prompt_en),strip(en)) + const ranges=(s:string)=>s.match(/^\d+-\d+/gm) + assert.deepEqual(ranges(fixed.video_prompt),ranges(fixed.video_prompt_en)) +}) + +test('retiming never shortens speech to squeeze an oversized line into one shot',()=>{ + const shot={shot_number:1,duration:15,video_prompt:`0-15秒：老师说：「${dialogue.repeat(3)}」`} + assert.deepEqual(fitDraftDialogueTiming(shot),shot) + assert.ok(dialogueTimingProblems(shot).length) +}) + +test('coverage requires repeated speech in original order',()=>{ + const [section]=buildStoryboardPlan('甲：准备好了。\n乙：检查窗户。\n甲：准备好了。') + const shot={shot_number:1,duration:15,source_ids:section.units.map(u=>u.id),video_prompt:'0-15秒：甲说：「准备好了。」乙说：「检查窗户。」'} + assert.equal(sectionCoverage(section,[shot]).ready,false) + shot.video_prompt+='甲说：「准备好了。」' + assert.equal(sectionCoverage(section,[shot]).ready,true) +}) diff --git a/huobao-source/backend/tests/storyboard-generated-prompt.test.ts b/huobao-source/backend/tests/storyboard-generated-prompt.test.ts new file mode 100644 index 0000000..afe608e --- /dev/null +++ b/huobao-source/backend/tests/storyboard-generated-prompt.test.ts @@ -0,0 +1,33 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { normalizeDraftVoices, normalizeDraftAssets } from '../src/utils/storyboard-generated-prompt.js' +import { storyboardSegmentProblems } from '../src/utils/storyboard-checks.js' + +const characters=[{id:1,name:'乔雨灵',voiceAudioUrl:null},{id:2,name:'高天明',voiceAudioUrl:'voice.wav'}, + {id:3,name:'林',voiceAudioUrl:null},{id:4,name:'林婉',voiceAudioUrl:'voice2.wav'}] +test('AI drafts do not require voices for known bound characters without uploaded audio',()=>{ + assert.equal(normalizeDraftVoices('0-3秒：#乔雨灵音色说：「别过来！」',characters,[1]),'0-3秒：@乔雨灵说：「别过来！」') +}) +test('keep real voices, unbound names, and dialogue literals intact',()=>{ + const text='0-3秒：#高天明说：「#乔雨灵，别过来！」 #陌生人 #乔雨灵' + assert.equal(normalizeDraftVoices(text,characters,[2]),text) + assert.equal(normalizeDraftVoices('#林婉音色 #林',characters,[3,4]),'#林婉音色 @林') +}) +test('spoken mention of blood-red text does not request drawing text',()=>{ + const p='0-9秒：@高天明低声念道：「血红文字写着，所有人都得死。」画面中不出现任何文字。' + assert.deepEqual(storyboardSegmentProblems({shot_number:1,duration:9,video_prompt:p,character_ids:[2]},characters),[]) +}) +test('screen text instructions are still rejected, including quoted instructions',()=>{ + for(const p of ['0-9秒：墙面显示血红文字。','0-9秒：屏幕显示：「血红文字」。']) { + assert.ok(storyboardSegmentProblems({shot_number:1,duration:9,video_prompt:p},[]).some(p=>p.includes('画面里出现文字'))) + } +}) +test('timeline and character binding checks still apply',()=>{ + const problems=storyboardSegmentProblems({shot_number:1,duration:12,video_prompt:'0-9秒：@高天明说：「血红文字。」'},characters) + assert.ok(problems.some(p=>p.includes('时间轴'))) + assert.ok(problems.some(p=>p.includes('没有绑定'))) +}) +test('normalize category-prefixed asset references without changing dialogue',()=>{ + assert.equal(normalizeDraftAssets('@场景民国八角戏楼·候场台，@角色乔雨灵说：「@场景民国八角戏楼·候场台」',['民国八角戏楼·候场台','乔雨灵']), + '@民国八角戏楼·候场台，@乔雨灵说：「@场景民国八角戏楼·候场台」') +}) diff --git a/huobao-source/backend/tests/storyboard-script-ending.test.ts b/huobao-source/backend/tests/storyboard-script-ending.test.ts new file mode 100644 index 0000000..ada0337 --- /dev/null +++ b/huobao-source/backend/tests/storyboard-script-ending.test.ts @@ -0,0 +1,13 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import {scriptEndingCue,endingShotIndex} from '../src/utils/storyboard-script-ending.js' +test('identify unique long terminal speech and ignore punctuation differences',()=>{ + const cue=scriptEndingCue('前情。\n高天明：（低声）乔雨灵，信我一次！按剧本喊出来！剩下的交给老子！') + assert.ok(cue) + assert.equal(endingShotIndex([{description:'前半段'},{video_prompt:'0-9秒：@高天明说：「乔雨灵，信我一次，按剧本喊出来！剩下的交给老子！」'},{description:'不应生成的新剧情'}],cue),1) +}) +test('do not prematurely stop on repeated or short dialogue or an action ending',()=>{ + assert.equal(scriptEndingCue('高天明：好！'),'') + assert.equal(scriptEndingCue('他离开房间，关上门。'),'') + assert.equal(scriptEndingCue('甲：这句对白前面已经出现了很多次一定不能作为终点。\n乙：这句对白前面已经出现了很多次一定不能作为终点。'),'') +}) diff --git a/huobao-source/frontend/app/components/ComposedFirstFrame.vue b/huobao-source/frontend/app/components/ComposedFirstFrame.vue new file mode 100644 index 0000000..116c852 --- /dev/null +++ b/huobao-source/frontend/app/components/ComposedFirstFrame.vue @@ -0,0 +1,27 @@ + + + diff --git a/huobao-source/frontend/app/components/SceneContinuity.vue b/huobao-source/frontend/app/components/SceneContinuity.vue new file mode 100644 index 0000000..d719623 --- /dev/null +++ b/huobao-source/frontend/app/components/SceneContinuity.vue @@ -0,0 +1,60 @@ +