diff --git a/.codex_douyin_login_account1.png b/.codex_douyin_login_account1.png new file mode 100644 index 0000000..db45fce Binary files /dev/null and b/.codex_douyin_login_account1.png differ diff --git a/.codex_douyin_login_account1_latest.png b/.codex_douyin_login_account1_latest.png new file mode 100644 index 0000000..bcee056 Binary files /dev/null and b/.codex_douyin_login_account1_latest.png differ diff --git a/.codex_douyin_login_account1_new.png b/.codex_douyin_login_account1_new.png new file mode 100644 index 0000000..77110e0 Binary files /dev/null and b/.codex_douyin_login_account1_new.png differ diff --git a/.workbuddy/memory/2026-08-27.md b/.workbuddy/memory/2026-08-27.md new file mode 100644 index 0000000..46e94b3 --- /dev/null +++ b/.workbuddy/memory/2026-08-27.md @@ -0,0 +1,221 @@ +# 2026-08-27 工作日志 + +## 抖音私信发送 decision=KICK 修复 + 第二套发送方案 + +### 根因诊断(已完成) +- 现象:账号1(my_uid=2609567359568155)收消息正常(UA=Chrome/148),发消息被安全网关 decision=KICK。 +- 根因:**UA 全链路不一致**——凭证采集时写入 storage_state/im_session_data 的 UA 是 Chrome/148,但 worker 构建 IM 会话时 `_load_user_agent()` 用 `resolve_user_agent` 回退默认 Chrome/120 覆盖了采集 UA。a_bogus 签名、IM 请求头、Protobuf body 用 Chrome/120,浏览器上下文/凭证却是 Chrome/148,设备指纹不一致触发 KICK。 +- 数据库证据:`account.user_agent=None`、`im_session_data.user_agent=Chrome/120`(被覆盖后写入)、`cookie_data.user_agent=Chrome/148`。 + +### 修改(backend/rpa_engine/playwright_worker.py) +1. `__init__` 新增 `self._raw_user_agent = ""`。 +2. 新增 `_load_raw_user_agent()`:读账号表显式配置的 UA(未配置返回空串)。 +3. `_build_im_session_from_storage`:账号表显式配置 UA 时用 `resolve_user_agent(raw_ua)`;否则**保留** storage_state 采集的 UA(不再覆盖),加日志。 +4. 新增 `send_im_via_browser_page(conversation_id, content)`:第二套发送方案——构建 IM session → `DouyinImHttpClient.resolve_conversation_meta` 解析 ticket/short_id → `ProtoBuilder.build_send_message_request` 构造 protobuf → 打开抖音 /message 页面(非 headless 最小化)→ 页面 JS fetch `/v1/message/send`(security-sdk 注入 a_bogus/bd-ticket-guard)→ `analyze_send_response` 判定。受 `KEFU_BROWSER_SEND_TIMEOUT`(默认45s)控制。 +5. `_run_im_direct_service` 注入 `send_fallback=self.send_im_via_browser_page`。 + +### 修改(backend/rpa_engine/douyin_im/service.py) +- `__init__` 新增 `send_fallback` 参数。 +- `_send_text` 失败分支:`last_error` 含 `DECISION=KICK`/`STATUS_CODE=7911`/`INVALID_REQUEST` 且存在 `send_fallback` 时调用兜底;成功则重置 `_session_invalid_strikes/_fired` 并返回 True;失败继续走 `_note_session_invalid` 自动下线。 + +### 方法签名验证(全部通过,py_compile OK) +- `resolve_conversation_meta(auth, conv_id, my_uid, peer_uid) → (conv_id, short_id, ticket)` +- `ProtoBuilder.build_send_message_request(auth, conv_id, short_id, ticket, msg_content, message_type=7)` 静态方法;`build_msg_payload(spec) → (msg_content, message_type)` 展开正确 +- `parse_reply_content(raw) → dict`(含 `type` 键,`"image"` 时兜底拒绝) +- `analyze_send_response(bytes) → dict`(ok/decision/status_code/summary/server_message_id) +- `DouyinAuth.from_im_session` / `ImSession.can_direct_im/cookies/my_uid/user_agent` / `DouyinImHttpClient(session, account_id)` async with 均匹配 + +### 待办/注意 +- 后端当前未运行;下次启动自动生效(无需重启)。 +- 浏览器兜底暂不支持图片回复(返回"请改用纯文字")。 +- 运行时验证未执行:需要账号重新托管后实际触发发送观察;KICK 后兜底页面需等待 security-sdk 加载(15s 轮询)。 + +## 登录态常驻方案(A 保活 + B 自动重登录)✅ 已实施并通过编译/导入验证 + +背景:sessionid/passport 无 refresh token 可自动换新;IM 通道活跃**不刷新** passport 登录态,托管约 30 天失效 → 抖音返回「用户未登录」。 + +### A 保活(backend/rpa_engine/playwright_worker.py) +- 配置:`KEFU_KEEPALIVE_INTERVAL`(默认 21600s=6h,最小 300s)、`KEFU_KEEPALIVE_DISABLED`(设为 1 关闭)。 +- `_keepalive_loop`:每间隔调用 `_keepalive_touch`。 +- `_keepalive_touch`:`controller.browser_slot(account_id,"keepalive")` 串行 → 非 headless 最小化 Chromium → storage_state + `self._user_agent` 建 context → **临时挂载 `(self.playwright/browser/context/page)`** 复用 `_has_visible_login_prompt/_persist_cookies` → goto douyin.com → sleep 6s → 检测登录提示(有则返回失败"将触发自动重登录")→ `_persist_cookies()` → finally 恢复引用并关浏览器。 +- 挂载点:`_run_im_direct_service` run() 前 `_start_keepalive()`,finally 后 `_stop_keepalive()`;`stop()` 开头也 `_stop_keepalive()`(防悬挂)。 + +### B 失效自动重登录(backend/main.py + worker 侧) +- worker:`__init__` 新增 `relogin_hook: Optional[Callable[[int], Awaitable[None]]]`;`on_im_session_invalid` 末尾 await hook(manager 层驱动,避免 worker 自 cancel 扭曲 + browser_slot 竞争)。 +- manager:`_AUTO_RELOGIN_COOLDOWN`(默认 1800s=30min 防抖,env `KEFU_AUTO_RELOGIN_COOLDOWN`);`_schedule_auto_relogin` 快速返回(防抖→查重→create_task);`_auto_relogin_account`:轮询等旧 worker 退出(100×0.2s)→ DB 置 `logging_in`、清 qr/error → `start_worker(login_mode="browser", wait_until_ready=False)` → 浏览器弹码 → 前端账号卡片轮询展示 qr_code_base64 → 扫码自动恢复托管;异常置 offline+error_message。 +- 前端无需改(Accounts.vue 已支持 logging_in + 二维码)。 + +### 验证记录 +- `py_compile playwright_worker.py main.py` → COMPILE_OK。 +- AST 检查修正版(须同时匹配 `ast.AsyncFunctionDef`!)确认:DouyinWorker 含 `_keepalive_interval(sync)/_keepalive_disabled(sync)/_start_keepalive/_stop_keepalive/_keepalive_loop/_keepalive_touch/on_im_session_invalid`(均 async);WorkerManager 含 `_schedule_auto_relogin/_auto_relogin_account`(均 async)。 +- venv 导入冒烟(`.venv/Scripts/python.exe`)→ IMPORT_SMOKE_OK;`DouyinWorker.__init__` 参数含 `relogin_hook`。 + +### 运行时验证(待后端启动后) +1. 托管后 6h 内观察保活日志(打开 douyin.com + 持久化 cookie)。 +2. 手动清 passport cookie 或等失效 → 观察自动重登录链路:offline→logging_in→弹码→扫码→恢复托管;确认 30min 防抖生效(扫码失败不会死循环)。 +3. 唯一绕不开的人工动作是扫码(抖音无免扫码续期通道)。 + +## 二维码截图太小/无法扫描 ✅ 已修复 + +### 问题 +自动重登录时前端弹窗显示的是**整页截图兜底**,二维码被包在整个网页截图中,尺寸过小,手机无法扫描。 + +### 诊断 +- 后端 `_grab_qr_data_url()` 的精确选择器未命中当前 Douyin 二维码元素,导致一直回退到 `page.screenshot()`。 +- 使用 Playwright 诊断脚本 `backend/debug_qr_inspect.py` 观察:headless 环境触发的是验证码 iframe(`lf-rc1.yhgfb-cn-static.com/.../rmc-nocaptcha`),与真实有头浏览器弹出的扫码登录弹窗不同,因此重点改为增强二维码抓取鲁棒性。 + +### 修改(backend/rpa_engine/playwright_worker.py) +1. 新增 `from PIL import Image`、`import io`。 +2. 扩充 `_QR_SELECTORS`:新增登录弹窗内 `img`/`canvas` 选择器(`#login-pannel`、`*login-guide*`、`*login-panel*`、`*account_login*` 等)。 +3. 扩充 `_QR_CONTAINER_SELECTORS` / `_LOGIN_PANEL_SELECTORS`。 +4. `_grab_qr_in_frames`:匹配元素后增加 `_looks_like_qr_box()` 校验(80~600px、长宽比≥0.75);对 `` 优先用 `canvas.toDataURL()` 提取;对 `` 优先读 data-src/http-src;返回前统一经 `_upscale_qr_image()` 放大。 +5. 新增 `_grab_qr_generic_in_frames()`:在所有 frame 中泛化扫描登录容器内最大方型 `img`/`canvas`,作为精确选择器未命中时的兜底。 +6. `_grab_login_panel_shot()`:改为遍历所有 frame(含 iframe),并调用 `_ensure_panel_fits()` 临时放大 viewport 保证截图清晰。 +7. 新增 `_crop_center_viewport_shot()`:截取视口中央 700x800 区域,替代直接整页截图,避免二维码过小。 +8. 新增 `_upscale_qr_image()`:对小于 280px 的二维码用 Pillow 最近邻放大,提高手机扫描成功率。 +9. `_grab_qr_data_url()` 增加分阶段日志;把「中心区域截图」放在「整页截图」之前;整页截图加 15s timeout 防字体加载卡死。 +10. `_check_and_grab_captcha()` 的整页兜底改为先 `_crop_center_viewport_shot()`。 + +### 验证 +- `py_compile rpa_engine/playwright_worker.py` → COMPILE_OK。 +- venv 导入冒烟 → IMPORT_OK;新增方法列表:`_looks_like_qr_box`、`_grab_qr_generic_in_frames`、`_ensure_panel_fits`、`_crop_center_viewport_shot`、`_upscale_qr_image`。 + +### 待验证 +- 重启后端后触发自动重登录/首次托管,观察前端二维码是否清晰可扫;查看日志应出现 `Captured QR via precise selector` / `generic scan` / `login panel screenshot` 等字样,而不是 `fell back to full-viewport screenshot`。 + +## 排查:KICK 是否由代码主动退出登录触发 ✅ 结论:否 + +### 用户疑问 +收到 `decision=KICK` 错误,怀疑代码里有主动退出登录的逻辑。 + +### 排查结果 +全库搜索 `logout / 退出登录 / clear_cookie / sessionid.*None / passport.*logout` 等,**没有发现任何主动调用抖音退出登录接口或自动清空 session cookie 的代码**。 + +- `clear_cookie_file()` 仅在 3 个**手动 API** 中被调用: + - `DELETE /api/accounts/{id}`(删除账号) + - `POST /api/accounts/{id}/clear-cookie`(手动清除 Cookie) + - `_reset_account_credentials`(重置账号凭证接口) +- `on_im_session_invalid()` 只停止 worker、更新 DB 状态为 offline、触发 `relogin_hook` 自动重登录,**不会删除 cookie / session**。 +- `_keepalive_touch()` 仅访问 `https://www.douyin.com/` 并持久化 cookie,不会登出。 + +`decision=KICK` 是**抖音服务端安全网关返回的**,常见原因: +1. passport/sessionid 自然过期; +2. 账号在其它设备/浏览器登录,挤掉当前会话; +3. 设备指纹/签名不一致触发风控; +4. 服务端主动下线。 + +### 同步更新提示文案 +发现 `http_client.py` 和 `service.py` 中 KICK 提示仍写着"请停止托管后用浏览器模式重新登录...",与已实施的自动重登录方案矛盾。已修改为"系统正在自动重登录,请留意账号卡片上的登录二维码并扫码"。 + +### 修改文件 +- `backend/rpa_engine/douyin_im/http_client.py` +- `backend/rpa_engine/douyin_im/service.py` + +### 验证 +- `py_compile` 两个文件 → COMPILE_OK。 + +## UA 全链路一致性修复(接收链路)✅ 已实施并通过编译/导入验证 + +### 需求 +用户明确要求:接收消息与发送消息使用同一 User-Agent,账号配置里改了 UA,发送、接收都要同步生效。 + +### 背景 +- 发送链路上一轮已一致:`_build_im_session_from_storage` 保留采集 UA → `session.user_agent` → `DouyinAuth.from_im_session(session)` 设 `auth.user_agent`。 +- 接收链路存在 2 个硬编码漏网点(Chrome/120 DEFAULT_USER_AGENT)+ 多处无参 `DouyinAuth()` 构造导致 `self.user_agent` 未设置。 + +### 修改 +1. **`douyin_im/auth.py`**: + - `__init__` 增加 `self.user_agent = None`。 + - `perepare_auth` 增加 `user_agent: str = ""` 参数,非空时保存 `self.user_agent`(避免覆盖 from_im_session 已设值)。 + - `query_my_uid()` 的请求头与 `generate_a_bogus` 改用 `ua = self.user_agent or DEFAULT_USER_AGENT`(消除硬编码)。 + - `from_im_session` 在 perepare_auth 时直接传 `user_agent=session.user_agent or DEFAULT_USER_AGENT`。 +2. **`douyin_im/dy_util.py`**:`generate_webid(auth=None, url="", user_agent="")` 新增参数;内部 UA 优先级:显式参数 > `auth.user_agent` > DEFAULT(消除硬编码)。 +3. **调用点全部显式传 UA**(`session.user_agent or DEFAULT_USER_AGENT`): + - `frontier.py fetch_device_id`、`follower_poll.py`、`main.py:2911`(conversations 兜底)、`service.py send_message`(uid 兜底) + - `peer_profile.py _build_auth`、`image_upload.py`、`account_profile.py _build_auth`(传 `ua` 变量) + +### 保留原样(非风险点) +- `image_upload.py:496/599` 的 `DEFAULT_USER_AGENT`:VOD 存储上传(腾讯云 VOD,AWS SigV4/JWT 独立鉴权,非抖音 web API、无 a_bogus),不会触发 7911。 +- `proto_builder.py _ua_headers`、`http_client.py:740`:已走 `auth.user_agent`,自动生效。 +- `device_profiles.py` DEFAULT_USER_AGENT:仅作为无显式配置时的默认 profile。 + +### 验证 +- `py_compile` 9 个文件 → PY_COMPILE_OK。 +- venv 导入冒烟:perepare_auth 签名含 user_agent、设置/不设置路径正确、generate_webid 签名与缓存命中正确 → IMPORT_SMOKE_OK。 +- Grep 确认接收链路(douyin_im)无 `user_agent=DEFAULT_USER_AGENT` / `"User-Agent": DEFAULT_USER_AGENT` 残留。 + +### 运行时验证(待重启后端) +配置账号 UA(account.user_agent)→ 重启后端 → 观察日志确认 a_bogus 签名与请求头 UA 一致(Chrome/148 或其他配置值,而非 Chrome/120)。 + +## 凭证登录头(user_agent)自动回填账号 ✅ 已实施并通过编译/冒烟测试 + +### 需求 +用户问:「获取登录凭证里有没有登录头,如果有的话保存账户的时候直接填上去」。确认采集器导出的 storage_state 顶层带 `user_agent`(如 Chrome/148),保存 Cookie 时之前只存进 `cookie_data`,`account.user_agent` 列不回填。 + +### 修改 +1. **`backend/utils/cookie_store.py`**:新增 `extract_user_agent_from_cookie_data(cookie_data) -> str`——解析 JSON 取顶层 `user_agent`(兼容 DYCRED 的 `ua` 键),无效/缺失返回空串。 +2. **`backend/main.py`**: + - 新增 `_backfill_user_agent_from_cookie(account, standard_json_str)`:仅当 `account.user_agent` 为空且凭证含 UA 时回填;**已有自定义 UA 保持不变**(用户配置优先)。 + - `create_account`(POST /api/accounts)与 `update_account_cookie`(PUT /api/accounts/{id}/cookie)保存 Cookie 后调用回填。 + - `AccountCookieResponse` 增加 `user_agent` / `user_agent_label` 字段,`_build_cookie_response` 填充,前端保存凭证后可确认回填结果。 + +### 验证 +- `py_compile main.py utils/cookie_store.py` → PY_COMPILE_OK。 +- 冒烟测试:storage_state(顶层 user_agent)→ validate_cookie_json 保留 → 提取 Chrome/148 → 空列回填成功;已有自定义 UA 不被覆盖;DYCRED 输入(ua 键)也能提取 → ALL_SMOKE_OK。 + +### 行为说明 +- 保存/更新凭证时自动回填,发送+接收链路经 `_build_account_im_session` 统一走该 UA。 +- 若用户在配置页显式改过 UA,凭证里的登录头不会覆盖它。 +- 清空凭证(DELETE cookie)不会清 `account.user_agent` 列(保留设备指纹配置,可在配置页手动改)。 + +### 待验证 +重启后端 → 保存一份带 user_agent 的凭证 → 观察账号响应中 `user_agent` 已回填(非空)且 `user_agent_label` 正确。 + +## 咨询:能否用抖音 APP 凭证登录 → 结论:不能,已改为优化保活 ✅ + +### 用户提问 +能否搞到抖音 APP 登录凭证(APP sessionid)用来登录/托管。 + +### 结论(已给用户,含对比图) +- APP sessionid 与 Web sessionid 分属不同端,当前架构(frontier WS aid=2906 + web API + a_bogus)只认 Web 登录态 + security-sdk 密钥,APP 凭证直接喂入 → 设备指纹/签名不匹配 → 风控/限号/封号。 +- 获取 APP 凭证本身要 root + 抓包 + frida(ssl pinning 反抓包),且绑定设备指纹,跨环境使用高风控;等于重写整套 APP 协议层,不推荐。 +- 用户动机 = 摆脱频繁扫码/登录失效 → 方向改为优化 Web 保活。 + +### 保活增强(backend/rpa_engine/playwright_worker.py) +1. `_keepalive_touch` 访问目标从首页改为 `https://www.douyin.com/im`(私信页,更贴近真实活跃、触发 IM 域请求),可用 `KEFU_KEEPALIVE_URL` 覆盖;/im 异常时回退首页。 +2. 随机节奏:停留 4~8s + 一次 `mouse.wheel` 滚动,避免固定机械行为。 +3. 新增 `_cookie_expires_map` / `_fmt_expires_map` 静态方法:保活前后读取 sid_guard/sessionid/sid_tt 等 passport cookie 的 expires 并打日志(before/after/renewed),用于实测抖音 Web 是否对持续活跃账号滑动续期。 + +### 验证 +- py_compile → PY_COMPILE_OK。 +- AST + venv 冒烟:辅助方法存在且为 static;expires 提取(含 session cookie -1→0、空值跳过、非 passport 键过滤)、格式化、renewed 判定逻辑全部通过 → KEEPALIVE_SMOKE_OK。 + +### 待观察(重启后端后) +- 保活日志出现 `keepalive passport expires before[...] after[...] renewed=...`:若 renewed 有值 → 保活确实续期,可继续调优频率;若持续 renewed=none → 抖音 Web passport 不滑动续期,30 天到期仍需扫码一次(自动重登录已保证只扫一次)。 + +## KICK 循环修复(发送被踢下线 → 重登 → 又被踢) + +### 根因(实锤) +- `DouyinImSession.from_storage_state` 把新版 `__tea_cache_tokens_6383.user_unique_id`(实际是 web_id,如 7678646545793812008)误当账号 UID → my_uid=web_id,而 device_id 取 `web_runtime_security_uid`(真实 UID)→ **device_id != my_uid** → 安全网关 decision=KICK → 自动重登 → 新旧登录态混合 → 循环。 +- 重登后 cookie 落库仍混合(normalize 只修顶层 my_uid 与 web_runtime_security_uid,不清理 localStorage 混合 tea)。 +- worker 的 `profile_matches_cookie` 要求 profile_updated_at >= cookie_updated_at:资料同步滞后时错误 UID 带病运行。 + +### 修复(4 个文件) +1. `session.py from_storage_state`:改两遍扫描收集(ls_sec_uid/ls_web_id/ls_tea_pairs),my_uid 优先级:extra/顶层 > web_runtime_security_uid > tea(user_unique_id!=web_id 才可信);device_id:extra/cookies > ls_sec_uid > my_uid;末尾加一致性收敛(my_uid 与 device_id 不一致时以 my_uid 为准)。tea 解析对非 dict 值(如 int 1)类型保护。 +2. `cookie_store.py _parse_tea_from_ls`:`if not isinstance(parsed, dict): continue`,修复 `__tea_cache_first_*` 值为 1 时的 AttributeError 崩溃(保存凭证 500)。 +3. `playwright_worker.py _build_im_session_from_storage`:权威 UID 覆盖改为 `accounts.douyin_uid` 无条件优先(拿 cookie 拉的,最可靠),`profile.uid` 仅在资料不早于 cookie 更新时可信(防串号,账号 9 的 profile 表就存了账号 8 的 UID);覆盖 my_uid 时同步 device_id=verified_uid。 +4. `credential.py build_im_session_from_storage`:saved.uid_verified 覆盖 my_uid 时同步 device_id(之前只改 my_uid 不改 device_id,凭证残留设备号导致不一致)。 + +### 数据修复 +- 备份:backend/.bak/kefu*.db.before_uid_fix_20260827_* +- 账号 1:im_session_data.my_uid 7678646545793812008(web_id) → 2609567359568155,uid_verified=True +- 账号 8/9:仅补 uid_verified=True +- 验证:3 账号均 device_id==my_uid==douyin_uid,ALL_CONSISTENT + +### 测试 +- test_sec_user_id_guard + test_im_receive_path 共 52 个用例全过(更新 3 处断言:device_id 同步为新行为;3 处 mock 从 _load_user_agent 改为 _load_raw_user_agent)。 +- 全套 228 个用例剩 6 个失败为**预先存在的测试 mock 签名过期**(get_owned_account 新增 write_permission 参数,测试 mock 未更新),与本次改动无关。 + +### 待观察(重启后端后) +- 账号 1 再发送是否还 KICK(预期不再循环);日志出现 "replaced collected IM uid ..." / "synced device_id" 即覆盖生效。 +- 账号 9 的 localStorage 仍混合(www 域残留 7657072144060941834),等重新登录后由 normalize + 新解析逻辑自然清洗。 diff --git a/backend/.bak/kefu.db.before_uid_fix_20260827_180651 b/backend/.bak/kefu.db.before_uid_fix_20260827_180651 new file mode 100644 index 0000000..f9db025 Binary files /dev/null and b/backend/.bak/kefu.db.before_uid_fix_20260827_180651 differ diff --git a/backend/.bak/kefu1.db.before_uid_fix_20260827_180652 b/backend/.bak/kefu1.db.before_uid_fix_20260827_180652 new file mode 100644 index 0000000..a176b85 Binary files /dev/null and b/backend/.bak/kefu1.db.before_uid_fix_20260827_180652 differ diff --git a/backend/debug_qr/frame0_IMG_16_58.png b/backend/debug_qr/frame0_IMG_16_58.png new file mode 100644 index 0000000..b6fc7af Binary files /dev/null and b/backend/debug_qr/frame0_IMG_16_58.png differ diff --git a/backend/debug_qr/frame0_IMG_184_1013.png b/backend/debug_qr/frame0_IMG_184_1013.png new file mode 100644 index 0000000..9cd505b Binary files /dev/null and b/backend/debug_qr/frame0_IMG_184_1013.png differ diff --git a/backend/debug_qr/frame0_IMG_184_412.png b/backend/debug_qr/frame0_IMG_184_412.png new file mode 100644 index 0000000..d61a4c5 Binary files /dev/null and b/backend/debug_qr/frame0_IMG_184_412.png differ diff --git a/backend/debug_qr/frame0_IMG_184_717.png b/backend/debug_qr/frame0_IMG_184_717.png new file mode 100644 index 0000000..7766212 Binary files /dev/null and b/backend/debug_qr/frame0_IMG_184_717.png differ diff --git a/backend/debug_qr/frame0_IMG_539_108.png b/backend/debug_qr/frame0_IMG_539_108.png new file mode 100644 index 0000000..8c8d483 Binary files /dev/null and b/backend/debug_qr/frame0_IMG_539_108.png differ diff --git a/backend/debug_qr/frame0_IMG_539_412.png b/backend/debug_qr/frame0_IMG_539_412.png new file mode 100644 index 0000000..8e4518f Binary files /dev/null and b/backend/debug_qr/frame0_IMG_539_412.png differ diff --git a/backend/debug_qr/frame0_IMG_539_708.png b/backend/debug_qr/frame0_IMG_539_708.png new file mode 100644 index 0000000..728dcee Binary files /dev/null and b/backend/debug_qr/frame0_IMG_539_708.png differ diff --git a/backend/debug_qr/frame0_IMG_894_108.png b/backend/debug_qr/frame0_IMG_894_108.png new file mode 100644 index 0000000..815c1bc Binary files /dev/null and b/backend/debug_qr/frame0_IMG_894_108.png differ diff --git a/backend/debug_qr/frame0_IMG_894_412.png b/backend/debug_qr/frame0_IMG_894_412.png new file mode 100644 index 0000000..16c5681 Binary files /dev/null and b/backend/debug_qr/frame0_IMG_894_412.png differ diff --git a/backend/debug_qr/frame0_IMG_894_708.png b/backend/debug_qr/frame0_IMG_894_708.png new file mode 100644 index 0000000..1b979c7 Binary files /dev/null and b/backend/debug_qr/frame0_IMG_894_708.png differ diff --git a/backend/debug_qr/page_after_login_click.html b/backend/debug_qr/page_after_login_click.html new file mode 100644 index 0000000..7ba7ccc --- /dev/null +++ b/backend/debug_qr/page_after_login_click.html @@ -0,0 +1,1556 @@ +抖音-记录美好生活
读屏标签已关闭

充钻石

壁纸

    通知

    消息

投稿

    【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
    02:03
    16.3万
    【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
    @苏晚 · 7月25日
    法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
    35:12
    62.7万
    法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
    @猛虎电影 · 7月11日
    你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
    03:29
    4.2万
    你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
    @饺子WTF · 7月26日
    被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
    32:40
    173.3万
    被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
    @乔佳旭 · 8月7日
    一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
    32:48
    16.4万
    一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
    @Vvstar · 8月17日
    当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
    03:00
    175.5万
    当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
    @侯绿萝 · 6天前
    轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元  #剧情 #反转 #扮猪吃虎名场面
    29:45
    7.0万
    轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元 #剧情 #反转 #扮猪吃虎名场面
    @雾里 · 23小时前
    深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
    35:55
    2.3万
    深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
    @四爷说剧 · 8月16日
    当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家  #洲人洲事
    06:48
    3.4万
    当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家 #洲人洲事
    @尾巴(三角洲行动) · 1天前
    本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
    02:14
    16.1万
    本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
    @开心锤锤 · 23小时前
    【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
    02:03
    16.3万
    【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
    @苏晚 · 7月25日
    法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
    35:12
    62.7万
    法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
    @猛虎电影 · 7月11日
    你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
    03:29
    4.2万
    你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
    @饺子WTF · 7月26日
    被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
    32:40
    173.3万
    被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
    @乔佳旭 · 8月7日
    一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
    32:48
    16.4万
    一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
    @Vvstar · 8月17日
    当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
    03:00
    175.5万
    当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
    @侯绿萝 · 6天前
    轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元  #剧情 #反转 #扮猪吃虎名场面
    29:45
    7.0万
    轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元 #剧情 #反转 #扮猪吃虎名场面
    @雾里 · 23小时前
    深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
    35:55
    2.3万
    深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
    @四爷说剧 · 8月16日
    当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家  #洲人洲事
    06:48
    3.4万
    当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家 #洲人洲事
    @尾巴(三角洲行动) · 1天前
    本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
    02:14
    16.1万
    本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
    @开心锤锤 · 23小时前
    \ No newline at end of file diff --git a/backend/debug_qr/page_initial.html b/backend/debug_qr/page_initial.html new file mode 100644 index 0000000..d5cb6f5 --- /dev/null +++ b/backend/debug_qr/page_initial.html @@ -0,0 +1,1546 @@ +抖音-记录美好生活

    充钻石

    壁纸

      通知

      消息

    投稿

      【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
      02:03
      16.3万
      【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园
      @苏晚 · 7月25日
      法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
      35:12
      62.7万
      法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说
      @猛虎电影 · 7月11日
      你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
      03:29
      4.2万
      你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享
      @饺子WTF · 7月26日
      被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
      32:40
      173.3万
      被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候
      @乔佳旭 · 8月7日
      一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
      32:48
      16.4万
      一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选
      @Vvstar · 8月17日
      当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
      03:00
      175.5万
      当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素
      @侯绿萝 · 6天前
      轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元  #剧情 #反转 #扮猪吃虎名场面
      29:45
      7.0万
      轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元 #剧情 #反转 #扮猪吃虎名场面
      @雾里 · 23小时前
      深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
      35:55
      2.3万
      深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史
      @四爷说剧 · 8月16日
      当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家  #洲人洲事
      06:48
      3.4万
      当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家 #洲人洲事
      @尾巴(三角洲行动) · 1天前
      本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
      02:14
      16.1万
      本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划
      @开心锤锤 · 23小时前
      \ No newline at end of file diff --git a/backend/debug_qr/report.json b/backend/debug_qr/report.json new file mode 100644 index 0000000..9ead687 --- /dev/null +++ b/backend/debug_qr/report.json @@ -0,0 +1,855 @@ +{ + "frames": [ + { + "index": 0, + "url": "https://www.douyin.com/", + "qrcodes": [ + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "AXNt5Hoz", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "Wzqh8kMJ", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "MaDupF4a", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png", + "alt": "", + "class": "RSP3dVtx", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png", + "alt": "", + "class": "M3dFOzE4", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "sB_GUV4n", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "BD9BarA8", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png", + "alt": "", + "class": "jMPyhzfG", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png", + "alt": "", + "class": "efPPcdLl", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png", + "alt": "", + "class": "RnpNMA46", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "FqRV7w1P", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "_Whzlv1b", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "j1TwxzPC", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png", + "alt": "", + "class": "zYgniQaG", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png", + "alt": "", + "class": "uJKU1tdN", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "HkY6seUs", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "khDBMSjy", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png", + "alt": "", + "class": "GkION2OQ", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png", + "alt": "", + "class": "q2uupcgz", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 16, + "y": 58, + "width": 128, + "height": 40 + }, + "src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png", + "alt": "", + "class": "n6fjbOcQ", + "outer": "", + "screenshot": "frame0_IMG_16_58.png" + }, + { + "selector": "img", + "tag": "IMG", + "visible": true, + "box": { + "x": 539.328125, + "y": 108, + "width": 339.328125, + "height": 190.859375 + }, + "src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_ea6e0dc453683550de835cc2ded929b0~tplv-dy-resize-walign-adapt-aq:540:q7", + "alt": "法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说", + "class": "XTdkxrLI discover-video-card-img", + "outer": "\"一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 { + const nodes = [...document.querySelectorAll('button, span, div, a, p')]; + for (const el of nodes) { + const t = (el.innerText || '').trim(); + if ((t.includes('登录') || t.toLowerCase().includes('login')) && el.offsetParent) { + el.click(); + return t; + } + } + return ''; + }""") + print("clicked via js:", clicked) + if clicked: + break + except Exception as e: + print(f"js click attempt {attempt} err: {e}") + await asyncio.sleep(1) + await asyncio.sleep(4) + + html = await page.content() + with open(os.path.join(OUT_DIR, "page_after_login_click.html"), "w", encoding="utf-8") as f: + f.write(html) + print("saved page_after_login_click.html") + + try: + await page.screenshot(path=os.path.join(OUT_DIR, "00_viewport.png"), full_page=False, timeout=10000) + print("saved 00_viewport.png") + except Exception as e: + print("viewport screenshot failed:", e) + + # 尝试切换「扫码登录」 + for frame in page.frames: + try: + switched = await frame.evaluate("""() => { + const nodes = [...document.querySelectorAll('span, div, a, button, p')]; + let best = null; + for (const el of nodes) { + const t = (el.innerText || '').trim(); + if ((t === '扫码登录' || t === '扫码') && el.offsetParent) { + if (!best || el.children.length < best.children.length) best = el; + } + } + if (best) { best.click(); return 'switched'; } + return ''; + }""") + print(f"frame {frame.url[:60]} switched={switched}") + except Exception as e: + print(f"frame switch err: {e}") + await asyncio.sleep(2) + + report = {"frames": [], "candidates": []} + + # 遍历所有 frame,查找二维码相关元素 + for idx, frame in enumerate(page.frames): + frame_report = {"index": idx, "url": frame.url, "qrcodes": [], "panels": []} + selectors = [ + "[class*='qrcode'] img", + "[class*='QrCode'] img", + "[class*='qr-code'] img", + "img[class*='qrcode']", + "img[src*='qrcode']", + "img[alt*='二维码']", + "img[alt*='qr']", + "[class*='qrcode'] canvas", + "canvas[class*='qrcode']", + "[class*='scan'] img", + "[class*='scan'] canvas", + "img", + "canvas", + ] + for sel in selectors: + try: + els = await frame.query_selector_all(sel) + for el in els: + try: + visible = await el.is_visible() + box = await el.bounding_box() + tag = await el.evaluate("e => e.tagName") + src = await el.get_attribute("src") or "" + alt = await el.get_attribute("alt") or "" + cls = await el.get_attribute("class") or "" + outer = await el.evaluate("e => e.outerHTML.slice(0, 300)") + info = { + "selector": sel, + "tag": tag, + "visible": visible, + "box": box, + "src_prefix": src[:120] if src else "", + "alt": alt, + "class": cls, + "outer": outer, + } + if (tag.lower() in ("img", "canvas") and box and box.get("width", 0) > 40 and visible): + frame_report["qrcodes"].append(info) + # 截图该元素 + safe_name = f"frame{idx}_{tag}_{int(box['x'])}_{int(box['y'])}.png" + try: + await el.screenshot(path=os.path.join(OUT_DIR, safe_name)) + info["screenshot"] = safe_name + except Exception as e: + info["screenshot_err"] = str(e) + except Exception as e: + print(f" el inspect err: {e}") + except Exception as e: + print(f"frame {idx} selector {sel} err: {e}") + + # 登录面板/容器 + panel_selectors = [ + "[class*='qrcode-container']", + "[class*='qrcodeContainer']", + "[class*='qrcode']", + "[class*='QrCode']", + "[class*='login-scan']", + "[class*='scan-code']", + "#login-pannel", + "[class*='login_panel']", + "[class*='login-panel']", + "[class*='account_login']", + ] + for sel in panel_selectors: + try: + els = await frame.query_selector_all(sel) + for el in els: + visible = await el.is_visible() + box = await el.bounding_box() + cls = await el.get_attribute("class") or "" + if visible and box and box.get("width", 0) > 80: + frame_report["panels"].append({ + "selector": sel, + "class": cls, + "box": box, + }) + safe_name = f"frame{idx}_panel_{int(box['x'])}_{int(box['y'])}.png" + try: + await el.screenshot(path=os.path.join(OUT_DIR, safe_name)) + frame_report["panels"][-1]["screenshot"] = safe_name + except Exception as e: + frame_report["panels"][-1]["screenshot_err"] = str(e) + except Exception as e: + pass + + report["frames"].append(frame_report) + + # 尝试用 JS 暴力查找所有 img/canvas 中可能为二维码的 + all_candidates = await page.evaluate("""() => { + const out = []; + document.querySelectorAll('img, canvas').forEach((el, i) => { + const rect = el.getBoundingClientRect(); + if (rect.width > 30 && rect.height > 30 && rect.width < 600 && rect.height < 600) { + const style = window.getComputedStyle(el); + out.push({ + tag: el.tagName, + index: i, + width: rect.width, + height: rect.height, + x: rect.x, + y: rect.y, + src: el.tagName === 'IMG' ? (el.src || '').slice(0, 200) : '', + alt: el.alt || '', + class: el.className || '', + parentText: (el.parentElement ? el.parentElement.innerText : '').slice(0, 80), + }); + } + }); + return out; + }""") + report["candidates"] = all_candidates + + with open(os.path.join(OUT_DIR, "report.json"), "w", encoding="utf-8") as f: + json.dump(report, f, ensure_ascii=False, indent=2) + + print("report saved to", os.path.join(OUT_DIR, "report.json")) + print("found qrcode-like elements:", sum(len(f["qrcodes"]) for f in report["frames"])) + print("found panels:", sum(len(f["panels"]) for f in report["frames"])) + + await browser.close() + + +if __name__ == "__main__": + try: + asyncio.run(inspect()) + except Exception as e: + print("FATAL:", e, file=sys.stderr) + import traceback + traceback.print_exc() + raise diff --git a/backend/kefu.db-shm b/backend/kefu.db-shm index 0ca214a..9b62e0d 100644 Binary files a/backend/kefu.db-shm and b/backend/kefu.db-shm differ diff --git a/backend/kefu.db-wal b/backend/kefu.db-wal index 92b8bbc..55a5ef1 100644 Binary files a/backend/kefu.db-wal and b/backend/kefu.db-wal differ diff --git a/backend/kefu1.db b/backend/kefu1.db index a176b85..81b491d 100644 Binary files a/backend/kefu1.db and b/backend/kefu1.db differ diff --git a/backend/main.py b/backend/main.py index e35b1ee..c6974ba 100644 --- a/backend/main.py +++ b/backend/main.py @@ -4,6 +4,7 @@ import json import asyncio import ipaddress import logging +import time import uuid from datetime import datetime, timezone from io import BytesIO @@ -92,6 +93,7 @@ from utils.cookie_store import ( cookie_summary, validate_cookie_json, analyze_cookie, + extract_user_agent_from_cookie_data, ) from rpa_engine.device_profiles import list_device_profiles, profile_label_for_ua, resolve_user_agent from rpa_engine.egress_channels import ( @@ -100,6 +102,22 @@ from rpa_engine.egress_channels import ( ) logger = logging.getLogger("main") + +def _ui_conversation_page_budget() -> int: + """用户点开会话列表时允许翻的收件箱页数。 + + 抖音收件箱按游标分页,一次请求只给一页(实测每页约 100-500KB);某账号翻 + 6 页拿到 35 个会话仍未翻完。所以这里必须有预算:只拿一页会把「其中一页」 + 当成完整列表,不设上限又可能为一次点击拉下好几 MB。 + 默认 3 页只是个折中——真正花多少流量换多完整的列表是业务取舍, + 用 KEFU_UI_CONVERSATION_PAGES 调整;翻不完时仍会并入本地历史, + 并且不会把残缺列表伪装成完整列表。 + """ + try: + value = int(os.getenv("KEFU_UI_CONVERSATION_PAGES", "3") or 3) + except (TypeError, ValueError): + value = 3 + return max(1, min(20, value)) from utils import system_logger app = FastAPI(title="抖音多账号自动回复管理系统 API") @@ -113,11 +131,17 @@ app.add_middleware( ) # RPA 任务管理器 +# 自动重登录防抖:同账号 30 分钟内最多触发一次,防止「扫码失败→失效→再重登录」死循环 +_AUTO_RELOGIN_COOLDOWN = float(os.getenv("KEFU_AUTO_RELOGIN_COOLDOWN", "1800") or 1800) + + class WorkerManager: def __init__(self): self.workers = {} # account_id -> DouyinWorker self._account_locks: dict[int, asyncio.Lock] = {} self._preparation_locks: dict[int, asyncio.Lock] = {} + self._auto_relogin_tasks: dict[int, asyncio.Task] = {} + self._last_auto_relogin_at: dict[int, float] = {} def _account_lock(self, account_id: int) -> asyncio.Lock: return self._account_locks.setdefault(int(account_id), asyncio.Lock()) @@ -145,6 +169,10 @@ class WorkerManager: account_id, login_mode=login_mode, credential_prevalidated=credential_prevalidated, + # 登录态失效(KICK/INVALID_REQUEST/用户未登录)时自动重登录: + # 重新以 browser 模式拉起 worker,浏览器探测未登录 → 弹二维码 + # → 用户扫码 → 自动采集凭证并恢复托管。 + relogin_hook=self._schedule_auto_relogin, ) self.workers[account_id] = worker await worker.start() @@ -209,6 +237,98 @@ class WorkerManager: worker = self.workers.get(account_id) return worker.is_running if worker else False + async def _schedule_auto_relogin(self, account_id: int) -> None: + """登录态失效后由 worker 回调:防抖 + 后台异步执行自动重登录。 + + 注意:本方法在 service 的发送协程里被 await,必须快速返回, + 实际的浏览器重登录流程放到独立 task 中执行。 + """ + now = time.monotonic() + last = self._last_auto_relogin_at.get(account_id, 0.0) + if now - last < _AUTO_RELOGIN_COOLDOWN: + logger.info( + f"Account {account_id}: auto relogin skipped " + f"(cooldown {_AUTO_RELOGIN_COOLDOWN}s)" + ) + return + self._last_auto_relogin_at[account_id] = now + prev = self._auto_relogin_tasks.get(account_id) + if prev and not prev.done(): + logger.info(f"Account {account_id}: auto relogin already in progress") + return + task = asyncio.create_task( + self._auto_relogin_account(account_id), + name=f"auto-relogin-{account_id}", + ) + self._auto_relogin_tasks[account_id] = task + + def _cleanup(done_task: asyncio.Task) -> None: + if self._auto_relogin_tasks.get(account_id) is done_task: + self._auto_relogin_tasks.pop(account_id, None) + + task.add_done_callback(_cleanup) + + async def _auto_relogin_account(self, account_id: int) -> None: + """自动重登录:等旧 worker 退出,置 logging_in,以 browser 模式重启。 + + 新 worker 的浏览器流程会先探测页面登录态:未登录则自动弹二维码 + (qr_code_base64 写库,前端账号卡片轮询展示),用户扫码成功后自动 + 采集 IM 凭证并恢复托管;登录超时/失败则回落到 offline 等人工处理。 + """ + try: + # 1) 等旧 worker 完全退出(on_im_session_invalid 已置 is_running=False, + # _run_loop 收尾需要一点时间) + for _ in range(100): + worker = self.workers.get(account_id) + if worker is None or not worker.is_running: + break + await asyncio.sleep(0.2) + + # 2) 置 logging_in(前端显示「等待扫码」,二维码由新 worker 生成) + async with AsyncSessionLocal() as db: + account = ( + await db.execute( + select(Account).where(Account.id == account_id) + ) + ).scalar_one_or_none() + if account is None: + return + account.status = "logging_in" + account.qr_code_base64 = None + account.error_message = None + await db.commit() + system_logger.record( + "登录态失效,正在自动重登录", + detail=( + "系统检测到抖音登录态失效,已自动打开登录流程。" + "请留意账号卡片上的二维码,用抖音 App 扫码后托管将自动恢复。" + ), + level="warning", + category="auth", + account_id=account_id, + ) + + # 3) 以 browser 模式重启:浏览器探测未登录 → 弹二维码 → 扫码 → 恢复托管。 + # 不等待就绪(wait_until_ready=False),让新 worker 自行走完整登录流程。 + await self.start_worker(account_id, login_mode="browser") + except asyncio.CancelledError: + raise + except Exception as exc: + logger.error(f"Account {account_id}: auto relogin failed: {exc}") + try: + async with AsyncSessionLocal() as db: + await db.execute( + update(Account) + .where(Account.id == account_id) + .values( + status="offline", + error_message=f"自动重登录失败:{exc}", + ) + ) + await db.commit() + except Exception: + pass + manager = WorkerManager() UPLOAD_DIR = os.path.join(os.path.dirname(__file__), "uploads", "messages") @@ -443,6 +563,26 @@ def _account_has_cookie(account: Account) -> bool: return os.path.exists(get_cookie_path(account.id)) +def _backfill_user_agent_from_cookie(account: Account, standard_json_str: Optional[str]) -> None: + """凭证里带登录头(user_agent)且账号未显式配置 UA 时自动回填。 + + 已有自定义 UA(account.user_agent 非空)的账号保持不变,避免覆盖用户选择; + cookie_data 为空或解析失败时静默跳过。回填后发送/接收链路经 + _build_account_im_session 统一走 resolve_user_agent(account.user_agent), + 保证 UA 全链路一致。 + """ + if account.user_agent: + return + if not standard_json_str: + return + try: + ua = extract_user_agent_from_cookie_data(standard_json_str) + except Exception: + ua = "" + if ua: + account.user_agent = ua + + def _build_account_im_session(account: Account) -> DouyinImSession: cookie_data = _get_account_cookie_data(account) storage = json.loads(cookie_data) if cookie_data else {} @@ -941,6 +1081,8 @@ class AccountCookieResponse(BaseModel): im_status: Optional[str] = None can_skip_browser: bool = False should_reset: bool = False + user_agent: Optional[str] = None + user_agent_label: Optional[str] = None class AccountVideoItem(BaseModel): @@ -1019,6 +1161,8 @@ async def _build_cookie_response( im_status=im_detail["im_status"], can_skip_browser=im_detail["can_skip_browser"], should_reset=im_detail.get("should_reset", False), + user_agent=account.user_agent or None, + user_agent_label=profile_label_for_ua(account.user_agent), ) @@ -2054,6 +2198,7 @@ async def update_account_cookie( account.cookie_path = cookie_path account.cookie_updated_at = datetime.utcnow() account.updated_at = datetime.utcnow() + _backfill_user_agent_from_cookie(account, standard_json_str) await db.execute( update(AccountProfileDetail) .where(AccountProfileDetail.account_id == account_id) @@ -2126,6 +2271,7 @@ async def create_account( account.cookie_data = standard_json_str account.cookie_path = cookie_path account.cookie_updated_at = datetime.utcnow() + _backfill_user_agent_from_cookie(account, standard_json_str) try: from rpa_engine.account_profile import apply_douyin_profile @@ -2760,7 +2906,29 @@ async def get_account_conversations( if not conversations: async with DouyinImHttpClient(session, account_id=account_id) as http: - conversations = await http.get_conversations() + # 用户点开会话列表:从头翻,而不是「最近半小时有动静的会话」。 + # 抖音没有「一次取回全部会话」的接口,收件箱是按游标分页的, + # 所以这里花一个翻页预算;翻不完时把 DB 历史并进来补齐, + # 免得把其中一页当成完整会话列表展示给用户。 + page_budget = _ui_conversation_page_budget() + conversations = await http.get_conversations( + lookback_seconds=0, + max_pages=page_budget, + ) + if http.inbox_truncated: + logger.info( + "Account %s conversation list truncated at %d pages; " + "merging local history", + account_id, + page_budget, + ) + known = { + str(c.get("conversation_id") or "") for c in conversations + } + my_uid = session.my_uid or 0 + for item in await _conversations_from_logs(db, account_id, my_uid): + if str(item.get("conversation_id") or "") not in known: + conversations.append(item) if not conversations: my_uid = session.my_uid or 0 @@ -2771,6 +2939,7 @@ async def get_account_conversations( session.cookie_header(), session.web_protect_str, session.keys_str, + user_agent=session.user_agent or "", ) my_uid = auth.get_uid() or 0 conversations = await _conversations_from_logs(db, account_id, my_uid) diff --git a/backend/models/db_migrate.py b/backend/models/db_migrate.py index e090640..861c663 100644 --- a/backend/models/db_migrate.py +++ b/backend/models/db_migrate.py @@ -53,12 +53,42 @@ def add_index_if_missing( conn.execute(text(f"CREATE INDEX {index_name} ON {table} ({safe_columns})")) +def widen_mysql_text_columns(conn, table: str, columns: tuple[str, ...]) -> None: + """Upgrade large account payloads from TEXT to LONGTEXT on MySQL. + + SQLite and PostgreSQL TEXT values are not limited to 64 KiB, while MySQL + TEXT is. Browser storage_state and full-page QR/captcha screenshots can + legitimately exceed that size. + """ + if _dialect(conn) != "mysql": + return + allowed = {"cookie_data", "im_session_data", "qr_code_base64"} + try: + reflected = { + item["name"]: str(item.get("type") or "").upper() + for item in inspect(conn).get_columns(table) + } + except Exception: + return + for column in columns: + if column not in allowed or column not in reflected: + continue + if reflected[column] == "LONGTEXT": + continue + conn.execute( + text(f"ALTER TABLE {table} MODIFY COLUMN {column} LONGTEXT NULL") + ) + + def migrate_accounts_table(conn) -> None: add_column_if_missing( conn, "accounts", "cookie_data", - {"default": "ALTER TABLE accounts ADD COLUMN cookie_data TEXT"}, + { + "default": "ALTER TABLE accounts ADD COLUMN cookie_data TEXT", + "mysql": "ALTER TABLE accounts ADD COLUMN cookie_data LONGTEXT", + }, ) add_column_if_missing( conn, @@ -73,7 +103,10 @@ def migrate_accounts_table(conn) -> None: conn, "accounts", "im_session_data", - {"default": "ALTER TABLE accounts ADD COLUMN im_session_data TEXT"}, + { + "default": "ALTER TABLE accounts ADD COLUMN im_session_data TEXT", + "mysql": "ALTER TABLE accounts ADD COLUMN im_session_data LONGTEXT", + }, ) add_column_if_missing( conn, @@ -138,6 +171,11 @@ def migrate_accounts_table(conn) -> None: "douyin_uid", {"default": "ALTER TABLE accounts ADD COLUMN douyin_uid VARCHAR(64)"}, ) + widen_mysql_text_columns( + conn, + "accounts", + ("cookie_data", "im_session_data", "qr_code_base64"), + ) def migrate_account_videos_table(conn) -> None: diff --git a/backend/models/models.py b/backend/models/models.py index 4e73a16..b749711 100644 --- a/backend/models/models.py +++ b/backend/models/models.py @@ -1,5 +1,6 @@ from datetime import datetime from sqlalchemy import Column, Integer, String, Boolean, DateTime, ForeignKey, Index, Text, UniqueConstraint +from sqlalchemy.dialects.mysql import LONGTEXT from sqlalchemy.orm import relationship, validates from .database import Base from utils.log_limits import bound_error_log_content, bound_message_log_content @@ -108,9 +109,9 @@ class Account(Base): phone = Column(String(20), nullable=True) # 绑定的手机号(可选) status = Column(String(50), default="offline") # offline, logging_in, online, error cookie_path = Column(String(255), nullable=True) # 存储 cookie/session 的路径 - cookie_data = Column(Text, nullable=True) # Playwright storage_state JSON + cookie_data = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # Playwright storage_state JSON cookie_updated_at = Column(DateTime, nullable=True) # Cookie 最近更新时间 - im_session_data = Column(Text, nullable=True) # IM 直连会话 (WS URL / device_id 等) + im_session_data = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # IM 直连会话 (WS URL / device_id 等) reply_delay_seconds = Column(Integer, default=0) # 账号回复排队间隔;0/NULL=继承系统默认 reply_cooldown_seconds = Column(Integer, nullable=True) # 自动回复冷却秒数;NULL=继承全局设置 follow_welcome_enabled = Column(Boolean, default=False) # 新粉丝关注后自动发送欢迎语 @@ -118,7 +119,7 @@ class Account(Base): user_agent = Column(Text, nullable=True) # 伪装设备头(User-Agent),空=默认 egress_public_ip = Column(String(64), nullable=True) # 指定公网出口;空=自动选择 egress_auto_attempts = Column(Integer, nullable=False, default=1) # 发送失败时最多串行尝试的出口数 - qr_code_base64 = Column(Text, nullable=True) # 当前登录二维码的 base64 字符串 + qr_code_base64 = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # 当前登录二维码的 base64 字符串 error_message = Column(Text, nullable=True) # 错误信息 quota_disabled = Column(Boolean, default=False, index=True) # 额度不足被停用 created_at = Column(DateTime, default=datetime.utcnow) diff --git a/backend/rpa_engine/account_profile.py b/backend/rpa_engine/account_profile.py index df1e2a5..f8c54d3 100644 --- a/backend/rpa_engine/account_profile.py +++ b/backend/rpa_engine/account_profile.py @@ -1,1081 +1,1137 @@ -"""从 Cookie 抓取抖音账号资料(昵称 / 头像 / UID)及作品列表。""" - -from __future__ import annotations - -import asyncio -import json -import logging -from datetime import datetime -from typing import Any, Optional - -import requests -from sqlalchemy import delete, select -from sqlalchemy.ext.asyncio import AsyncSession - -from models.models import Account, AccountProfileDetail, AccountVideo -from rpa_engine.douyin_im.auth import DouyinAuth -from rpa_engine.douyin_im.dy_util import ( - DEFAULT_USER_AGENT, - generate_a_bogus, - generate_msToken, - generate_webid, - splice_url, -) -from rpa_engine.douyin_im.protocol import _pick_avatar_url -from rpa_engine.douyin_im.session import DouyinImSession -from rpa_engine.device_profiles import resolve_user_agent - -logger = logging.getLogger("account_profile") - - -def _pick_str(data: dict, *keys: str) -> str: - for key in keys: - value = data.get(key) - if value is not None and str(value).strip(): - return str(value).strip() - return "" - - -def _pick_int(data: dict, *keys: str) -> Optional[int]: - for key in keys: - value = data.get(key) - if value is None or value == "": - continue - try: - return int(value) - except (TypeError, ValueError): - continue - return None - - -def _iter_profile_nodes(data: Any): - if not isinstance(data, dict): - return - yield data - for key in ("user", "user_info", "data", "creator"): - node = data.get(key) - if isinstance(node, dict): - yield node - - -def _extract_profile_from_payload(data: Any) -> dict[str, str]: - for node in _iter_profile_nodes(data): - uid = _pick_str(node, "uid", "user_uid", "user_id", "creator_user_id", "id") - nickname = _pick_str( - node, - "nickname", - "nick_name", - "unique_id", - "display_name", - "name", - ) - avatar = _pick_avatar_url(node) - if uid or nickname or avatar: - return { - "uid": uid, - "nickname": nickname, - "avatar_url": avatar, - } - return {} - - -def _extract_detail_from_payload(data: Any) -> dict[str, Any]: - """从抖音资料接口响应中提取详细统计。""" - detail: dict[str, Any] = {} - for node in _iter_profile_nodes(data): - if not detail.get("uid"): - uid = _pick_str(node, "uid", "user_uid", "user_id", "creator_user_id", "id") - if uid: - detail["uid"] = uid - if not detail.get("nickname"): - nickname = _pick_str( - node, "nickname", "nick_name", "display_name", "name" - ) - if nickname: - detail["nickname"] = nickname - if not detail.get("avatar_url"): - avatar = _pick_avatar_url(node) - if avatar: - detail["avatar_url"] = avatar - if not detail.get("unique_id"): - unique_id = _pick_str(node, "unique_id", "short_id", "douyin_id") - if unique_id: - detail["unique_id"] = unique_id - if not detail.get("sec_user_id"): - sec_user_id = _pick_str(node, "sec_uid", "sec_user_id") - if sec_user_id: - detail["sec_user_id"] = sec_user_id - if not detail.get("signature"): - signature = _pick_str(node, "signature", "desc", "bio") - if signature: - detail["signature"] = signature - - for field, keys in ( - ("video_count", ("aweme_count", "post_count", "video_count", "works_count")), - ("follower_count", ("follower_count", "fans_count")), - ("following_count", ("following_count", "follow_count")), - ("total_favorited", ("total_favorited", "total_favorited_count", "like_count")), - ("favoriting_count", ("favoriting_count", "favorite_count")), - ): - if detail.get(field) is None: - val = _pick_int(node, *keys) - if val is not None: - detail[field] = val - return detail - - -def _pick_url_list(data: Any) -> str: - if isinstance(data, str) and data.startswith("http"): - return data - if isinstance(data, dict): - urls = data.get("url_list") or data.get("urlList") or [] - if isinstance(urls, list): - for item in urls: - if isinstance(item, str) and item.startswith("http"): - return item - if isinstance(data, list): - for item in data: - if isinstance(item, str) and item.startswith("http"): - return item - return "" - - -def _pick_video_play_url(video: dict[str, Any]) -> str: - play_addr = video.get("play_addr") or video.get("download_addr") or {} - urls = play_addr.get("url_list") or [] - if not isinstance(urls, list): - urls = [] - for item in urls: - if not isinstance(item, str) or not item.startswith("http"): - continue - lower = item.lower() - if lower.endswith(".mp3") or "/ies-music/" in lower: - continue - if "douyinvod.com" in lower or "/video/" in lower or "mime_type=video" in lower: - return item.replace("playwm", "play") - for item in urls: - if isinstance(item, str) and item.startswith("http") and not item.lower().endswith(".mp3"): - return item.replace("playwm", "play") - bit_rate = video.get("bit_rate") - if isinstance(bit_rate, list): - for entry in bit_rate: - if not isinstance(entry, dict): - continue - play = entry.get("play_addr") or {} - url = _pick_url_list(play) - if url and not url.lower().endswith(".mp3"): - return url.replace("playwm", "play") - return "" - - -def _is_published_aweme(item: dict[str, Any]) -> bool: - """仅保留已公开发布的作品(排除私密/审核中/已删除)。""" - if not isinstance(item, dict): - return False - if item.get("is_private"): - return False - - status = item.get("status") - if isinstance(status, dict): - if status.get("is_delete"): - return False - if status.get("in_reviewing"): - return False - if status.get("is_prohibited"): - return False - private_status = status.get("private_status") - if private_status not in (None, 0, "0"): - return False - part_see = status.get("part_see") - if part_see not in (None, 0, "0", False): - return False - review = status.get("review_result") - if isinstance(review, dict): - review_status = review.get("review_status") - if review_status not in (None, 0, "0"): - return False - - rate = item.get("rate") - if rate in (10, 11, 12): - return False - return True - - -def _parse_aweme_item(item: dict[str, Any]) -> dict[str, Any] | None: - if not isinstance(item, dict): - return None - aweme_id = _pick_str(item, "aweme_id", "awemeId", "item_id") - if not aweme_id: - return None - - title = _pick_str(item, "desc", "title", "content") - video = item.get("video") if isinstance(item.get("video"), dict) else {} - cover_url = _pick_url_list(video.get("cover") or video.get("origin_cover") or item.get("cover")) - video_url = _pick_video_play_url(video) - if not cover_url and isinstance(item.get("images"), list) and item["images"]: - first_image = item["images"][0] - if isinstance(first_image, dict): - cover_url = _pick_url_list(first_image.get("url_list") or first_image) - - share_url = _pick_str(item, "share_url", "share_link") - if not share_url: - share_url = f"https://www.douyin.com/video/{aweme_id}" - - create_time = item.get("create_time") or item.get("createTime") - create_dt: datetime | None = None - if create_time: - try: - create_dt = datetime.utcfromtimestamp(int(create_time)) - except (TypeError, ValueError): - create_dt = None - - statistics = item.get("statistics") if isinstance(item.get("statistics"), dict) else {} - if not title: - title = f"作品 {aweme_id}" - - aweme_type = item.get("aweme_type") - if video_url: - media_type = "video" - elif aweme_type == 68 or item.get("images"): - media_type = "image" - elif aweme_type in (0, 51, 55, 61): - media_type = "video" - else: - media_type = "other" - - return { - "aweme_id": aweme_id, - "title": title, - "cover_url": cover_url, - "video_url": video_url, - "share_url": share_url, - "create_time": create_dt, - "media_type": media_type, - "digg_count": _pick_int(statistics, "digg_count", "like_count"), - "comment_count": _pick_int(statistics, "comment_count"), - "play_count": _pick_int(statistics, "play_count", "view_count"), - } - - -def _parse_creator_item(item: dict[str, Any]) -> dict[str, Any] | None: - if not isinstance(item, dict): - return None - aweme_id = _pick_str(item, "item_id", "item_id_plain", "aweme_id") - if not aweme_id: - return None - title = _pick_str(item, "title", "desc") or f"作品 {aweme_id}" - cover_url = _pick_str(item, "cover_image_url", "cover_url") - share_url = _pick_str(item, "item_link", "share_url") or f"https://www.douyin.com/video/{aweme_id}" - create_time = item.get("create_time") - create_dt: datetime | None = None - if create_time: - try: - create_dt = datetime.utcfromtimestamp(int(create_time)) - except (TypeError, ValueError): - create_dt = None - return { - "aweme_id": aweme_id, - "title": title, - "cover_url": cover_url or None, - "video_url": None, - "share_url": share_url, - "create_time": create_dt, - "media_type": "image" if _pick_int(item, "media_type") == 2 else "other", - "digg_count": _pick_int(item, "digg_count", "like_count"), - "comment_count": _pick_int(item, "comment_count"), - "play_count": _pick_int(item, "play_count", "view_count"), - } - - -def _extract_creator_item_list(data: Any) -> list[dict[str, Any]]: - if not isinstance(data, dict): - return [] - for key in ("item_info_list", "aweme_list", "item_list"): - node = data.get(key) - if isinstance(node, list) and node: - return [x for x in node if isinstance(x, dict)] - return [] - - -def _extract_aweme_list(data: Any) -> list[dict[str, Any]]: - if not isinstance(data, dict): - return [] - for key in ("aweme_list", "awemeList", "item_list", "items", "data"): - node = data.get(key) - if isinstance(node, list) and node: - return [x for x in node if isinstance(x, dict)] - if isinstance(node, dict): - nested = node.get("aweme_list") or node.get("item_list") - if isinstance(nested, list): - return [x for x in nested if isinstance(x, dict)] - return [] - - -def _douyin_get_json( - auth: DouyinAuth, - ua: str, - url: str, - base_params: dict[str, str], - referer: str = "https://www.douyin.com/", -) -> dict[str, Any] | None: - headers = { - "User-Agent": ua, - "Referer": referer, - "Accept": "application/json, text/plain, */*", - } - proxies = _requests_proxies() - try: - params = dict(base_params) - query = splice_url(params) - params["a_bogus"] = generate_a_bogus(query, user_agent=ua) - resp = requests.get( - url, - params=params, - headers=headers, - cookies=auth.cookie, - verify=False, - timeout=15, - proxies=proxies, - ) - data = resp.json() - return data if isinstance(data, dict) else None - except Exception as exc: - logger.debug(f"douyin get failed for {url}: {exc}") - return None - - -def _fetch_aweme_post_page( - auth: DouyinAuth, - ua: str, - sec_user_id: str, - max_count: int, - strategy_type: str, -) -> list[dict[str, Any]]: - base_params = { - "device_platform": "webapp", - "aid": "6383", - "channel": "channel_pc_web", - "sec_user_id": sec_user_id, - "max_cursor": "0", - "count": str(min(max_count, 35)), - "publish_video_strategy_type": strategy_type, - "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "webid": generate_webid(auth, "https://www.douyin.com/"), - "msToken": auth.msToken or generate_msToken(), - "pc_client_type": "1", - } - - videos: list[dict[str, Any]] = [] - max_cursor = 0 - while len(videos) < max_count: - params = dict(base_params) - params["max_cursor"] = str(max_cursor) - params["count"] = str(min(35, max_count - len(videos))) - data = _douyin_get_json( - auth, - ua, - "https://www.douyin.com/aweme/v1/web/aweme/post/", - params, - referer=f"https://www.douyin.com/user/{sec_user_id}", - ) - if not data: - break - - status_code = data.get("status_code") - if status_code not in (None, 0): - break - - batch = _extract_aweme_list(data) - if not batch: - break - - for item in batch: - if not _is_published_aweme(item): - continue - parsed = _parse_aweme_item(item) - if parsed and parsed["aweme_id"] not in {v["aweme_id"] for v in videos}: - videos.append(parsed) - if len(videos) >= max_count: - break - - has_more = bool(data.get("has_more") or data.get("hasMore")) - next_cursor = data.get("max_cursor") or data.get("maxCursor") - try: - next_cursor = int(next_cursor or 0) - except (TypeError, ValueError): - next_cursor = 0 - - if not has_more or next_cursor == max_cursor: - break - max_cursor = next_cursor - return videos - - -def fetch_douyin_user_videos_sync( - cookie_data: str, - sec_user_id: str, - user_agent: Optional[str] = None, - max_count: int = 50, -) -> dict[str, Any]: - """抓取用户已公开发布的作品列表(与抖音主页「作品」一致)。""" - result: dict[str, Any] = {"videos": [], "fetched": False, "message": ""} - sec_user_id = (sec_user_id or "").strip() - try: - auth, ua = _build_auth(cookie_data, user_agent) - except Exception as exc: - result["message"] = f"Cookie 无效: {exc}" - return result - - if not sec_user_id: - result["message"] = "缺少 sec_user_id,无法拉取已发布作品" - return result - - videos = _fetch_aweme_post_page(auth, ua, sec_user_id, max_count, "2") - result["videos"] = videos - result["fetched"] = bool(videos) - if not result["fetched"]: - result["message"] = "未获取到已发布作品,请确认 Cookie 有效且主页有公开作品" - return result - - -def _fetch_own_videos_sync( - auth: DouyinAuth, - ua: str, - max_count: int, -) -> dict[str, Any]: - """创作者中心作品列表(适用于当前登录账号)。""" - result: dict[str, Any] = {"videos": [], "fetched": False} - endpoints = [ - ( - "https://creator.douyin.com/aweme/v1/creator/item/list/", - { - "status": "1", - "count": str(min(max_count, 35)), - "max_cursor": "0", - "aid": "6383", - "device_platform": "webapp", - }, - "https://creator.douyin.com/creator-micro/content/manage", - "creator_item", - ), - ( - "https://creator.douyin.com/aweme/v1/creator/aweme/list/", - { - "status": "1", - "count": str(min(max_count, 35)), - "max_cursor": "0", - "aid": "6383", - "device_platform": "webapp", - }, - "https://creator.douyin.com/creator-micro/content/manage", - "aweme", - ), - ( - "https://creator.douyin.com/web/api/media/aweme/post/", - { - "status": "1", - "count": str(min(max_count, 35)), - "max_cursor": "0", - }, - "https://creator.douyin.com/creator-micro/content/manage", - "aweme", - ), - ] - videos: list[dict[str, Any]] = [] - for url, base_params, referer, parser in endpoints: - data = _douyin_get_json(auth, ua, url, base_params, referer=referer) - if not data: - continue - if parser == "creator_item": - batch = _extract_creator_item_list(data) - parse_fn = _parse_creator_item - else: - batch = _extract_aweme_list(data) - parse_fn = _parse_aweme_item - for item in batch: - parsed = parse_fn(item) - if parsed and parsed["aweme_id"] not in {v["aweme_id"] for v in videos}: - videos.append(parsed) - if len(videos) >= max_count: - break - if videos: - break - result["videos"] = videos - result["fetched"] = bool(videos) - return result - - -def _requests_proxies() -> dict | None: - try: - from rpa_engine.runtime_config import requests_proxies - - return requests_proxies() - except Exception: - return None - - -def _build_auth(cookie_data: str, user_agent: Optional[str] = None) -> tuple[DouyinAuth, str]: - storage = json.loads(cookie_data) - session = DouyinImSession.from_storage_state(storage or {}) - ua = resolve_user_agent(user_agent or session.user_agent or DEFAULT_USER_AGENT) - auth = DouyinAuth() - auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str) - auth.user_agent = ua - auth.web_id = session.web_id or session.device_id or None - return auth, ua - - -def fetch_douyin_profile_sync( - cookie_data: str, - user_agent: Optional[str] = None, -) -> dict[str, str]: - detail = fetch_douyin_profile_detail_sync(cookie_data, user_agent) - return { - "uid": detail.get("uid") or "", - "nickname": detail.get("nickname") or "", - "avatar_url": detail.get("avatar_url") or "", - } - - -def fetch_douyin_profile_detail_sync( - cookie_data: str, - user_agent: Optional[str] = None, -) -> dict[str, Any]: - """抓取抖音账号详细资料(昵称/头像/UID/作品数/粉丝等)。""" - result: dict[str, Any] = { - "uid": "", - "nickname": "", - "avatar_url": "", - "unique_id": "", - "signature": "", - "sec_user_id": "", - "video_count": None, - "follower_count": None, - "following_count": None, - "total_favorited": None, - "favoriting_count": None, - "fetched": False, - "message": "", - } - try: - auth, ua = _build_auth(cookie_data, user_agent) - except Exception as exc: - logger.warning(f"build auth for profile failed: {exc}") - result["message"] = "Cookie 无效,无法解析登录凭证" - return result - - uid = auth.get_uid() - if uid: - result["uid"] = str(uid) - - headers = { - "User-Agent": ua, - "Referer": "https://www.douyin.com/", - "Accept": "application/json, text/plain, */*", - } - endpoints: list[tuple[str, dict[str, str]]] = [ - ( - "https://www.douyin.com/aweme/v1/web/query/user/", - { - "device_platform": "webapp", - "aid": "6383", - "channel": "channel_pc_web", - "publish_video_strategy_type": "2", - "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "webid": generate_webid(auth, "https://www.douyin.com/"), - "msToken": auth.msToken or generate_msToken(), - }, - ), - ( - "https://creator.douyin.com/aweme/v1/creator/user/info/", - { - "device_platform": "webapp", - "aid": "6383", - }, - ), - ( - "https://www.douyin.com/aweme/v1/web/user/profile/self/", - { - "device_platform": "webapp", - "aid": "6383", - "channel": "channel_pc_web", - }, - ), - ] - - proxies = _requests_proxies() - valid_profile_response = False - for url, base_params in endpoints: - try: - params = dict(base_params) - query = splice_url(params) - params["a_bogus"] = generate_a_bogus(query, user_agent=ua) - resp = requests.get( - url, - params=params, - headers=headers, - cookies=auth.cookie, - verify=False, - timeout=12, - proxies=proxies, - ) - data = resp.json() - if not isinstance(data, dict): - continue - - status_code = data.get("status_code") - if status_code is not None: - try: - if int(status_code) != 0: - continue - except (TypeError, ValueError): - continue - - basic = _extract_profile_from_payload(data) - stats = _extract_detail_from_payload(data) - payload_has_profile = bool( - data.get("user_uid") - or basic.get("uid") - or basic.get("nickname") - or basic.get("avatar_url") - or stats.get("uid") - or stats.get("sec_user_id") - or stats.get("unique_id") - ) - if not payload_has_profile: - # 401/风控响应也可能是 JSON;没有任何用户资料节点时 - # 只能视为传输/鉴权未知,不能确认 sec_user_id 缺失。 - continue - valid_profile_response = True - - if data.get("user_uid") and not result["uid"]: - result["uid"] = str(data["user_uid"]) - - if basic.get("uid") and not result["uid"]: - result["uid"] = basic["uid"] - if basic.get("nickname") and not result["nickname"]: - result["nickname"] = basic["nickname"] - if basic.get("avatar_url") and not result["avatar_url"]: - result["avatar_url"] = basic["avatar_url"] - - for key, value in stats.items(): - if value is not None and value != "" and result.get(key) in (None, "", 0): - result[key] = value - - if result.get("video_count") is not None: - result["fetched"] = True - break - if result["uid"] and result["nickname"] and result["avatar_url"]: - result["fetched"] = True - except Exception as exc: - logger.debug(f"profile detail fetch failed for {url}: {exc}") - - if not result["fetched"] and valid_profile_response: - result["fetched"] = True - - if not result["fetched"]: - result["message"] = result["message"] or "未能从抖音获取账号资料,请确认 Cookie 有效" - result["profile_response_valid"] = valid_profile_response - return result - - -async def fetch_douyin_profile( - cookie_data: str, - user_agent: Optional[str] = None, -) -> dict[str, str]: - return await asyncio.to_thread(fetch_douyin_profile_sync, cookie_data, user_agent) - - -async def fetch_douyin_profile_detail( - cookie_data: str, - user_agent: Optional[str] = None, -) -> dict[str, Any]: - return await asyncio.to_thread(fetch_douyin_profile_detail_sync, cookie_data, user_agent) - - -def _resolve_sec_user_id_by_uid_sync( - cookie_data: str, - user_agent: Optional[str], - uid: str, -) -> tuple[str, bool]: - """Return ``(sec_user_id, request_completed)`` for the UID fallback.""" - try: - auth, ua = _build_auth(cookie_data, user_agent) - extra = _douyin_get_json( - auth, - ua, - "https://www.douyin.com/aweme/v1/web/user/profile/other/", - { - "device_platform": "webapp", - "aid": "6383", - "channel": "channel_pc_web", - "user_id": str(uid), - "sec_user_id": "", - "publish_video_strategy_type": "2", - "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "webid": generate_webid(auth, "https://www.douyin.com/"), - "msToken": auth.msToken or generate_msToken(), - }, - ) - except Exception as exc: - logger.debug(f"resolve sec_user_id failed: {exc}") - return "", False - if extra is None: - return "", False - status_code = extra.get("status_code") - if status_code is not None: - try: - if int(status_code) != 0: - return "", False - except (TypeError, ValueError): - return "", False - extracted = _extract_detail_from_payload(extra) - basic = _extract_profile_from_payload(extra) - sec_user_id = (extracted.get("sec_user_id") or "").strip() - if not ( - sec_user_id - or extracted.get("uid") - or basic.get("uid") - or basic.get("nickname") - or basic.get("avatar_url") - ): - return "", False - return sec_user_id, True - - -async def fetch_douyin_profile_detail_with_sec_user_id( - cookie_data: str, - user_agent: Optional[str] = None, -) -> dict[str, Any]: - """Fetch profile data and run the existing UID fallback for sec_user_id. - - ``sec_user_id_status`` distinguishes a confirmed missing field from a - transient request failure so hosting is never stopped merely because the - profile endpoint was temporarily unavailable. - """ - detail = await fetch_douyin_profile_detail(cookie_data, user_agent) - sec_user_id = str(detail.get("sec_user_id") or "").strip() - if sec_user_id: - detail["sec_user_id"] = sec_user_id - detail["sec_user_id_status"] = "found" - return detail - - if not detail.get("fetched"): - detail["sec_user_id_status"] = "unknown" - return detail - - uid = str(detail.get("uid") or "").strip() - if uid: - sec_user_id, request_completed = await asyncio.to_thread( - _resolve_sec_user_id_by_uid_sync, - cookie_data, - user_agent, - uid, - ) - if sec_user_id: - detail["sec_user_id"] = sec_user_id - detail["sec_user_id_status"] = "found" - else: - detail["sec_user_id_status"] = ( - "missing" if request_completed else "unknown" - ) - return detail - - detail["sec_user_id_status"] = "missing" - return detail - - -async def fetch_douyin_user_videos( - cookie_data: str, - sec_user_id: str, - user_agent: Optional[str] = None, - max_count: int = 50, -) -> dict[str, Any]: - return await asyncio.to_thread( - fetch_douyin_user_videos_sync, - cookie_data, - sec_user_id, - user_agent, - max_count, - ) - - -def _douyin_profile_url(sec_user_id: Optional[str]) -> Optional[str]: - sec = (sec_user_id or "").strip() - if not sec: - return None - return f"https://www.douyin.com/user/{sec}" - - -def _profile_detail_to_dict( - profile: AccountProfileDetail | None, - account: Account, - videos: list[AccountVideo] | None = None, -) -> dict[str, Any]: - video_rows = videos or [] - cached_count = len(video_rows) - - def _row_media_type(row: AccountVideo) -> str: - if row.media_type: - return row.media_type - return "video" if row.video_url else "image" - - video_work_count = sum(1 for v in video_rows if _row_media_type(v) == "video") - image_work_count = sum(1 for v in video_rows if _row_media_type(v) == "image") - playable_count = video_work_count - if profile: - api_count = profile.video_count - if cached_count: - display_count = cached_count - else: - display_count = api_count - return { - "account_id": account.id, - "uid": profile.uid or account.douyin_uid, - "nickname": profile.nickname or account.username, - "avatar_url": profile.avatar_url or account.avatar_url, - "unique_id": profile.unique_id, - "signature": profile.signature, - "sec_user_id": profile.sec_user_id, - "profile_url": _douyin_profile_url(profile.sec_user_id), - "video_count": display_count, - "video_count_douyin": api_count, - "cached_work_count": cached_count, - "playable_video_count": playable_count, - "video_work_count": video_work_count, - "image_work_count": image_work_count, - "follower_count": profile.follower_count, - "following_count": profile.following_count, - "total_favorited": profile.total_favorited, - "favoriting_count": profile.favoriting_count, - "fetched": bool(profile.synced_at), - "message": profile.sync_message, - "synced_at": profile.synced_at.isoformat() if profile.synced_at else None, - "profile_aweme_count": api_count, - "videos": [ - { - "id": v.id, - "aweme_id": v.aweme_id, - "title": v.title or "", - "cover_url": v.cover_url, - "video_url": v.video_url, - "share_url": v.share_url, - "create_time": v.create_time.isoformat() if v.create_time else None, - "digg_count": v.digg_count, - "comment_count": v.comment_count, - "play_count": v.play_count, - "media_type": v.media_type, - } - for v in video_rows - ], - } - return { - "account_id": account.id, - "uid": account.douyin_uid, - "nickname": account.username, - "avatar_url": account.avatar_url, - "unique_id": None, - "signature": None, - "sec_user_id": None, - "profile_url": None, - "video_count": None, - "video_count_douyin": None, - "cached_work_count": 0, - "playable_video_count": 0, - "video_work_count": 0, - "image_work_count": 0, - "follower_count": None, - "following_count": None, - "total_favorited": None, - "favoriting_count": None, - "fetched": False, - "message": "暂无本地资料,请点击刷新从抖音同步", - "synced_at": None, - "videos": [], - } - - -async def load_account_profile_from_db( - db: AsyncSession, - account: Account, -) -> dict[str, Any]: - profile = ( - await db.execute( - select(AccountProfileDetail).where(AccountProfileDetail.account_id == account.id) - ) - ).scalar_one_or_none() - videos: list[AccountVideo] = [] - if profile: - videos = list( - ( - await db.execute( - select(AccountVideo) - .where(AccountVideo.account_id == account.id) - .order_by(AccountVideo.sort_order.asc(), AccountVideo.id.asc()) - ) - ).scalars().all() - ) - return _profile_detail_to_dict(profile, account, videos) - - -async def sync_account_profile_to_db( - db: AsyncSession, - account: Account, - cookie_data: str, - max_videos: int = 50, -) -> dict[str, Any]: - """从抖音拉取资料与作品并写入本地数据库。""" - detail = await fetch_douyin_profile_detail_with_sec_user_id( - cookie_data, - account.user_agent, - ) - sec_user_id = (detail.get("sec_user_id") or "").strip() - - if detail.get("sec_user_id_status") == "unknown": - message = ( - detail.get("message") - or "暂时无法核验 sec_user_id,已保留原有账号资料,请稍后重试" - ) - profile = ( - await db.execute( - select(AccountProfileDetail).where( - AccountProfileDetail.account_id == account.id - ) - ) - ).scalar_one_or_none() - if profile: - profile.sync_message = message - await db.commit() - cached = await load_account_profile_from_db(db, account) - cached["message"] = message - return cached - - if detail.get("fetched"): - try: - await apply_douyin_profile(db, account, cookie_data) - except Exception as exc: - logger.warning(f"apply douyin profile failed: {exc}") - - video_result = await fetch_douyin_user_videos( - cookie_data, - sec_user_id, - account.user_agent, - max_count=max_videos, - ) - - now = datetime.utcnow() - profile = ( - await db.execute( - select(AccountProfileDetail).where(AccountProfileDetail.account_id == account.id) - ) - ).scalar_one_or_none() - if not profile: - profile = AccountProfileDetail(account_id=account.id) - db.add(profile) - - profile.uid = detail.get("uid") or account.douyin_uid - profile.nickname = detail.get("nickname") or account.username - profile.avatar_url = detail.get("avatar_url") or account.avatar_url - profile.unique_id = detail.get("unique_id") or None - profile.signature = detail.get("signature") or None - sec_user_id_status = detail.get("sec_user_id_status") - if sec_user_id: - profile.sec_user_id = sec_user_id - elif sec_user_id_status == "missing": - # Only an authoritative successful response may clear the identity. - # Network/parse failures keep the last known value so a profile refresh - # cannot accidentally force a running account offline. - profile.sec_user_id = None - profile.follower_count = detail.get("follower_count") - profile.following_count = detail.get("following_count") - profile.total_favorited = detail.get("total_favorited") - profile.favoriting_count = detail.get("favoriting_count") - profile.synced_at = now - - messages: list[str] = [] - if detail.get("message"): - messages.append(str(detail["message"])) - - fetched_videos = video_result.get("videos") or [] - await db.execute(delete(AccountVideo).where(AccountVideo.account_id == account.id)) - for idx, item in enumerate(fetched_videos): - db.add( - AccountVideo( - account_id=account.id, - aweme_id=item["aweme_id"], - title=item.get("title"), - cover_url=item.get("cover_url"), - video_url=item.get("video_url"), - share_url=item.get("share_url"), - create_time=item.get("create_time"), - digg_count=item.get("digg_count"), - comment_count=item.get("comment_count"), - play_count=item.get("play_count"), - media_type=item.get("media_type"), - sort_order=idx, - synced_at=now, - ) - ) - - profile.video_count = len(fetched_videos) - if fetched_videos: - profile.sync_message = ";".join(messages) if messages else None - else: - empty_msg = str(video_result.get("message") or "主页暂无已公开发布作品") - if messages: - messages.append(empty_msg) - else: - messages = [empty_msg] - profile.sync_message = ";".join(messages) - - await db.commit() - await db.refresh(account) - return await load_account_profile_from_db(db, account) - - -async def _pick_unique_username( - db: AsyncSession, - account: Account, - nickname: str, - uid: str, -) -> str: - candidates: list[str] = [] - if nickname and uid: - candidates.extend([nickname, f"{nickname}_{uid}"]) - elif nickname: - candidates.append(nickname) - elif uid: - candidates.append(f"用户{uid}") - candidates.append(f"{(nickname or '账号')}_{account.id}") - - for candidate in candidates: - name = candidate[:100] - stmt = select(Account.id).where(Account.username == name, Account.id != account.id) - conflict = (await db.execute(stmt)).scalar_one_or_none() - if not conflict: - return name - return f"账号_{account.id}" - - -async def apply_douyin_profile( - db: AsyncSession, - account: Account, - cookie_data: str, -) -> dict[str, str]: - """抓取并写入账号资料,返回抓取结果。""" - profile = await fetch_douyin_profile(cookie_data, account.user_agent) - uid = (profile.get("uid") or "").strip() - nickname = (profile.get("nickname") or "").strip() - avatar = (profile.get("avatar_url") or "").strip() - - if uid: - account.douyin_uid = uid - if avatar: - account.avatar_url = avatar - if nickname or uid: - account.username = await _pick_unique_username(db, account, nickname, uid) - - return profile +"""从 Cookie 抓取抖音账号资料(昵称 / 头像 / UID)及作品列表。""" + +from __future__ import annotations + +import asyncio +import json +import logging +from datetime import datetime +from typing import Any, Optional + +import requests +from sqlalchemy import delete, select +from sqlalchemy.ext.asyncio import AsyncSession + +from models.models import Account, AccountProfileDetail, AccountVideo +from rpa_engine.douyin_im.auth import DouyinAuth +from rpa_engine.douyin_im.dy_util import ( + DEFAULT_USER_AGENT, + generate_a_bogus, + generate_msToken, + generate_webid, + splice_url, +) +from rpa_engine.douyin_im.protocol import _pick_avatar_url +from rpa_engine.douyin_im.session import DouyinImSession +from rpa_engine.device_profiles import resolve_user_agent + +logger = logging.getLogger("account_profile") + + +def _pick_str(data: dict, *keys: str) -> str: + for key in keys: + value = data.get(key) + if value is not None and str(value).strip(): + return str(value).strip() + return "" + + +def _pick_int(data: dict, *keys: str) -> Optional[int]: + for key in keys: + value = data.get(key) + if value is None or value == "": + continue + try: + return int(value) + except (TypeError, ValueError): + continue + return None + + +def _iter_profile_nodes(data: Any): + if not isinstance(data, dict): + return + yield data + for key in ("user", "user_info", "data", "creator"): + node = data.get(key) + if isinstance(node, dict): + yield node + + +def _extract_profile_from_payload(data: Any) -> dict[str, str]: + for node in _iter_profile_nodes(data): + uid = _pick_str(node, "uid", "user_uid", "user_id", "creator_user_id", "id") + nickname = _pick_str( + node, + "nickname", + "nick_name", + "unique_id", + "display_name", + "name", + ) + avatar = _pick_avatar_url(node) + if uid or nickname or avatar: + return { + "uid": uid, + "nickname": nickname, + "avatar_url": avatar, + } + return {} + + +def _extract_detail_from_payload(data: Any) -> dict[str, Any]: + """从抖音资料接口响应中提取详细统计。""" + detail: dict[str, Any] = {} + for node in _iter_profile_nodes(data): + if not detail.get("uid"): + uid = _pick_str(node, "uid", "user_uid", "user_id", "creator_user_id", "id") + if uid: + detail["uid"] = uid + if not detail.get("nickname"): + nickname = _pick_str( + node, "nickname", "nick_name", "display_name", "name" + ) + if nickname: + detail["nickname"] = nickname + if not detail.get("avatar_url"): + avatar = _pick_avatar_url(node) + if avatar: + detail["avatar_url"] = avatar + if not detail.get("unique_id"): + unique_id = _pick_str(node, "unique_id", "short_id", "douyin_id") + if unique_id: + detail["unique_id"] = unique_id + if not detail.get("sec_user_id"): + sec_user_id = _pick_str(node, "sec_uid", "sec_user_id") + if sec_user_id: + detail["sec_user_id"] = sec_user_id + if not detail.get("signature"): + signature = _pick_str(node, "signature", "desc", "bio") + if signature: + detail["signature"] = signature + + for field, keys in ( + ("video_count", ("aweme_count", "post_count", "video_count", "works_count")), + ("follower_count", ("follower_count", "fans_count")), + ("following_count", ("following_count", "follow_count")), + ("total_favorited", ("total_favorited", "total_favorited_count", "like_count")), + ("favoriting_count", ("favoriting_count", "favorite_count")), + ): + if detail.get(field) is None: + val = _pick_int(node, *keys) + if val is not None: + detail[field] = val + return detail + + +def _pick_url_list(data: Any) -> str: + if isinstance(data, str) and data.startswith("http"): + return data + if isinstance(data, dict): + urls = data.get("url_list") or data.get("urlList") or [] + if isinstance(urls, list): + for item in urls: + if isinstance(item, str) and item.startswith("http"): + return item + if isinstance(data, list): + for item in data: + if isinstance(item, str) and item.startswith("http"): + return item + return "" + + +def _pick_video_play_url(video: dict[str, Any]) -> str: + play_addr = video.get("play_addr") or video.get("download_addr") or {} + urls = play_addr.get("url_list") or [] + if not isinstance(urls, list): + urls = [] + for item in urls: + if not isinstance(item, str) or not item.startswith("http"): + continue + lower = item.lower() + if lower.endswith(".mp3") or "/ies-music/" in lower: + continue + if "douyinvod.com" in lower or "/video/" in lower or "mime_type=video" in lower: + return item.replace("playwm", "play") + for item in urls: + if isinstance(item, str) and item.startswith("http") and not item.lower().endswith(".mp3"): + return item.replace("playwm", "play") + bit_rate = video.get("bit_rate") + if isinstance(bit_rate, list): + for entry in bit_rate: + if not isinstance(entry, dict): + continue + play = entry.get("play_addr") or {} + url = _pick_url_list(play) + if url and not url.lower().endswith(".mp3"): + return url.replace("playwm", "play") + return "" + + +def _is_published_aweme(item: dict[str, Any]) -> bool: + """仅保留已公开发布的作品(排除私密/审核中/已删除)。""" + if not isinstance(item, dict): + return False + if item.get("is_private"): + return False + + status = item.get("status") + if isinstance(status, dict): + if status.get("is_delete"): + return False + if status.get("in_reviewing"): + return False + if status.get("is_prohibited"): + return False + private_status = status.get("private_status") + if private_status not in (None, 0, "0"): + return False + part_see = status.get("part_see") + if part_see not in (None, 0, "0", False): + return False + review = status.get("review_result") + if isinstance(review, dict): + review_status = review.get("review_status") + if review_status not in (None, 0, "0"): + return False + + rate = item.get("rate") + if rate in (10, 11, 12): + return False + return True + + +def _parse_aweme_item(item: dict[str, Any]) -> dict[str, Any] | None: + if not isinstance(item, dict): + return None + aweme_id = _pick_str(item, "aweme_id", "awemeId", "item_id") + if not aweme_id: + return None + + title = _pick_str(item, "desc", "title", "content") + video = item.get("video") if isinstance(item.get("video"), dict) else {} + cover_url = _pick_url_list(video.get("cover") or video.get("origin_cover") or item.get("cover")) + video_url = _pick_video_play_url(video) + if not cover_url and isinstance(item.get("images"), list) and item["images"]: + first_image = item["images"][0] + if isinstance(first_image, dict): + cover_url = _pick_url_list(first_image.get("url_list") or first_image) + + share_url = _pick_str(item, "share_url", "share_link") + if not share_url: + share_url = f"https://www.douyin.com/video/{aweme_id}" + + create_time = item.get("create_time") or item.get("createTime") + create_dt: datetime | None = None + if create_time: + try: + create_dt = datetime.utcfromtimestamp(int(create_time)) + except (TypeError, ValueError): + create_dt = None + + statistics = item.get("statistics") if isinstance(item.get("statistics"), dict) else {} + if not title: + title = f"作品 {aweme_id}" + + aweme_type = item.get("aweme_type") + if video_url: + media_type = "video" + elif aweme_type == 68 or item.get("images"): + media_type = "image" + elif aweme_type in (0, 51, 55, 61): + media_type = "video" + else: + media_type = "other" + + return { + "aweme_id": aweme_id, + "title": title, + "cover_url": cover_url, + "video_url": video_url, + "share_url": share_url, + "create_time": create_dt, + "media_type": media_type, + "digg_count": _pick_int(statistics, "digg_count", "like_count"), + "comment_count": _pick_int(statistics, "comment_count"), + "play_count": _pick_int(statistics, "play_count", "view_count"), + } + + +def _parse_creator_item(item: dict[str, Any]) -> dict[str, Any] | None: + if not isinstance(item, dict): + return None + aweme_id = _pick_str(item, "item_id", "item_id_plain", "aweme_id") + if not aweme_id: + return None + title = _pick_str(item, "title", "desc") or f"作品 {aweme_id}" + cover_url = _pick_str(item, "cover_image_url", "cover_url") + share_url = _pick_str(item, "item_link", "share_url") or f"https://www.douyin.com/video/{aweme_id}" + create_time = item.get("create_time") + create_dt: datetime | None = None + if create_time: + try: + create_dt = datetime.utcfromtimestamp(int(create_time)) + except (TypeError, ValueError): + create_dt = None + return { + "aweme_id": aweme_id, + "title": title, + "cover_url": cover_url or None, + "video_url": None, + "share_url": share_url, + "create_time": create_dt, + "media_type": "image" if _pick_int(item, "media_type") == 2 else "other", + "digg_count": _pick_int(item, "digg_count", "like_count"), + "comment_count": _pick_int(item, "comment_count"), + "play_count": _pick_int(item, "play_count", "view_count"), + } + + +def _extract_creator_item_list(data: Any) -> list[dict[str, Any]]: + if not isinstance(data, dict): + return [] + for key in ("item_info_list", "aweme_list", "item_list"): + node = data.get(key) + if isinstance(node, list) and node: + return [x for x in node if isinstance(x, dict)] + return [] + + +def _extract_aweme_list(data: Any) -> list[dict[str, Any]]: + if not isinstance(data, dict): + return [] + for key in ("aweme_list", "awemeList", "item_list", "items", "data"): + node = data.get(key) + if isinstance(node, list) and node: + return [x for x in node if isinstance(x, dict)] + if isinstance(node, dict): + nested = node.get("aweme_list") or node.get("item_list") + if isinstance(nested, list): + return [x for x in nested if isinstance(x, dict)] + return [] + + +def _douyin_get_json( + auth: DouyinAuth, + ua: str, + url: str, + base_params: dict[str, str], + referer: str = "https://www.douyin.com/", +) -> dict[str, Any] | None: + headers = { + "User-Agent": ua, + "Referer": referer, + "Accept": "application/json, text/plain, */*", + } + proxies = _requests_proxies() + try: + params = dict(base_params) + query = splice_url(params) + params["a_bogus"] = generate_a_bogus(query, user_agent=ua) + resp = requests.get( + url, + params=params, + headers=headers, + cookies=auth.cookie, + verify=False, + timeout=15, + proxies=proxies, + ) + data = resp.json() + return data if isinstance(data, dict) else None + except Exception as exc: + logger.debug(f"douyin get failed for {url}: {exc}") + return None + + +def _fetch_aweme_post_page( + auth: DouyinAuth, + ua: str, + sec_user_id: str, + max_count: int, + strategy_type: str, +) -> list[dict[str, Any]]: + base_params = { + "device_platform": "webapp", + "aid": "6383", + "channel": "channel_pc_web", + "sec_user_id": sec_user_id, + "max_cursor": "0", + "count": str(min(max_count, 35)), + "publish_video_strategy_type": strategy_type, + "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "webid": generate_webid(auth, "https://www.douyin.com/"), + "msToken": auth.msToken or generate_msToken(), + "pc_client_type": "1", + } + + videos: list[dict[str, Any]] = [] + max_cursor = 0 + while len(videos) < max_count: + params = dict(base_params) + params["max_cursor"] = str(max_cursor) + params["count"] = str(min(35, max_count - len(videos))) + data = _douyin_get_json( + auth, + ua, + "https://www.douyin.com/aweme/v1/web/aweme/post/", + params, + referer=f"https://www.douyin.com/user/{sec_user_id}", + ) + if not data: + break + + status_code = data.get("status_code") + if status_code not in (None, 0): + break + + batch = _extract_aweme_list(data) + if not batch: + break + + for item in batch: + if not _is_published_aweme(item): + continue + parsed = _parse_aweme_item(item) + if parsed and parsed["aweme_id"] not in {v["aweme_id"] for v in videos}: + videos.append(parsed) + if len(videos) >= max_count: + break + + has_more = bool(data.get("has_more") or data.get("hasMore")) + next_cursor = data.get("max_cursor") or data.get("maxCursor") + try: + next_cursor = int(next_cursor or 0) + except (TypeError, ValueError): + next_cursor = 0 + + if not has_more or next_cursor == max_cursor: + break + max_cursor = next_cursor + return videos + + +def fetch_douyin_user_videos_sync( + cookie_data: str, + sec_user_id: str, + user_agent: Optional[str] = None, + max_count: int = 50, +) -> dict[str, Any]: + """抓取用户已公开发布的作品列表(与抖音主页「作品」一致)。""" + result: dict[str, Any] = {"videos": [], "fetched": False, "message": ""} + sec_user_id = (sec_user_id or "").strip() + try: + auth, ua = _build_auth(cookie_data, user_agent) + except Exception as exc: + result["message"] = f"Cookie 无效: {exc}" + return result + + if not sec_user_id: + result["message"] = "缺少 sec_user_id,无法拉取已发布作品" + return result + + videos = _fetch_aweme_post_page(auth, ua, sec_user_id, max_count, "2") + result["videos"] = videos + result["fetched"] = bool(videos) + if not result["fetched"]: + result["message"] = "未获取到已发布作品,请确认 Cookie 有效且主页有公开作品" + return result + + +def _fetch_own_videos_sync( + auth: DouyinAuth, + ua: str, + max_count: int, +) -> dict[str, Any]: + """创作者中心作品列表(适用于当前登录账号)。""" + result: dict[str, Any] = {"videos": [], "fetched": False} + endpoints = [ + ( + "https://creator.douyin.com/aweme/v1/creator/item/list/", + { + "status": "1", + "count": str(min(max_count, 35)), + "max_cursor": "0", + "aid": "6383", + "device_platform": "webapp", + }, + "https://creator.douyin.com/creator-micro/content/manage", + "creator_item", + ), + ( + "https://creator.douyin.com/aweme/v1/creator/aweme/list/", + { + "status": "1", + "count": str(min(max_count, 35)), + "max_cursor": "0", + "aid": "6383", + "device_platform": "webapp", + }, + "https://creator.douyin.com/creator-micro/content/manage", + "aweme", + ), + ( + "https://creator.douyin.com/web/api/media/aweme/post/", + { + "status": "1", + "count": str(min(max_count, 35)), + "max_cursor": "0", + }, + "https://creator.douyin.com/creator-micro/content/manage", + "aweme", + ), + ] + videos: list[dict[str, Any]] = [] + for url, base_params, referer, parser in endpoints: + data = _douyin_get_json(auth, ua, url, base_params, referer=referer) + if not data: + continue + if parser == "creator_item": + batch = _extract_creator_item_list(data) + parse_fn = _parse_creator_item + else: + batch = _extract_aweme_list(data) + parse_fn = _parse_aweme_item + for item in batch: + parsed = parse_fn(item) + if parsed and parsed["aweme_id"] not in {v["aweme_id"] for v in videos}: + videos.append(parsed) + if len(videos) >= max_count: + break + if videos: + break + result["videos"] = videos + result["fetched"] = bool(videos) + return result + + +def _requests_proxies() -> dict | None: + try: + from rpa_engine.runtime_config import requests_proxies + + return requests_proxies() + except Exception: + return None + + +def _build_auth(cookie_data: str, user_agent: Optional[str] = None) -> tuple[DouyinAuth, str]: + storage = json.loads(cookie_data) + session = DouyinImSession.from_storage_state(storage or {}) + ua = resolve_user_agent(user_agent or session.user_agent or DEFAULT_USER_AGENT) + auth = DouyinAuth() + auth.perepare_auth( + session.cookie_header(), + session.web_protect_str, + session.keys_str, + user_agent=ua, + ) + auth.user_agent = ua + auth.web_id = session.web_id or session.device_id or None + return auth, ua + + +def fetch_douyin_profile_sync( + cookie_data: str, + user_agent: Optional[str] = None, +) -> dict[str, str]: + detail = fetch_douyin_profile_detail_sync(cookie_data, user_agent) + return { + "uid": detail.get("uid") or "", + "nickname": detail.get("nickname") or "", + "avatar_url": detail.get("avatar_url") or "", + } + + +def fetch_douyin_profile_detail_sync( + cookie_data: str, + user_agent: Optional[str] = None, +) -> dict[str, Any]: + """抓取抖音账号详细资料(昵称/头像/UID/作品数/粉丝等)。""" + result: dict[str, Any] = { + "uid": "", + "nickname": "", + "avatar_url": "", + "unique_id": "", + "signature": "", + "sec_user_id": "", + "video_count": None, + "follower_count": None, + "following_count": None, + "total_favorited": None, + "favoriting_count": None, + "fetched": False, + # 抖音明确回「用户未登录」时置位:Cookie 还在,但服务端已判定登录失效。 + "logged_out": False, + "message": "", + } + try: + auth, ua = _build_auth(cookie_data, user_agent) + except Exception as exc: + logger.warning(f"build auth for profile failed: {exc}") + result["message"] = "Cookie 无效,无法解析登录凭证" + return result + + headers = { + "User-Agent": ua, + "Referer": "https://www.douyin.com/", + "Accept": "application/json, text/plain, */*", + } + # (url, params, 是否账号资料源) + # query/user 是「会话/设备」查询接口,不是资料接口:它返回的 id 是浏览器 + # 设备注册号,user_uid 也与账号资料 UID 可以是两个不同的号(实测 + # user_uid=938334054809296 而资料 UID=2609567359568155)。以前它被当成资料 + # 源解析,_pick_str 会把 user_uid 当作 uid,第一个请求就把身份定死了。 + endpoints: list[tuple[str, dict[str, str], bool]] = [ + ( + "https://www.douyin.com/aweme/v1/web/query/user/", + { + "device_platform": "webapp", + "aid": "6383", + "channel": "channel_pc_web", + "publish_video_strategy_type": "2", + "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "webid": generate_webid(auth, "https://www.douyin.com/"), + "msToken": auth.msToken or generate_msToken(), + }, + False, + ), + ( + "https://creator.douyin.com/aweme/v1/creator/user/info/", + { + "device_platform": "webapp", + "aid": "6383", + }, + True, + ), + ( + "https://www.douyin.com/aweme/v1/web/user/profile/self/", + { + "device_platform": "webapp", + "aid": "6383", + "channel": "channel_pc_web", + }, + True, + ), + ] + + proxies = _requests_proxies() + valid_profile_response = False + query_user_uid = "" + for url, base_params, is_profile_source in endpoints: + try: + params = dict(base_params) + query = splice_url(params) + params["a_bogus"] = generate_a_bogus(query, user_agent=ua) + resp = requests.get( + url, + params=params, + headers=headers, + cookies=auth.cookie, + verify=False, + timeout=12, + proxies=proxies, + ) + data = resp.json() + if not isinstance(data, dict): + continue + + status_code = data.get("status_code") + if status_code is not None: + try: + status_code = int(status_code) + except (TypeError, ValueError): + continue + if status_code != 0: + # status_code=8「用户未登录」是抖音的权威答复:Cookie 还在, + # 但服务端已判定登录失效。以前只是 continue,于是同步“成功” + # 却什么都没更新,UI 仍显示「Cookie 有效」,账号资料、 + # sec_user_id、私信收件箱全是空的,排查不到原因。 + if status_code == 8 or "未登录" in str( + data.get("status_msg") or "" + ): + result["logged_out"] = True + continue + + # 会话/设备接口的 user_uid 只是最弱兜底:等所有资料接口都没给出 + # UID 时才用,否则第一个请求就把身份定死,权威资料 UID 永远写不进来。 + if data.get("user_uid") and not query_user_uid: + query_user_uid = str(data["user_uid"]) + + if not is_profile_source: + continue + + basic = _extract_profile_from_payload(data) + stats = _extract_detail_from_payload(data) + payload_has_profile = bool( + basic.get("uid") + or basic.get("nickname") + or basic.get("avatar_url") + or stats.get("uid") + or stats.get("sec_user_id") + or stats.get("unique_id") + ) + if not payload_has_profile: + # 401/风控响应也可能是 JSON;没有任何用户资料节点时 + # 只能视为传输/鉴权未知,不能确认 sec_user_id 缺失。 + continue + valid_profile_response = True + + if basic.get("uid") and not result["uid"]: + result["uid"] = basic["uid"] + if basic.get("nickname") and not result["nickname"]: + result["nickname"] = basic["nickname"] + if basic.get("avatar_url") and not result["avatar_url"]: + result["avatar_url"] = basic["avatar_url"] + + for key, value in stats.items(): + if value is not None and value != "" and result.get(key) in (None, "", 0): + result[key] = value + + if result.get("video_count") is not None: + result["fetched"] = True + break + if result["uid"] and result["nickname"] and result["avatar_url"]: + result["fetched"] = True + except Exception as exc: + logger.debug(f"profile detail fetch failed for {url}: {exc}") + + # UID 优先级:资料接口 > query/user 的 user_uid > cookie/接口兜底。 + # 以前 auth.get_uid() 在请求前就把 result["uid"] 占住,且后面所有赋值都带 + # `not result["uid"]` 守卫,导致权威资料 UID 永远写不进来:账号卡片显示成 + # 「用户」,按这个 UID 反查 sec_user_id 也必然查不到。 + if not result["uid"] and query_user_uid: + result["uid"] = query_user_uid + if not result["uid"]: + fallback_uid = auth.get_uid() + if fallback_uid: + result["uid"] = str(fallback_uid) + + if not result["fetched"] and valid_profile_response: + result["fetched"] = True + + if not result["fetched"]: + if result["logged_out"]: + result["message"] = ( + "抖音返回「用户未登录」:Cookie 仍在但服务端登录态已失效," + "请停止托管后重新扫码登录该账号。" + ) + else: + result["message"] = ( + result["message"] or "未能从抖音获取账号资料,请确认 Cookie 有效" + ) + result["profile_response_valid"] = valid_profile_response + return result + + +async def fetch_douyin_profile( + cookie_data: str, + user_agent: Optional[str] = None, +) -> dict[str, str]: + return await asyncio.to_thread(fetch_douyin_profile_sync, cookie_data, user_agent) + + +async def fetch_douyin_profile_detail( + cookie_data: str, + user_agent: Optional[str] = None, +) -> dict[str, Any]: + return await asyncio.to_thread(fetch_douyin_profile_detail_sync, cookie_data, user_agent) + + +def _resolve_sec_user_id_by_uid_sync( + cookie_data: str, + user_agent: Optional[str], + uid: str, +) -> tuple[str, bool]: + """Return ``(sec_user_id, request_completed)`` for the UID fallback.""" + try: + auth, ua = _build_auth(cookie_data, user_agent) + extra = _douyin_get_json( + auth, + ua, + "https://www.douyin.com/aweme/v1/web/user/profile/other/", + { + "device_platform": "webapp", + "aid": "6383", + "channel": "channel_pc_web", + "user_id": str(uid), + "sec_user_id": "", + "publish_video_strategy_type": "2", + "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "webid": generate_webid(auth, "https://www.douyin.com/"), + "msToken": auth.msToken or generate_msToken(), + }, + ) + except Exception as exc: + logger.debug(f"resolve sec_user_id failed: {exc}") + return "", False + if extra is None: + return "", False + status_code = extra.get("status_code") + if status_code is not None: + try: + if int(status_code) != 0: + return "", False + except (TypeError, ValueError): + return "", False + extracted = _extract_detail_from_payload(extra) + basic = _extract_profile_from_payload(extra) + sec_user_id = (extracted.get("sec_user_id") or "").strip() + if not ( + sec_user_id + or extracted.get("uid") + or basic.get("uid") + or basic.get("nickname") + or basic.get("avatar_url") + ): + return "", False + return sec_user_id, True + + +async def fetch_douyin_profile_detail_with_sec_user_id( + cookie_data: str, + user_agent: Optional[str] = None, +) -> dict[str, Any]: + """Fetch profile data and run the existing UID fallback for sec_user_id. + + ``sec_user_id_status`` distinguishes a confirmed missing field from a + transient request failure so hosting is never stopped merely because the + profile endpoint was temporarily unavailable. + """ + detail = await fetch_douyin_profile_detail(cookie_data, user_agent) + sec_user_id = str(detail.get("sec_user_id") or "").strip() + if sec_user_id: + detail["sec_user_id"] = sec_user_id + detail["sec_user_id_status"] = "found" + return detail + + if not detail.get("fetched"): + detail["sec_user_id_status"] = "unknown" + return detail + + uid = str(detail.get("uid") or "").strip() + if uid: + sec_user_id, request_completed = await asyncio.to_thread( + _resolve_sec_user_id_by_uid_sync, + cookie_data, + user_agent, + uid, + ) + if sec_user_id: + detail["sec_user_id"] = sec_user_id + detail["sec_user_id_status"] = "found" + else: + detail["sec_user_id_status"] = ( + "missing" if request_completed else "unknown" + ) + return detail + + detail["sec_user_id_status"] = "missing" + return detail + + +async def fetch_douyin_user_videos( + cookie_data: str, + sec_user_id: str, + user_agent: Optional[str] = None, + max_count: int = 50, +) -> dict[str, Any]: + return await asyncio.to_thread( + fetch_douyin_user_videos_sync, + cookie_data, + sec_user_id, + user_agent, + max_count, + ) + + +def _douyin_profile_url(sec_user_id: Optional[str]) -> Optional[str]: + sec = (sec_user_id or "").strip() + if not sec: + return None + return f"https://www.douyin.com/user/{sec}" + + +def _profile_detail_to_dict( + profile: AccountProfileDetail | None, + account: Account, + videos: list[AccountVideo] | None = None, +) -> dict[str, Any]: + video_rows = videos or [] + cached_count = len(video_rows) + + def _row_media_type(row: AccountVideo) -> str: + if row.media_type: + return row.media_type + return "video" if row.video_url else "image" + + video_work_count = sum(1 for v in video_rows if _row_media_type(v) == "video") + image_work_count = sum(1 for v in video_rows if _row_media_type(v) == "image") + playable_count = video_work_count + if profile: + api_count = profile.video_count + if cached_count: + display_count = cached_count + else: + display_count = api_count + return { + "account_id": account.id, + "uid": profile.uid or account.douyin_uid, + "nickname": profile.nickname or account.username, + "avatar_url": profile.avatar_url or account.avatar_url, + "unique_id": profile.unique_id, + "signature": profile.signature, + "sec_user_id": profile.sec_user_id, + "profile_url": _douyin_profile_url(profile.sec_user_id), + "video_count": display_count, + "video_count_douyin": api_count, + "cached_work_count": cached_count, + "playable_video_count": playable_count, + "video_work_count": video_work_count, + "image_work_count": image_work_count, + "follower_count": profile.follower_count, + "following_count": profile.following_count, + "total_favorited": profile.total_favorited, + "favoriting_count": profile.favoriting_count, + "fetched": bool(profile.synced_at), + "message": profile.sync_message, + "synced_at": profile.synced_at.isoformat() if profile.synced_at else None, + "profile_aweme_count": api_count, + "videos": [ + { + "id": v.id, + "aweme_id": v.aweme_id, + "title": v.title or "", + "cover_url": v.cover_url, + "video_url": v.video_url, + "share_url": v.share_url, + "create_time": v.create_time.isoformat() if v.create_time else None, + "digg_count": v.digg_count, + "comment_count": v.comment_count, + "play_count": v.play_count, + "media_type": v.media_type, + } + for v in video_rows + ], + } + return { + "account_id": account.id, + "uid": account.douyin_uid, + "nickname": account.username, + "avatar_url": account.avatar_url, + "unique_id": None, + "signature": None, + "sec_user_id": None, + "profile_url": None, + "video_count": None, + "video_count_douyin": None, + "cached_work_count": 0, + "playable_video_count": 0, + "video_work_count": 0, + "image_work_count": 0, + "follower_count": None, + "following_count": None, + "total_favorited": None, + "favoriting_count": None, + "fetched": False, + "message": "暂无本地资料,请点击刷新从抖音同步", + "synced_at": None, + "videos": [], + } + + +async def load_account_profile_from_db( + db: AsyncSession, + account: Account, +) -> dict[str, Any]: + profile = ( + await db.execute( + select(AccountProfileDetail).where(AccountProfileDetail.account_id == account.id) + ) + ).scalar_one_or_none() + videos: list[AccountVideo] = [] + if profile: + videos = list( + ( + await db.execute( + select(AccountVideo) + .where(AccountVideo.account_id == account.id) + .order_by(AccountVideo.sort_order.asc(), AccountVideo.id.asc()) + ) + ).scalars().all() + ) + return _profile_detail_to_dict(profile, account, videos) + + +async def sync_account_profile_to_db( + db: AsyncSession, + account: Account, + cookie_data: str, + max_videos: int = 50, +) -> dict[str, Any]: + """从抖音拉取资料与作品并写入本地数据库。""" + detail = await fetch_douyin_profile_detail_with_sec_user_id( + cookie_data, + account.user_agent, + ) + sec_user_id = (detail.get("sec_user_id") or "").strip() + + if detail.get("sec_user_id_status") == "unknown": + message = ( + detail.get("message") + or "暂时无法核验 sec_user_id,已保留原有账号资料,请稍后重试" + ) + profile = ( + await db.execute( + select(AccountProfileDetail).where( + AccountProfileDetail.account_id == account.id + ) + ) + ).scalar_one_or_none() + if profile: + profile.sync_message = message + await db.commit() + cached = await load_account_profile_from_db(db, account) + cached["message"] = message + return cached + + if detail.get("fetched"): + try: + # 复用刚拿到的 detail,不要再发一次请求:两次抓取会各自走一遍 + # uid 兜底逻辑,任何一次抖动都会让「账号卡片」和「详细资料」写进 + # 不同的 UID/昵称,出现同步成功但卡片没更新的现象。 + await apply_profile_to_account(db, account, detail) + except Exception as exc: + logger.warning(f"apply douyin profile failed: {exc}") + + video_result = await fetch_douyin_user_videos( + cookie_data, + sec_user_id, + account.user_agent, + max_count=max_videos, + ) + + now = datetime.utcnow() + profile = ( + await db.execute( + select(AccountProfileDetail).where(AccountProfileDetail.account_id == account.id) + ) + ).scalar_one_or_none() + if not profile: + profile = AccountProfileDetail(account_id=account.id) + db.add(profile) + + profile.uid = detail.get("uid") or account.douyin_uid + profile.nickname = detail.get("nickname") or account.username + profile.avatar_url = detail.get("avatar_url") or account.avatar_url + profile.unique_id = detail.get("unique_id") or None + profile.signature = detail.get("signature") or None + sec_user_id_status = detail.get("sec_user_id_status") + if sec_user_id: + profile.sec_user_id = sec_user_id + elif sec_user_id_status == "missing": + # Only an authoritative successful response may clear the identity. + # Network/parse failures keep the last known value so a profile refresh + # cannot accidentally force a running account offline. + profile.sec_user_id = None + profile.follower_count = detail.get("follower_count") + profile.following_count = detail.get("following_count") + profile.total_favorited = detail.get("total_favorited") + profile.favoriting_count = detail.get("favoriting_count") + profile.synced_at = now + + messages: list[str] = [] + if detail.get("message"): + messages.append(str(detail["message"])) + + fetched_videos = video_result.get("videos") or [] + await db.execute(delete(AccountVideo).where(AccountVideo.account_id == account.id)) + for idx, item in enumerate(fetched_videos): + db.add( + AccountVideo( + account_id=account.id, + aweme_id=item["aweme_id"], + title=item.get("title"), + cover_url=item.get("cover_url"), + video_url=item.get("video_url"), + share_url=item.get("share_url"), + create_time=item.get("create_time"), + digg_count=item.get("digg_count"), + comment_count=item.get("comment_count"), + play_count=item.get("play_count"), + media_type=item.get("media_type"), + sort_order=idx, + synced_at=now, + ) + ) + + profile.video_count = len(fetched_videos) + if fetched_videos: + profile.sync_message = ";".join(messages) if messages else None + else: + empty_msg = str(video_result.get("message") or "主页暂无已公开发布作品") + if messages: + messages.append(empty_msg) + else: + messages = [empty_msg] + profile.sync_message = ";".join(messages) + + await db.commit() + await db.refresh(account) + return await load_account_profile_from_db(db, account) + + +async def _pick_unique_username( + db: AsyncSession, + account: Account, + nickname: str, + uid: str, +) -> str: + candidates: list[str] = [] + if nickname and uid: + candidates.extend([nickname, f"{nickname}_{uid}"]) + elif nickname: + candidates.append(nickname) + elif uid: + candidates.append(f"用户{uid}") + candidates.append(f"{(nickname or '账号')}_{account.id}") + + for candidate in candidates: + name = candidate[:100] + stmt = select(Account.id).where(Account.username == name, Account.id != account.id) + conflict = (await db.execute(stmt)).scalar_one_or_none() + if not conflict: + return name + return f"账号_{account.id}" + + +async def apply_profile_to_account( + db: AsyncSession, + account: Account, + profile: dict[str, Any], +) -> dict[str, Any]: + """把已抓取的资料写入 accounts 行(昵称/UID/头像)。""" + uid = str(profile.get("uid") or "").strip() + nickname = str(profile.get("nickname") or "").strip() + avatar = str(profile.get("avatar_url") or "").strip() + + if uid: + account.douyin_uid = uid + if avatar: + account.avatar_url = avatar + if nickname or uid: + account.username = await _pick_unique_username(db, account, nickname, uid) + + return profile + + +async def apply_douyin_profile( + db: AsyncSession, + account: Account, + cookie_data: str, +) -> dict[str, str]: + """抓取并写入账号资料,返回抓取结果。""" + profile = await fetch_douyin_profile(cookie_data, account.user_agent) + return await apply_profile_to_account(db, account, profile) diff --git a/backend/rpa_engine/credential.py b/backend/rpa_engine/credential.py index 62674db..92fc883 100644 --- a/backend/rpa_engine/credential.py +++ b/backend/rpa_engine/credential.py @@ -52,7 +52,19 @@ def build_im_session_from_storage( session.keys_str = saved.keys_str if saved.web_protect_str and not session.web_protect_str: session.web_protect_str = saved.web_protect_str - if saved.my_uid and not session.my_uid: + # A UID verified from the account profile must win over collector + # guesses such as web_runtime_security_uid. Persisting this flag + # keeps API/manual-send builders on the same identity as hosting. + if saved.uid_verified and saved.my_uid: + session.my_uid = saved.my_uid + # device_id 必须与 my_uid 指向同一账号:protobuf/frontier 的 + # device_id 优先取 session.device_id,凭证里残留的旧设备号 + # (如 www 域 web_runtime_security_uid)会导致 device_id != my_uid + # -> 安全网关 decision=KICK。用已核验 UID 同步 device_id。 + if str(session.device_id or "") != str(saved.my_uid): + session.device_id = str(saved.my_uid) + session.uid_verified = True + elif saved.my_uid and not session.my_uid: session.my_uid = saved.my_uid if saved.device_id and not session.device_id: session.device_id = saved.device_id diff --git a/backend/rpa_engine/douyin_im/auth.py b/backend/rpa_engine/douyin_im/auth.py index 659c031..734df1b 100644 --- a/backend/rpa_engine/douyin_im/auth.py +++ b/backend/rpa_engine/douyin_im/auth.py @@ -64,8 +64,15 @@ class DouyinAuth: self.msToken = None self.web_id = None self.source_ip = "" + self.user_agent = None - def perepare_auth(self, cookieStr: str, web_protect_: str = "", keys_: str = ""): + def perepare_auth( + self, + cookieStr: str, + web_protect_: str = "", + keys_: str = "", + user_agent: str = "", + ): self.cookie = trans_cookies(cookieStr) self.cookie_str = cookieStr self.msToken = self.cookie["msToken"] if "msToken" in self.cookie else generate_msToken() @@ -89,6 +96,11 @@ class DouyinAuth: except Exception as e: logger.debug(f"keys parse failed: {e}") + if user_agent: + # 让签名上下文记住调用方 UA:query_my_uid / generate_webid 等后续 + # 请求会复用它,避免退回硬编码 DEFAULT_USER_AGENT 造成 UA 不一致。 + self.user_agent = user_agent + def is_sign_ready(self) -> bool: return bool( self.private_key @@ -109,9 +121,11 @@ class DouyinAuth: session.cookie_header(), session.web_protect_str, session.keys_str, + user_agent=session.user_agent or DEFAULT_USER_AGENT, ) auth.web_id = session.web_id or session.device_id or None - auth.user_agent = session.user_agent or DEFAULT_USER_AGENT + # device_id 是设备注册号(query/user 的 id),不是账号 UID; + # my_uid 只作为最后兜底,由 resolve_proto_device_id 内部处理。 auth.device_id = resolve_proto_device_id( session.device_id, session.web_id, session.my_uid ) @@ -139,9 +153,10 @@ class DouyinAuth: return self.uid def query_my_uid(self) -> int: + ua = self.user_agent or DEFAULT_USER_AGENT url = 'https://www.douyin.com/aweme/v1/web/query/user/' headers = { - "User-Agent": DEFAULT_USER_AGENT, + "User-Agent": ua, "Referer": "https://www.douyin.com/", "Accept": "application/json, text/plain, */*", } @@ -156,7 +171,7 @@ class DouyinAuth: "msToken": self.msToken } query = splice_url(params) - abogus = generate_a_bogus(query, user_agent=DEFAULT_USER_AGENT) + abogus = generate_a_bogus(query, user_agent=ua) params['a_bogus'] = abogus with source_bound_requests_session(self.source_ip) as client: diff --git a/backend/rpa_engine/douyin_im/dy_util.py b/backend/rpa_engine/douyin_im/dy_util.py index 866be0e..86c695c 100644 --- a/backend/rpa_engine/douyin_im/dy_util.py +++ b/backend/rpa_engine/douyin_im/dy_util.py @@ -123,17 +123,24 @@ def generate_fake_webid(random_length=19): return random_str -def generate_webid(auth=None, url=""): +def generate_webid(auth=None, url="", user_agent=""): # 优先用已采集到的 web_id(避免每次发送都发起一次阻塞的 HTTP 请求,导致事件循环卡顿) cached = getattr(auth, "web_id", None) if auth is not None else None if cached: return str(cached) if url == "": url = "https://www.douyin.com/discover?modal_id=7376449060384935209" + # UA 优先级:显式参数 > auth.user_agent(from_im_session / perepare_auth 已带)> 全局默认。 + # 必须与 a_bogus 签名、其余请求头使用同一个 UA,否则服务端重算失配 -> 7911。 + ua = ( + user_agent + or (getattr(auth, "user_agent", "") if auth is not None else "") + or DEFAULT_USER_AGENT + ) try: from .auth import DouyinAuth headers = { - "User-Agent": DEFAULT_USER_AGENT, + "User-Agent": ua, "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8", "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", "upgrade-insecure-requests": "1" diff --git a/backend/rpa_engine/douyin_im/follower_poll.py b/backend/rpa_engine/douyin_im/follower_poll.py index f8c0bf5..7d57642 100644 --- a/backend/rpa_engine/douyin_im/follower_poll.py +++ b/backend/rpa_engine/douyin_im/follower_poll.py @@ -1,183 +1,188 @@ -"""抖音网页版「粉丝列表」拉取,用于检测新粉丝(关注欢迎语功能)。 - -复用与 peer_profile / account_profile 相同的 a_bogus + msToken + cookie 签名方式, -调用 https://www.douyin.com/aweme/v1/web/user/follower/list/ 拉取本账号最近的粉丝。 - -返回的每个粉丝含:uid / sec_uid / nickname / follow_status / follower_status。 -其中 follow_status 表示「我」与对方的关系:0=未关注 1=我已关注 2=互相关注(互关)。 -""" - -from __future__ import annotations - -import asyncio -import logging -from typing import Any - -from .dy_util import ( - DEFAULT_USER_AGENT, - generate_a_bogus, - generate_msToken, - generate_webid, - splice_url, -) -from .auth import DouyinAuth - -logger = logging.getLogger("douyin_im.follower_poll") - -FOLLOWER_LIST_URL = "https://www.douyin.com/aweme/v1/web/user/follower/list/" - - -def _requests_proxies() -> dict | None: - try: - from rpa_engine.runtime_config import requests_proxies - - return requests_proxies() - except Exception: - return None - - -def _to_int(value: Any) -> int: - try: - return int(value) - except (TypeError, ValueError): - return 0 - - -def _extract_followers(data: dict[str, Any]) -> list[dict[str, Any]]: - raw = data.get("followers") - if not isinstance(raw, list): - return [] - out: list[dict[str, Any]] = [] - for item in raw: - if not isinstance(item, dict): - continue - uid = str(item.get("uid") or item.get("user_id") or "").strip() - if not uid: - continue - out.append( - { - "uid": uid, - "sec_uid": str(item.get("sec_uid") or item.get("sec_user_id") or "").strip(), - "nickname": str(item.get("nickname") or item.get("nick_name") or "").strip(), - # follow_status:我对对方的关系(2=互关);follower_status:对方对我的关系 - "follow_status": _to_int(item.get("follow_status")), - "follower_status": _to_int(item.get("follower_status")), - } - ) - return out - - -def fetch_recent_followers_sync( - session, - sec_user_id: str, - count: int = 20, - max_time: int = 0, -) -> list[dict[str, Any]]: - """同步拉取最近粉丝(第一页)。失败返回 [],并在日志里写明原因。""" - import requests - - sec_user_id = (sec_user_id or "").strip() - if not sec_user_id: - logger.warning("fetch followers skipped: 缺少本账号 sec_user_id") - return [] - - try: - auth = DouyinAuth() - auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str) - except Exception as exc: - logger.warning("fetch followers: build auth failed: %s", exc) - return [] - - ua = session.user_agent or DEFAULT_USER_AGENT - params = { - "device_platform": "webapp", - "aid": "6383", - "channel": "channel_pc_web", - "sec_user_id": sec_user_id, - "count": str(count), - "max_time": str(max_time), - "min_time": "0", - "offset": "0", - "source_type": "1", - "gps_access": "0", - "address_book_access": "0", - "is_top": "1", - "update_version_code": "170400", - "pc_client_type": "1", - "version_code": "170400", - "version_name": "17.4.0", - "cookie_enabled": "true", - "screen_width": "1536", - "screen_height": "960", - "browser_language": "zh-CN", - "browser_platform": "Win32", - "browser_name": "Chrome", - "browser_version": "120.0.0.0", - "browser_online": "true", - "os_name": "Windows", - "os_version": "10", - "platform": "PC", - "webid": generate_webid(auth, "https://www.douyin.com/"), - "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", - "msToken": auth.msToken or generate_msToken(), - } - query = splice_url(params) - params["a_bogus"] = generate_a_bogus(query, user_agent=ua) - - headers = { - "User-Agent": ua, - "Referer": "https://www.douyin.com/", - "Accept": "application/json, text/plain, */*", - } - try: - resp = requests.get( - FOLLOWER_LIST_URL, - params=params, - headers=headers, - cookies=auth.cookie, - timeout=15, - verify=False, - proxies=_requests_proxies(), - ) - try: - data = resp.json() - except Exception: - snippet = (resp.text or "")[:200].replace("\n", " ") - logger.warning( - "fetch followers: 非 JSON 响应 (HTTP %s): %s", resp.status_code, snippet - ) - return [] - if not isinstance(data, dict): - logger.warning("fetch followers: 响应不是 JSON 对象") - return [] - status_code = data.get("status_code") - if status_code not in (None, 0): - logger.warning( - "fetch followers: status_code=%s msg=%s", - status_code, - data.get("status_msg") or data.get("message") or "", - ) - return [] - followers = _extract_followers(data) - logger.info( - "fetch followers ok: 拿到 %s 个粉丝 (has_more=%s total=%s)", - len(followers), - data.get("has_more"), - data.get("total"), - ) - return followers - except Exception as exc: - logger.warning("fetch followers failed: %s", exc) - return [] - - -async def fetch_recent_followers( - session, - sec_user_id: str, - count: int = 20, - max_time: int = 0, -) -> list[dict[str, Any]]: - return await asyncio.to_thread( - fetch_recent_followers_sync, session, sec_user_id, count, max_time - ) +"""抖音网页版「粉丝列表」拉取,用于检测新粉丝(关注欢迎语功能)。 + +复用与 peer_profile / account_profile 相同的 a_bogus + msToken + cookie 签名方式, +调用 https://www.douyin.com/aweme/v1/web/user/follower/list/ 拉取本账号最近的粉丝。 + +返回的每个粉丝含:uid / sec_uid / nickname / follow_status / follower_status。 +其中 follow_status 表示「我」与对方的关系:0=未关注 1=我已关注 2=互相关注(互关)。 +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any + +from .dy_util import ( + DEFAULT_USER_AGENT, + generate_a_bogus, + generate_msToken, + generate_webid, + splice_url, +) +from .auth import DouyinAuth + +logger = logging.getLogger("douyin_im.follower_poll") + +FOLLOWER_LIST_URL = "https://www.douyin.com/aweme/v1/web/user/follower/list/" + + +def _requests_proxies() -> dict | None: + try: + from rpa_engine.runtime_config import requests_proxies + + return requests_proxies() + except Exception: + return None + + +def _to_int(value: Any) -> int: + try: + return int(value) + except (TypeError, ValueError): + return 0 + + +def _extract_followers(data: dict[str, Any]) -> list[dict[str, Any]]: + raw = data.get("followers") + if not isinstance(raw, list): + return [] + out: list[dict[str, Any]] = [] + for item in raw: + if not isinstance(item, dict): + continue + uid = str(item.get("uid") or item.get("user_id") or "").strip() + if not uid: + continue + out.append( + { + "uid": uid, + "sec_uid": str(item.get("sec_uid") or item.get("sec_user_id") or "").strip(), + "nickname": str(item.get("nickname") or item.get("nick_name") or "").strip(), + # follow_status:我对对方的关系(2=互关);follower_status:对方对我的关系 + "follow_status": _to_int(item.get("follow_status")), + "follower_status": _to_int(item.get("follower_status")), + } + ) + return out + + +def fetch_recent_followers_sync( + session, + sec_user_id: str, + count: int = 20, + max_time: int = 0, +) -> list[dict[str, Any]]: + """同步拉取最近粉丝(第一页)。失败返回 [],并在日志里写明原因。""" + import requests + + sec_user_id = (sec_user_id or "").strip() + if not sec_user_id: + logger.warning("fetch followers skipped: 缺少本账号 sec_user_id") + return [] + + try: + auth = DouyinAuth() + auth.perepare_auth( + session.cookie_header(), + session.web_protect_str, + session.keys_str, + user_agent=session.user_agent or DEFAULT_USER_AGENT, + ) + except Exception as exc: + logger.warning("fetch followers: build auth failed: %s", exc) + return [] + + ua = session.user_agent or DEFAULT_USER_AGENT + params = { + "device_platform": "webapp", + "aid": "6383", + "channel": "channel_pc_web", + "sec_user_id": sec_user_id, + "count": str(count), + "max_time": str(max_time), + "min_time": "0", + "offset": "0", + "source_type": "1", + "gps_access": "0", + "address_book_access": "0", + "is_top": "1", + "update_version_code": "170400", + "pc_client_type": "1", + "version_code": "170400", + "version_name": "17.4.0", + "cookie_enabled": "true", + "screen_width": "1536", + "screen_height": "960", + "browser_language": "zh-CN", + "browser_platform": "Win32", + "browser_name": "Chrome", + "browser_version": "120.0.0.0", + "browser_online": "true", + "os_name": "Windows", + "os_version": "10", + "platform": "PC", + "webid": generate_webid(auth, "https://www.douyin.com/"), + "verifyFp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "fp": auth.cookie.get("s_v_web_id", "") if auth.cookie else "", + "msToken": auth.msToken or generate_msToken(), + } + query = splice_url(params) + params["a_bogus"] = generate_a_bogus(query, user_agent=ua) + + headers = { + "User-Agent": ua, + "Referer": "https://www.douyin.com/", + "Accept": "application/json, text/plain, */*", + } + try: + resp = requests.get( + FOLLOWER_LIST_URL, + params=params, + headers=headers, + cookies=auth.cookie, + timeout=15, + verify=False, + proxies=_requests_proxies(), + ) + try: + data = resp.json() + except Exception: + snippet = (resp.text or "")[:200].replace("\n", " ") + logger.warning( + "fetch followers: 非 JSON 响应 (HTTP %s): %s", resp.status_code, snippet + ) + return [] + if not isinstance(data, dict): + logger.warning("fetch followers: 响应不是 JSON 对象") + return [] + status_code = data.get("status_code") + if status_code not in (None, 0): + logger.warning( + "fetch followers: status_code=%s msg=%s", + status_code, + data.get("status_msg") or data.get("message") or "", + ) + return [] + followers = _extract_followers(data) + logger.info( + "fetch followers ok: 拿到 %s 个粉丝 (has_more=%s total=%s)", + len(followers), + data.get("has_more"), + data.get("total"), + ) + return followers + except Exception as exc: + logger.warning("fetch followers failed: %s", exc) + return [] + + +async def fetch_recent_followers( + session, + sec_user_id: str, + count: int = 20, + max_time: int = 0, +) -> list[dict[str, Any]]: + return await asyncio.to_thread( + fetch_recent_followers_sync, session, sec_user_id, count, max_time + ) diff --git a/backend/rpa_engine/douyin_im/frontier.py b/backend/rpa_engine/douyin_im/frontier.py index d6bf33e..1e8a971 100644 --- a/backend/rpa_engine/douyin_im/frontier.py +++ b/backend/rpa_engine/douyin_im/frontier.py @@ -8,7 +8,7 @@ from urllib.parse import unquote import requests from .auth import DouyinAuth -from .dy_util import generate_a_bogus, generate_msToken, generate_webid, splice_url +from .dy_util import DEFAULT_USER_AGENT, generate_a_bogus, generate_msToken, generate_webid, splice_url from .session import DouyinImSession, is_frontier_ws_url logger = logging.getLogger("douyin_im.frontier") @@ -42,6 +42,7 @@ def fetch_device_id(session: DouyinImSession) -> str: session.cookie_header(), session.web_protect_str, session.keys_str, + user_agent=session.user_agent or DEFAULT_USER_AGENT, ) url = "https://www.douyin.com/aweme/v1/web/query/user" headers = { @@ -134,12 +135,31 @@ def ensure_frontier_ws(session: DouyinImSession) -> Optional[str]: logger.info("Using captured real frontier WS URL (with sdk_cert)") return url + # Only fpid=9 is Douyin private messaging. Other Frontier products (for + # example fpid=971 opened by the generic /chat page shell) also handshake + # successfully but never carry this account's IM push stream. + for url in session.ws_urls: + if ( + is_frontier_ws_url(url) + and "zijieapi.com" in url + and "access_key=" in url + and re.search(r"[?&]fpid=9(?:&|$)", url) + and _ws_device_matches_session(session, url) + ): + session.ws_urls = [url] + logger.info("Using captured browser frontier WebSocket URL") + return url + for url in session.ws_urls: if is_frontier_ws_url(url) and _ws_token_looks_encoded(url): session.ws_urls = [url] logger.info("Using captured frontier WS URL") return url + # frontier 按 device_id 寻址推送,它不是账号 UID:抖音 query/user 返回的 + # id 才是本浏览器的设备注册号(my_uid 走 DouyinAuth,两者不能互换)。 + # 用 my_uid 拼出来的地址握手同样成功,但订阅的是另一个地址, + # 于是长连接一直是「连上但收不到任何私信」。 device_id = resolve_frontier_device_id(session) if not device_id: session.ws_urls = [] diff --git a/backend/rpa_engine/douyin_im/http_client.py b/backend/rpa_engine/douyin_im/http_client.py index 5db2576..8ed6b90 100644 --- a/backend/rpa_engine/douyin_im/http_client.py +++ b/backend/rpa_engine/douyin_im/http_client.py @@ -1,6 +1,8 @@ import asyncio import json import logging +import re +import time from typing import Any, Optional from urllib.parse import urlparse @@ -13,8 +15,9 @@ from rpa_engine.egress_channels import ( resolve_send_channels, ) from .conv_util import build_conversation_id, normalize_conversation_id, resolve_peer_uid +from .message_content import format_im_message, serialize_message_content from .peer_profile import enrich_conversation_item, fetch_peer_profile, is_generic_peer_name -from .protocol import normalize_im_payload, normalize_im_payload_from_bytes, _pick_avatar_url +from .protocol import normalize_im_payload_from_bytes, _pick_avatar_url from .session import DouyinImSession logger = logging.getLogger("douyin_im.http") @@ -23,6 +26,27 @@ IMAPI_BASE = "https://imapi.douyin.com" # 抖音 IM「按会话拉取消息」cmd(与电商/web 一致);body 字段号 == cmd。 CMD_GET_MESSAGES_BY_CONVERSATION = 301 +# 「按用户拉取收件箱」cmd:抖音网页版打开私信时用它一次性同步各会话最新消息。 +CMD_GET_MESSAGES_BY_USER_INIT = 200 + +# MessageBody 的字段号(与 Response.proto 的 MessageBody 一致)。 +_MSG_FIELD_CONVERSATION_ID = 1 +_MSG_FIELD_SERVER_MESSAGE_ID = 3 +_MSG_FIELD_CONVERSATION_SHORT_ID = 5 +_MSG_FIELD_MESSAGE_TYPE = 6 +_MSG_FIELD_SENDER = 7 +_MSG_FIELD_CONTENT = 8 + +_CONVERSATION_ID_RE = re.compile(r"^0:\d+:\d+:\d+$") + +# 托管轮询只需覆盖长连接断开的时间窗;游标是微秒时间戳,窗口越小响应越小 +# (实测同一账号:游标 0 -> 113KB,回看 30 分钟 -> 约 2KB)。 +INBOX_POLL_LOOKBACK_SECONDS = 1800.0 + +# 会话列表被抖音拒绝时的系统日志节流:托管期间每个账号最多每 30 分钟记一次, +# 既保证「收不到私信」有据可查,又不会被每轮轮询刷屏。 +_CONVERSATION_REJECT_LOG_INTERVAL = 1800.0 +_conversation_reject_logged_at: dict[tuple[int, str], float] = {} def _pb_varint(n: int) -> bytes: @@ -94,6 +118,173 @@ def _pb_parse_fields(buf: bytes) -> list[tuple[int, int, Any]]: return out +def _as_int(value: Any) -> int: + try: + return int(str(value or 0)) + except (TypeError, ValueError): + return 0 + + +def _pb_message_body(buf: bytes) -> Optional[dict]: + """把一段字节按 MessageBody 解析;形状不像就返回 None。 + + 判据是「有合法的 conversation_id + server_message_id」,而不是它出现在 + 哪个字段号上——收件箱响应里 messages 挂在哪一层随接口而变。 + """ + msg: dict = {} + try: + fields = _pb_parse_fields(buf) + except Exception: + return None + for fn, wt, val in fields: + if fn == _MSG_FIELD_CONVERSATION_ID and wt == 2: + try: + conv_id = val.decode("utf-8") + except Exception: + return None + if not _CONVERSATION_ID_RE.match(conv_id): + return None + msg["conversation_id"] = conv_id + elif fn == _MSG_FIELD_SERVER_MESSAGE_ID and wt == 0: + msg["server_message_id"] = str(val) + elif fn == _MSG_FIELD_CONVERSATION_SHORT_ID and wt == 0: + msg["conversation_short_id"] = str(val) + elif fn == _MSG_FIELD_MESSAGE_TYPE and wt == 0: + msg["message_type"] = val + elif fn == _MSG_FIELD_SENDER and wt == 0: + msg["sender"] = str(val) + elif fn == _MSG_FIELD_CONTENT and wt == 2: + msg["content"] = val.decode("utf-8", errors="replace") + if msg.get("conversation_id") and msg.get("server_message_id"): + return msg + return None + + +def _pb_collect_message_bodies( + buf: bytes, + out: list[dict], + depth: int = 0, +) -> None: + """递归找出响应体里所有 MessageBody。""" + if depth > 6: + return + parsed = _pb_message_body(buf) + if parsed is not None: + out.append(parsed) + return + try: + fields = _pb_parse_fields(buf) + except Exception: + return + for _fn, wt, val in fields: + if wt == 2 and isinstance(val, bytes) and val: + _pb_collect_message_bodies(val, out, depth + 1) + + +def _pb_parse_inbox_messages(content: bytes, cmd: int) -> list[dict]: + """从 get_by_user_init 响应里取出各会话的最新消息。""" + out: list[dict] = [] + for fn, wt, val in _pb_parse_fields(content): + if fn != 6 or wt != 2: # Response.body + continue + for bfn, bwt, bval in _pb_parse_fields(val): + if bfn != cmd or bwt != 2: # ResponseBody. + continue + _pb_collect_message_bodies(bval, out) + return out + + +def _is_inbox_control_message(msg: dict) -> bool: + """收件箱里的会话控制/状态帧(不是用户发的消息)。""" + from .protocol import _is_control_payload + + try: + message_type = int(msg.get("message_type") or 0) + except (TypeError, ValueError): + message_type = 0 + content_json: Any = None + raw = msg.get("content") + if raw: + try: + content_json = json.loads(raw) + except Exception: + content_json = None + return _is_control_payload(content_json, message_type) + + +def _pb_parse_inbox_conversations(content: bytes, cmd: int) -> list[dict]: + """取出 cmd 200 响应里的会话条目。 + + 响应体除了 messages(字段 1) 还带一组会话条目(字段 6): + f1=conversation_short_id(varint) f4=conversation_id(string) + 游标为 0 时这组条目就是账号的完整会话列表,所以「列全部会话」不必再去拉 + cmd 203 的 1.5MB 全量快照。 + """ + out: list[dict] = [] + for fn, wt, val in _pb_parse_fields(content): + if fn != 6 or wt != 2: # Response.body + continue + for bfn, bwt, bval in _pb_parse_fields(val): + if bfn != cmd or bwt != 2: + continue + for cfn, cwt, cval in _pb_parse_fields(bval): + if cfn != 6 or cwt != 2: # repeated conversation entry + continue + short_id = "" + conv_id = "" + for efn, ewt, eval_ in _pb_parse_fields(cval): + if efn == 1 and ewt == 0: + short_id = str(eval_) + elif efn == 4 and ewt == 2: + try: + candidate = eval_.decode("utf-8") + except Exception: + continue + if _CONVERSATION_ID_RE.match(candidate): + conv_id = candidate + if conv_id: + out.append( + { + "conversation_id": conv_id, + "conversation_short_id": short_id, + } + ) + return out + + +def _pb_parse_inbox_page(content: bytes, cmd: int) -> tuple[int, bool]: + """返回收件箱这一页的 (next_cursor, has_more)。""" + next_cursor = 0 + has_more = False + for fn, wt, val in _pb_parse_fields(content): + if fn != 6 or wt != 2: + continue + for bfn, bwt, bval in _pb_parse_fields(val): + if bfn != cmd or bwt != 2: + continue + for cfn, cwt, cval in _pb_parse_fields(bval): + if cfn == 2 and cwt == 0: + next_cursor = int(cval) + elif cfn == 3 and cwt == 0: + has_more = bool(cval) + return next_cursor, has_more + + +def _pb_response_status(content: bytes) -> tuple[Optional[int], str]: + """返回 IM protobuf 响应的 (status_code, message)。""" + status: Optional[int] = None + message = "" + try: + for fn, wt, val in _pb_parse_fields(content): + if fn == 3 and wt == 0: + status = int(val) + elif fn == 4 and wt == 2: + message = val.decode("utf-8", errors="replace") + except Exception: + return None, "" + return status, message + + def _pb_parse_conversation_messages(content: bytes, cmd: int) -> list[dict]: """解析 get_by_conversation 的 protobuf 响应,返回消息列表。""" out: list[dict] = [] @@ -350,6 +541,13 @@ class DouyinImHttpClient: # another channel. Ambiguous read timeouts stay false to avoid duplicates. self.last_send_channel_retryable: bool = False self.last_request_debug: str = "" + # 会话列表被抖音判定为「请求本身不合法」时置位。换 payload、换 cookie + # 都修不好,上层据此停掉这轮轮询,别每 120 秒白打一次请求。 + self.conversation_list_unsupported: bool = False + # 最近一次收件箱响应里的会话条目(只覆盖翻到的那些页) + self._last_inbox_conversations: list[dict] = [] + # 翻页预算用尽但抖音还说 has_more:这次拿到的会话列表不完整 + self.inbox_truncated: bool = False self._proxy_url: str = "" self._source_ip_override = str(source_ip or "").strip() self._egress_public_ip_override = str(egress_public_ip or "").strip() @@ -448,16 +646,43 @@ class DouyinImHttpClient: def _set_error(self, msg: str) -> None: self.last_error = msg or "" - def _resolve_authoritative_uid(self, auth) -> int: - """用 query/user 接口核验当前账号真实 UID,并回写 session.my_uid。 + def _report_conversation_list_rejected(self, reason: str) -> None: + """把「会话列表被抖音拒绝」变成可见故障,而不是静默的空收件箱。""" + self._set_error(f"会话列表接口被抖音拒绝:{reason}") + self.conversation_list_unsupported = True + logger.warning( + "Conversation list rejected by IM API (account=%s): %s", + self.account_id, + reason, + ) + key = (int(self.account_id or 0), reason) + now = time.monotonic() + last_at = _conversation_reject_logged_at.get(key) + if last_at is not None and now - last_at < _CONVERSATION_REJECT_LOG_INTERVAL: + return + _conversation_reject_logged_at[key] = now + system_logger.record( + "会话列表接口被抖音拒绝,已停用轮询兜底", + detail=( + f"抖音返回:{reason}。该请求本身被判定为不合法,重试也修不好," + "本轮托管不再重复调用。私信改为完全依赖实时长连接接收;" + "长连接断开期间漏收的消息无法再通过轮询补齐。" + ), + level="error", + category="poll", + account_id=self.account_id, + ) - 采集端从 tea_cache 推断的 my_uid 可能取到访客/对方 id(导致 cmd=609 - INVALID_REQUEST、会话列表为 0)。query/user 返回的 user_uid 才是权威值。 - 核验成功后写回 session 并打标,避免每次发送都请求接口。 + def _resolve_authoritative_uid(self, auth) -> int: + """Resolve a usable UID without overwriting a known IM identity. + + Douyin's query/user ``user_uid`` can differ from the UID used by IM. + It is therefore only a last-resort value when the session has no UID; + a profile-verified or collected numeric UID always wins. """ sess = self.session auth.source_ip = self._source_ip - if getattr(sess, "uid_verified", False) and sess.my_uid: + if sess.my_uid: return int(sess.my_uid) resolved = None try: @@ -466,16 +691,10 @@ class DouyinImHttpClient: logger.warning(f"query/user 解析 my_uid 失败: {e}") if resolved and str(resolved).isdigit(): resolved = int(resolved) - old = int(sess.my_uid or 0) - if old and old != resolved: - logger.warning("my_uid 修正(query/user):采集值 %s -> 权威值 %s", old, resolved) sess.my_uid = resolved - # 本系统里 device_id 等同账号 uid(采集端常与 my_uid 一起取错,导致会话列表为 0)。 - # device_id 为空 / 非数字 / 等于旧的错误 my_uid 时,一并修正为权威 uid。 dev = str(sess.device_id or "") - if (not dev.isdigit()) or (old and dev == str(old)): + if not dev.isdigit(): sess.device_id = str(resolved) - sess.uid_verified = True return resolved return int(sess.my_uid or 0) @@ -734,7 +953,9 @@ class DouyinImHttpClient: cmd = CMD_GET_MESSAGES_BY_CONVERSATION try: - request = await asyncio.to_thread(ProtoBuilder.build_normal_request, auth, cmd) + # 同样要用 x_tt_token:带 auth.ticket 时抖音回 OK 但正文恒为空, + # 于是 WS 瘦推送的图片/语音一直补不全真实 content。 + request = await asyncio.to_thread(ProtoBuilder.build_read_request, auth, cmd) conv_req = ( _pb_str(1, conversation_id) + _pb_int(2, 1) @@ -815,83 +1036,207 @@ class DouyinImHttpClient: pass return total - async def get_conversations(self, *, enrich_profiles: bool = True) -> list[dict]: - """拉取会话列表,返回标准化会话""" - payloads = [ - {"cursor": 0, "count": 50, "inbox_type": 0}, - {"cursor": 0, "limit": 50}, - {}, - ] - conversations = [] - seen = set() + async def fetch_inbox_messages( + self, + limit: int = 50, + lookback_seconds: float = INBOX_POLL_LOOKBACK_SECONDS, + max_pages: int = 1, + ) -> list[dict]: + """用 protobuf 拉取收件箱消息,按游标翻页。 - for body in payloads: - data = await self._request("POST", "/v1/conversation/list", body) - # Only a transport/parse failure warrants trying GET. An empty - # JSON object/list can be a perfectly valid empty inbox response. - if data is None: - data = await self._request("GET", "/v1/conversation/list", body) - if data is None: - # Payload variants only help with schema compatibility. They - # cannot repair a network outage, so stop after POST + GET - # both fail instead of occupying a scarce global slot for up - # to four more full request timeouts. - logger.warning("Conversation poll transport failed; skipping payload fallbacks") - break + imapi.douyin.com 只接受 protobuf:发 JSON body 会被当成 protobuf 解析, + 固定返回 status_code=1 "unexepcted session length"(与 Cookie 无关, + 实测不带任何 Cookie 也是同一条错误)。这里用与发送/拉消息同一套 Request + 信封,抖音网页版打开私信时用的也是这个 cmd。 - status_code = data.get("status_code") if isinstance(data, dict) else None - error_text = "" - if isinstance(data, dict): - error_text = str( - data.get("error_desc") - or data.get("message") - or data.get("error") - or "" - ).strip().lower() - explicit_success = status_code in (0, "0") - structured_without_status = ( - isinstance(data, (dict, list)) and status_code is None + 响应是**分页**的:body 的 f2=next_cursor、f3=has_more,随附的会话条目 + 只覆盖这一页里出现过的会话。实测 cursor=0 返回 9 个会话且 has_more=1, + 翻 6 页后累计 35 个且仍未翻完——所以「一次请求 = 完整会话列表」是错的, + 翻不完时必须把 inbox_truncated 置位,别把一页伪装成全部。 + """ + from .auth import DouyinAuth + from .proto_builder import ProtoBuilder + + cmd = CMD_GET_MESSAGES_BY_USER_INIT + auth = DouyinAuth.from_im_session(self.session) + auth.source_ip = self._source_ip + # 字段 1 是游标(微秒时间戳)。托管轮询只回看一个窗口:这条链路只用来 + # 补齐长连接断开期间漏收的消息。lookback_seconds<=0 表示不设游标 + # (游标 0 = 从头翻),别算成「now」,那等于只要比此刻更新的消息。 + if lookback_seconds <= 0: + cursor = 0 + else: + cursor = max(0, int((time.time() - lookback_seconds) * 1_000_000)) + + messages: list[dict] = [] + conversations: list[dict] = [] + seen_conversations: set[str] = set() + self.inbox_truncated = False + pages = max(1, int(max_pages)) + for page in range(pages): + request = await asyncio.to_thread( + ProtoBuilder.build_read_request, auth, cmd ) - terminal_credential_error = ( - status_code not in (None, 0, "0") - and any( - marker in error_text - for marker in ( - "empty token", - "invalid token", - "token expired", - "credential expired", - "authentication", - "unauthorized", - "not login", - "not logged", - ) + body = _pb_int(1, cursor) + _pb_int(2, int(limit)) + payload = request.SerializeToString() + _pb_msg(8, _pb_msg(cmd, body)) + resp = await self._post_protobuf( + f"{IMAPI_BASE}/v1/message/get_by_user_init", + auth, + payload, + signed=False, + log_label="inbox", + ) + resp.raise_for_status() + status_code, message = _pb_response_status(resp.content) + if status_code is not None and status_code != 0: + self._report_conversation_list_rejected( + message or f"status_code={status_code}" ) - ) + return [] + if page == 0: + self._adopt_authoritative_uid(resp.content) + messages.extend(_pb_parse_inbox_messages(resp.content, cmd)) + for entry in _pb_parse_inbox_conversations(resp.content, cmd): + conv_id = str(entry.get("conversation_id") or "") + if conv_id and conv_id not in seen_conversations: + seen_conversations.add(conv_id) + conversations.append(entry) - normalized = normalize_im_payload(data) - for item in normalized: - name = item.get("sender_name") or "" - key = name or item.get("conversation_id") or "" - if key and key not in seen: - seen.add(key) - conversations.append(item) - - # 也从原始结构提取会话级 unread - self._extract_conversation_rows(data, conversations, seen) - - # Compatibility payloads are alternatives, not pagination. Stop - # after a successful empty response as well as a non-empty one; - # otherwise every idle account issues three identical endpoint - # calls on every poll. Credential errors cannot be repaired by - # changing only the JSON shape, so do not amplify those either. - if ( - conversations - or explicit_success - or structured_without_status - or terminal_credential_error - ): + next_cursor, has_more = _pb_parse_inbox_page(resp.content, cmd) + if not has_more: break + # 游标不前进就停:否则同一页会被无限翻下去。 + if not next_cursor or next_cursor == cursor: + break + cursor = next_cursor + if page == pages - 1: + self.inbox_truncated = True + logger.info( + "Inbox paging stopped at the %d-page budget for account %s; " + "%d conversations so far, more remain", + pages, + self.account_id, + len(conversations), + ) + + self._last_inbox_conversations = conversations + return messages + + def _adopt_authoritative_uid(self, content: bytes) -> None: + """响应字段 13 是抖音认定的本账号 IM uid,用它纠正 session.my_uid。 + + my_uid 取错时 _is_self_message 拦不住自己发的消息(机器人会自问自答), + resolve_peer_uid 也会把会话对端认成自己。 + """ + uid = 0 + try: + for fn, wt, val in _pb_parse_fields(content): + if fn == 13 and wt == 0: + uid = int(val) + break + except Exception: + return + if not uid or uid == int(self.session.my_uid or 0): + return + logger.warning( + "Account %s IM uid corrected from %s to %s (imapi response field 13)", + self.account_id, + self.session.my_uid, + uid, + ) + self.session.my_uid = uid + self.session.uid_verified = True + + async def get_conversations( + self, + *, + enrich_profiles: bool = True, + lookback_seconds: float = INBOX_POLL_LOOKBACK_SECONDS, + max_pages: int = 1, + ) -> list[dict]: + """拉取会话列表,返回标准化会话。 + + lookback_seconds=0 表示从头翻;max_pages 是翻页预算。抖音不提供「一次 + 取回全部会话」的接口,翻页预算用完时 inbox_truncated 会被置位——调用方 + 必须知道自己拿到的可能只是一部分,不能把一页当成完整会话列表。 + 托管轮询用默认的小窗口 + 单页,只为补齐长连接断开期间漏收的消息。 + """ + conversations: list[dict] = [] + try: + messages = await self.fetch_inbox_messages( + lookback_seconds=lookback_seconds, + max_pages=max_pages, + ) + except Exception as exc: + self._set_error(str(exc)) + logger.warning("Conversation poll transport failed: %s", exc) + return [] + + # 一个会话只保留最新的一条:server_message_id 单调递增。 + latest: dict[str, dict] = {} + for msg in messages: + conv_id = str(msg.get("conversation_id") or "") + if not conv_id: + continue + # 会话状态/已读位等控制帧不是用户消息:既不该当成会话预览, + # 更不该被 _handle_incoming 拿去匹配自动回复(WS 侧同样过滤)。 + if _is_inbox_control_message(msg): + continue + current = latest.get(conv_id) + if current is None or _as_int(msg.get("server_message_id")) > _as_int( + current.get("server_message_id") + ): + latest[conv_id] = msg + + # 窗口内没有消息、但账号里确实存在的会话也要出现在列表里, + # 否则「会话列表」会退化成「最近有动静的会话」。 + for entry in self._last_inbox_conversations: + conv_id = str(entry.get("conversation_id") or "") + short_id = str(entry.get("conversation_short_id") or "") + if short_id and short_id != "0": + self.session.conv_meta.setdefault(conv_id, {}) + self.session.conv_meta[conv_id]["conversation_short_id"] = short_id + if conv_id and conv_id not in latest: + latest[conv_id] = { + "conversation_id": conv_id, + "conversation_short_id": short_id, + "server_message_id": "0", + "content": "", + } + + for conv_id, msg in latest.items(): + content = str(msg.get("content") or "") + try: + message_type = int(msg.get("message_type") or 0) + except (TypeError, ValueError): + message_type = 0 + preview = content + try: + parsed = format_im_message(json.loads(content), message_type) + preview = serialize_message_content(parsed) if parsed else content + except Exception: + pass + short_id = str(msg.get("conversation_short_id") or "") + if short_id and short_id != "0": + # 发送私信需要 short_id;从收件箱顺手补上,省掉一次 create。 + self.session.conv_meta.setdefault(conv_id, {}) + self.session.conv_meta[conv_id]["conversation_short_id"] = short_id + conversations.append( + { + "conversation_id": conv_id, + "conversation_short_id": short_id, + "sender_name": "", + "sender_avatar": None, + "content": preview, + "raw_content": content, + "message_type": message_type, + "server_message_id": str(msg.get("server_message_id") or ""), + # sender 可能是自己(我方发出的最后一条),不能当 peer; + # 交给 enrich_conversation_item 从 conversation_id 推。 + "sender_uid": str(msg.get("sender") or ""), + "unread_count": 0, + } + ) my_uid = int(self.session.my_uid or 0) enriched: list[dict] = [] @@ -917,51 +1262,6 @@ class DouyinImHttpClient: logger.info(f"Fetched {len(enriched)} conversations from IM API") return enriched - def _extract_conversation_rows(self, data: Any, out: list, seen: set, depth: int = 0): - if depth > 10: - return - if isinstance(data, dict): - name = ( - data.get("nick_name") - or data.get("nickname") - or ( - (data.get("core_info") or {}).get("nick_name") - if isinstance(data.get("core_info"), dict) - else None - ) - ) - unread = data.get("unread_count") or data.get("unreadCount") or 0 - conv_id = data.get("conversation_id") or data.get("conversationId") or "" - preview = "" - last = data.get("last_message") or data.get("latest_message") - if isinstance(last, dict): - preview = last.get("content") or last.get("text") or "" - elif isinstance(last, str): - preview = last - - if isinstance(name, str) and name.strip(): - name = name.strip() - sender_avatar = _pick_avatar_url(data) - if name not in seen: - try: - unread = int(unread or 0) - except (TypeError, ValueError): - unread = 0 - out.append({ - "sender_name": name, - "sender_avatar": sender_avatar or None, - "content": str(preview or ""), - "conversation_id": str(conv_id or ""), - "unread_count": unread, - }) - seen.add(name) - - for v in data.values(): - self._extract_conversation_rows(v, out, seen, depth + 1) - elif isinstance(data, list): - for item in data: - self._extract_conversation_rows(item, out, seen, depth + 1) - async def send_text_message( self, conversation_id: str, @@ -1213,7 +1513,7 @@ class DouyinImHttpClient: self.last_send_channel_retryable = True detail = ( "抖音安全网关返回 decision=KICK,当前登录/安全会话已被服务端踢下线;" - "请停止托管后用浏览器模式重新登录,并打开一次私信页重新采集凭证" + "系统正在自动重登录,请留意账号卡片上的二维码并扫码" ) elif decision: detail = f"抖音安全网关拒绝发送 decision={decision}" diff --git a/backend/rpa_engine/douyin_im/image_upload.py b/backend/rpa_engine/douyin_im/image_upload.py index d935086..16cfe01 100644 --- a/backend/rpa_engine/douyin_im/image_upload.py +++ b/backend/rpa_engine/douyin_im/image_upload.py @@ -283,8 +283,13 @@ def _fetch_im_upload_sts(session, source_ip: str = "") -> tuple[str, str, str, s ) auth = DouyinAuth() - auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str) ua = session.user_agent or DEFAULT_USER_AGENT + auth.perepare_auth( + session.cookie_header(), + session.web_protect_str, + session.keys_str, + user_agent=ua, + ) params = { "device_platform": "webapp", diff --git a/backend/rpa_engine/douyin_im/peer_profile.py b/backend/rpa_engine/douyin_im/peer_profile.py index ae4ad87..c8554bd 100644 --- a/backend/rpa_engine/douyin_im/peer_profile.py +++ b/backend/rpa_engine/douyin_im/peer_profile.py @@ -73,9 +73,14 @@ def _requests_proxies() -> dict | None: def _build_auth(session: DouyinImSession) -> tuple[DouyinAuth, str]: - auth = DouyinAuth() - auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str) ua = resolve_user_agent(session.user_agent or DEFAULT_USER_AGENT) + auth = DouyinAuth() + auth.perepare_auth( + session.cookie_header(), + session.web_protect_str, + session.keys_str, + user_agent=ua, + ) return auth, ua diff --git a/backend/rpa_engine/douyin_im/proto_builder.py b/backend/rpa_engine/douyin_im/proto_builder.py index 83c306a..c21b871 100644 --- a/backend/rpa_engine/douyin_im/proto_builder.py +++ b/backend/rpa_engine/douyin_im/proto_builder.py @@ -59,6 +59,24 @@ class ProtoBuilder: request.sdk_cert = normalize_client_cert(auth.client_cert or "") return request + @staticmethod + def build_read_request(auth, cmd): + """读接口(收件箱/会话消息)的 Request 信封。 + + Request.token 必须是 x_tt_token cookie。build_normal_request 填的是 + auth.ticket(bd-ticket-guard 票据):长度合法,服务端照样回 + status_code=0 "OK",但把调用方当成匿名用户,body 恒为空——和「收件箱 + 里没有消息」完全无法区分。实测同一请求只换 token: + auth.ticket -> 73 字节 0 条;x_tt_token -> 113KB 47 条。 + 发送接口另有签名,沿用 build_normal_request,不在此处改动。 + """ + request = ProtoBuilder.build_normal_request(auth, cmd) + cookies = getattr(auth, "cookie", None) or {} + token = str(cookies.get("x_tt_token") or "").strip() + if token: + request.token = token + return request + @staticmethod def build_create_conversation_request(auth, toId, myId): request = ProtoBuilder.build_normal_request(auth, 609) diff --git a/backend/rpa_engine/douyin_im/protocol.py b/backend/rpa_engine/douyin_im/protocol.py index 0cbcd31..bd49e8f 100644 --- a/backend/rpa_engine/douyin_im/protocol.py +++ b/backend/rpa_engine/douyin_im/protocol.py @@ -204,9 +204,46 @@ def extract_json_objects(raw: bytes | str) -> list[dict]: return results +def _looks_like_push_frame(frame) -> bool: + """判断这段字节确实是 frontier 的 PushFrame 信封。 + + 真实帧一定带 service/method 和 frontier 自己的 headers/traceid;随手一段 + 二进制偶尔也能被 protobuf 宽松解析成 PushFrame,那种不算。 + """ + return bool( + frame.service + or frame.method + or frame.payloadType + or frame.payloadEncoding + or frame.logIdNew + or len(frame.headersList) + ) + + +def _decode_push_frame_payload(frame) -> bytes: + """取出 PushFrame 内层负载,按 payloadEncoding 解压。 + + frontier 会用 gzip 压缩 payload;直接把压缩字节喂给 Response.ParseFromString + 只会抛异常并被吞掉,整条私信就此丢失。 + """ + body = bytes(frame.payload or b"") + if not body: + return b"" + encoding = str(frame.payloadEncoding or "").lower() + if encoding in ("gzip", "gz"): + try: + return gzip.decompress(body) + except Exception as exc: + logger.warning("Failed to gunzip frontier frame payload: %s", exc) + return body + return body + + def parse_ws_payload(raw: bytes | str) -> list[dict]: """解析 WebSocket 二进制帧,返回标准化消息 dict 列表""" messages = [] + frame_payload = b"" + is_push_frame = False # 尝试 Protobuf 解包 if isinstance(raw, bytes): @@ -214,9 +251,14 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]: from .static import Live_pb2, Response_pb2 frame = Live_pb2.PushFrame() frame.ParseFromString(raw) - if frame.payloadType == 'pb': + is_push_frame = _looks_like_push_frame(frame) + frame_payload = _decode_push_frame_payload(frame) + # payloadType 不再作为判据:现网 frontier 帧会带 'pb'、'text/json' + # 或空值,之前只认 'pb' 会把其余帧整帧丢弃。真正的判据是解出来 + # 有没有 new_message_notify;解不出就照旧走下面的 JSON/文本兜底。 + if frame_payload: response = Response_pb2.Response() - response.ParseFromString(frame.payload) + response.ParseFromString(frame_payload) body = response.body if body.HasField("new_message_notify"): notify = body.new_message_notify @@ -306,6 +348,11 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]: if isinstance(raw, str): payloads = [raw.encode("utf-8", errors="ignore")] + elif is_push_frame: + # 已确认是 frontier PushFrame:只解析它的内层负载。整帧字节里还有 + # seqId / traceid / payloadType 等元数据,拿去做纯文本兜底会把每条 + # 「连接建立」等控制帧误当成一条用户私信记录并触发一次自动回复。 + payloads = [frame_payload] if frame_payload else [] else: payloads = [raw] # 尝试 gzip 解压(frontier 常见) diff --git a/backend/rpa_engine/douyin_im/service.py b/backend/rpa_engine/douyin_im/service.py index 051024d..870b572 100644 --- a/backend/rpa_engine/douyin_im/service.py +++ b/backend/rpa_engine/douyin_im/service.py @@ -320,6 +320,7 @@ class DouyinImService: reply_cooldown_seconds: Optional[int] = None, cooldown_resolver: Optional[Callable[[], Awaitable[int]]] = None, refresh_credentials: Optional[Callable[[], Awaitable[bool]]] = None, + send_fallback: Optional[Callable[[str, str], Awaitable[tuple[bool, str]]]] = None, follow_tick: Optional[Callable[[], Awaitable[None]]] = None, on_session_invalid: Optional[Callable[[str], Awaitable[None]]] = None, on_ready: Optional[ReadyFn] = None, @@ -354,6 +355,11 @@ class DouyinImService: self._cooldown_resolver = cooldown_resolver # 由 worker 注入:触发后台重新采集 web_protect/keys(刷新 ts_sign),返回是否刷新成功 self.refresh_credentials = refresh_credentials + # 由 worker 注入的第二套发送方案:当 HTTP 签名发送被安全网关拒绝 + # (decision=KICK / 7911 / INVALID_REQUEST)时,用浏览器页面上下文 + # 重新发送(真实 JS 生成 a_bogus/bd-ticket-guard,可自愈被踢的会话)。 + # 签名: async (conversation_id, content) -> (ok, detail) + self.send_fallback = send_fallback self._running = False self._replied_keys: set[str] = set() self._logged_keys: set[str] = set() @@ -363,6 +369,9 @@ class DouyinImService: self._conv_previews: dict[str, str] = {} self._conv_names: dict[str, str] = {} # uid/conv_id -> nickname self._conv_meta: dict[str, dict] = {} # conversation_id -> meta + # 抖音判定会话列表请求本身不合法时置位:这轮托管不再重复轮询该接口, + # 实时长连接成为唯一接收通道(已在系统日志里说明)。 + self._conversation_list_unsupported = False self._ws_client: Optional[DouyinImWsClient] = None self.last_error: str = "" @@ -1037,6 +1046,11 @@ class DouyinImService: initial: bool = False, defer_handlers: bool = False, ) -> list[dict]: + if self._conversation_list_unsupported: + # 抖音已明确拒绝过这个请求本身;重复调用只会每轮浪费一次请求, + # 并把同一条错误反复写进日志。原因已在首次拒绝时记录。 + return [] + controller = get_traffic_controller() async with controller.background_slot( self.account_id, @@ -1053,6 +1067,14 @@ class DouyinImService: account_id=self.account_id, ) as http: conversations = await http.get_conversations(enrich_profiles=False) + if http.conversation_list_unsupported: + self._conversation_list_unsupported = True + logger.warning( + "Account %s disabled conversation reconciliation; " + "the realtime WebSocket is now the only receive path", + self.account_id, + ) + return [] # Capture the previous preview before _index_conversations overwrites # _conv_meta. A conversation-list preview is not inherently a new @@ -1445,6 +1467,40 @@ class DouyinImService: if refreshed: continue break + + # 第二套发送方案(浏览器页面内发送): + # HTTP 签名发送被安全网关拒绝(KICK/7911/INVALID_REQUEST)时,交给 worker + # 用浏览器页面上下文重发——由抖音页面自带的 security-sdk 在真实环境生成 + # a_bogus/bd-ticket-guard,绕开我们 Node execjs 的签名模拟,可自愈被踢会话。 + upper_err = (self.last_error or "").upper() + if self.send_fallback and ( + "DECISION=KICK" in upper_err + or "STATUS_CODE=7911" in upper_err + or "INVALID_REQUEST" in upper_err + ): + try: + fb_ok, fb_detail = await self.send_fallback(conversation_id, content) + except Exception as exc: + logger.warning(f"send_fallback raised for {conversation_id}: {exc}") + fb_ok, fb_detail = False, f"浏览器兜底发送异常:{exc}" + if fb_ok: + self._session_invalid_strikes = 0 + self._session_invalid_fired = False # 兜底成功说明登录仍有效,撤销自动下线 + system_logger.record( + "浏览器兜底发送成功", + detail=f"会话 {conversation_id}:{fb_detail}", + level="success", + category="send", + account_id=self.account_id, + ) + return True, None + system_logger.record( + "浏览器兜底发送失败", + detail=f"会话 {conversation_id}:{fb_detail}", + level="error", + category="send", + account_id=self.account_id, + ) await self._note_session_invalid(self.last_error) return False, None @@ -1479,7 +1535,7 @@ class DouyinImService: system_logger.record( "IM 登录失效,自动下线", detail=f"{reason}({failure_detail})。" - "请停止托管后用浏览器模式重新登录并打开私信页,再重新启动托管。", + "系统正在自动重登录,请留意账号卡片上的登录二维码并扫码。", level="error", category="auth", account_id=self.account_id, @@ -1495,17 +1551,18 @@ class DouyinImService: """手动发送私信""" from .conv_util import normalize_conversation_id from .auth import DouyinAuth + from .dy_util import DEFAULT_USER_AGENT auth = DouyinAuth() auth.perepare_auth( self.session.cookie_header(), self.session.web_protect_str, self.session.keys_str, + user_agent=self.session.user_agent or DEFAULT_USER_AGENT, ) - if getattr(self.session, "uid_verified", False) and self.session.my_uid: - my_uid = self.session.my_uid - else: - my_uid = await asyncio.to_thread(lambda: auth.get_uid()) or self.session.my_uid + my_uid = self.session.my_uid + if not my_uid: + my_uid = await asyncio.to_thread(lambda: auth.get_uid()) or 0 if my_uid: conversation_id = normalize_conversation_id(conversation_id, my_uid) diff --git a/backend/rpa_engine/douyin_im/session.py b/backend/rpa_engine/douyin_im/session.py index 5614fee..b7f4cd9 100644 --- a/backend/rpa_engine/douyin_im/session.py +++ b/backend/rpa_engine/douyin_im/session.py @@ -14,9 +14,21 @@ def is_frontier_ws_url(url: str) -> bool: 真实抓包里 host 可能是 frontier-im.douyin.com,也可能是 frontierNN-normal.zijieapi.com 这类内部别名,二者都要认。 """ - if not url or "token=" not in url: + if not url: return False - return "frontier-im.douyin.com" in url or ("frontier" in url and "zijieapi.com" in url) + parsed = urlparse(url) + host = (parsed.hostname or "").lower() + query = parse_qs(parsed.query) + fpid = (query.get("fpid") or [""])[0] + legacy = host == "frontier-im.douyin.com" and "token" in query and fpid == "9" + browser_frontier = ( + "frontier" in host + and host.endswith("zijieapi.com") + and "access_key" in query + and "device_id" in query + and fpid == "9" + ) + return legacy or browser_frontier @dataclass @@ -34,7 +46,8 @@ class DouyinImSession: keys_str: str = "" web_protect_str: str = "" my_uid: int = 0 - # my_uid 是否已用 query/user 接口核验过(采集端推断的 my_uid 可能取错 tea_cache id) + # my_uid 是否已由账号资料 UID 等可靠来源核验。query/user 的 user_uid + # 不是所有账号的 IM UID,不能据此覆盖已采集/已同步的 my_uid。 uid_verified: bool = False conv_meta: dict = field(default_factory=dict) # 方案 A:直接复用浏览器抓到的真实 frontier 连接凭证(绕开我们自己推导 token/access_key 不准的问题) @@ -96,6 +109,14 @@ class DouyinImSession: my_uid = _as_uid(extra.get("my_uid")) or _as_uid(data.get("my_uid")) user_agent = str(extra.get("user_agent") or data.get("user_agent") or "").strip() + # 先整段扫描 localStorage,收集字段(避免遍历顺序导致取值不确定)。 + # 关键背景:新版抖音 web 端 __tea_cache_tokens_6383 的 user_unique_id 实际存的是 + # web_id(如 7678646545793812008),并非账号 UID;而 web_runtime_security_uid + # 才是账号真实 UID(如 2609567359568155)。混合登录态下若把 tea 的 user_unique_id + # 当 my_uid,会导致 device_id != my_uid,IM 发送被安全网关 KICK。 + ls_sec_uid = "" # web_runtime_security_uid(最可靠的账号 UID 来源) + ls_web_id = "" # 第一个 tea 条目的 web_id/user_unique_id + ls_tea_pairs = [] # [(user_unique_id, web_id), ...] 按出现顺序 if not device_id or not web_id or not keys_str or not web_protect_str or not my_uid: for origin in data.get("origins", []): for entry in origin.get("localStorage", []): @@ -107,27 +128,42 @@ class DouyinImSession: keys_str = value if name == "security-sdk/s_sdk_sign_data_key/web_protect" and not web_protect_str: web_protect_str = value - if "tea_cache_tokens" in name and not web_id: + if "tea_cache_tokens" in name: try: parsed = json.loads(value) - web_id = str( - parsed.get("web_id") - or parsed.get("user_unique_id") - or "" - ) - except Exception: - pass - if name == "web_runtime_security_uid" and not device_id: - if str(value or "").isdigit(): - device_id = value - if "tea_cache_tokens" in name and not my_uid: - try: - parsed = json.loads(value) - uid = parsed.get("user_unique_id") - if uid and str(uid).isdigit(): - my_uid = int(uid) + if isinstance(parsed, dict): + wid = str(parsed.get("web_id") or "") + uid = str(parsed.get("user_unique_id") or "") + if not ls_web_id: + ls_web_id = wid or uid + ls_tea_pairs.append((uid, wid)) except Exception: pass + if name == "web_runtime_security_uid": + v = str(value or "") + if v.isdigit() and not ls_sec_uid: + ls_sec_uid = v + + # web_id:extra 显式值 > localStorage tea + if not web_id: + web_id = ls_web_id + + # my_uid 优先级:extra/顶层 > web_runtime_security_uid(真实账号 UID)> + # tea 的 user_unique_id(仅当与自身 web_id 不同才可信,避免误取 web_id) + if not my_uid and ls_sec_uid: + my_uid = int(ls_sec_uid) + if not my_uid: + for uid, wid in ls_tea_pairs: + if uid.isdigit() and not (wid and uid == wid): + my_uid = int(uid) + break + + # device_id 优先级:extra/cookies > web_runtime_security_uid(与账号 UID 绑定) + if not device_id: + if ls_sec_uid: + device_id = ls_sec_uid + elif my_uid: + device_id = str(my_uid) if not my_uid: for item in data.get("cookies", []): @@ -144,6 +180,14 @@ class DouyinImSession: elif not device_id and web_id: device_id = web_id + # 最终一致性收敛:protobuf/frontier 的 device_id 优先取 session.device_id + # (resolve_proto_device_id),若凭证里残留旧设备号(如 www 域 + # web_runtime_security_uid),发送时 device_id != my_uid 会被安全网关 + # 判为设备指纹异常 -> decision=KICK。my_uid 此时已是权威账号 UID, + # 不一致时以 my_uid 收敛 device_id。 + if my_uid and device_id and str(device_id) != str(my_uid): + device_id = str(my_uid) + ws_urls = list(extra.get("ws_urls") or []) # 方案 A:凭证采集工具可携带浏览器抓到的真实 frontier 连接(含 token/sdk_cert/ts_sign)。 @@ -193,6 +237,7 @@ class DouyinImSession: "keys_str": self.keys_str, "web_protect_str": self.web_protect_str, "my_uid": self.my_uid, + "uid_verified": self.uid_verified, "conv_meta": self.conv_meta, "sdk_cert": self.sdk_cert, "frontier_ts_sign": self.frontier_ts_sign, @@ -212,6 +257,7 @@ class DouyinImSession: keys_str=str(data.get("keys_str") or ""), web_protect_str=str(data.get("web_protect_str") or ""), my_uid=int(data.get("my_uid") or 0), + uid_verified=bool(data.get("uid_verified", False)), conv_meta=dict(data.get("conv_meta") or {}), sdk_cert=str(data.get("sdk_cert") or ""), frontier_ts_sign=str(data.get("frontier_ts_sign") or ""), diff --git a/backend/rpa_engine/douyin_im/ws_client.py b/backend/rpa_engine/douyin_im/ws_client.py index 7a71e6a..fca9711 100644 --- a/backend/rpa_engine/douyin_im/ws_client.py +++ b/backend/rpa_engine/douyin_im/ws_client.py @@ -1,4 +1,5 @@ import asyncio +import gzip import logging import os import weakref @@ -14,6 +15,31 @@ logger = logging.getLogger("douyin_im.ws") MessageHandler = Callable[[dict], Awaitable[None]] + +def _safe_frame_metadata(payload: bytes) -> str: + """Return non-content protobuf metadata for early connection diagnostics.""" + try: + from .static import Live_pb2, Response_pb2 + + frame = Live_pb2.PushFrame() + frame.ParseFromString(payload) + body = bytes(frame.payload) + if str(frame.payloadEncoding or "").lower() == "gzip": + body = gzip.decompress(body) + response = Response_pb2.Response() + response.ParseFromString(body) + fields = [field.name for field, _ in response.body.ListFields()] + message = str(response.message or response.error_desc or "")[:80] + return ( + f"service={frame.service} method={frame.method} " + f"encoding={frame.payloadEncoding or 'none'} " + f"type={frame.payloadType or 'none'} payload_bytes={len(body)} " + f"cmd={response.cmd} body={','.join(fields) or 'none'} " + f"status={message or 'ok'}" + ) + except Exception as exc: + return f"metadata_unavailable={type(exc).__name__}" + # Both stages are finite. The transport queue gives the receive coroutine a # small amount of breathing room, while the application queue decouples Pong / # frame reads from potentially slow database and reply work. Once both fill, @@ -127,6 +153,8 @@ class DouyinImWsClient: self._last_connection_lifetime = 0.0 self._message_queue: Optional[asyncio.Queue[dict]] = None self._dispatcher_task: Optional[asyncio.Task] = None + self._received_frame_count = 0 + self._heartbeat_ack_logged = False async def start(self): if self._task and not self._task.done(): @@ -331,6 +359,16 @@ class DouyinImWsClient: headers.append(("Cookie", cookie)) return headers + @staticmethod + def _uses_browser_frontier(url: str) -> bool: + return "zijieapi.com" in url and "access_key=" in url + + async def _run_browser_heartbeat(self, websocket) -> None: + """Mirror Frontier's browser SDK application-level ``hi`` heartbeat.""" + while self._running: + await websocket.send("hi") + await asyncio.sleep(30) + async def _run_connection(self, url: str) -> None: """Open one connection and dispatch messages sequentially. @@ -343,6 +381,8 @@ class DouyinImWsClient: loop = asyncio.get_running_loop() connected_at: float | None = None connection: Optional[WebSocketClientProtocol] = None + heartbeat_task: Optional[asyncio.Task] = None + browser_frontier = self._uses_browser_frontier(url) source_ip = str(getattr(self.session, "egress_source_ip", "") or "").strip() connect_kwargs = {"local_addr": (source_ip, 0)} if source_ip else {} try: @@ -354,7 +394,9 @@ class DouyinImWsClient: user_agent_header=self.session.user_agent, compression="deflate", open_timeout=10, - ping_interval=20, + # The current Douyin browser Frontier SDK uses a text ``hi`` + # heartbeat instead of RFC WebSocket ping frames. + ping_interval=None if browser_frontier else 20, # A handler may legitimately wait up to SQLite's 30s busy # timeout. Leave enough headroom for queued work so a healthy # socket isn't mistaken for a dead peer during that stall. @@ -371,7 +413,10 @@ class DouyinImWsClient: self._connection = websocket connected_at = loop.time() self.connected = True - logger.info("IM WebSocket connected") + logger.info( + "IM WebSocket connected: subprotocol=%s", + getattr(websocket, "subprotocol", None) or "none", + ) self._record_connection_system_event( "connected", "实时接收通道已连接", @@ -379,10 +424,23 @@ class DouyinImWsClient: level="success", ) - async for raw in websocket: - if not self._running: - break - await self._dispatch(raw) + if browser_frontier: + heartbeat_task = asyncio.create_task( + self._run_browser_heartbeat(websocket), + name=f"im-ws-heartbeat-{self.account_id or 'na'}", + ) + try: + async for raw in websocket: + if not self._running: + break + await self._dispatch(raw) + finally: + if heartbeat_task and not heartbeat_task.done(): + heartbeat_task.cancel() + try: + await heartbeat_task + except asyncio.CancelledError: + pass finally: if connected_at is not None: self._last_connection_lifetime = max(0.0, loop.time() - connected_at) @@ -408,6 +466,11 @@ class DouyinImWsClient: ) async def _dispatch(self, raw): + if raw == "hi": + if not self._heartbeat_ack_logged: + logger.info("IM WebSocket application heartbeat acknowledged") + self._heartbeat_ack_logged = True + return self._ensure_dispatcher() queue = self._message_queue if queue is None: @@ -417,6 +480,17 @@ class DouyinImWsClient: else: payload = raw items = parse_ws_payload(payload) + self._received_frame_count += 1 + if self._received_frame_count <= 3: + metadata = _safe_frame_metadata(payload) if not items else "parsed-message" + logger.info( + "IM WebSocket frame received: seq=%d kind=%s bytes=%d parsed=%d %s", + self._received_frame_count, + "text" if isinstance(raw, str) else "binary", + len(payload), + len(items), + metadata, + ) for item in items: if not self._running: return diff --git a/backend/rpa_engine/playwright_worker.py b/backend/rpa_engine/playwright_worker.py index d52db98..ae6b387 100644 --- a/backend/rpa_engine/playwright_worker.py +++ b/backend/rpa_engine/playwright_worker.py @@ -4,8 +4,10 @@ import asyncio import base64 import logging import time +import io from datetime import datetime -from typing import Optional +from typing import Awaitable, Callable, Optional +from PIL import Image from sqlalchemy import select, update from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession @@ -39,6 +41,20 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(me logger = logging.getLogger("rpa_engine") +async def _start_playwright_for_browser(headless: Optional[bool] = None): + """Start the driver only after DISPLAY exists. + + Playwright's Node driver inherits the environment at ``start()`` time and + later launches Chromium itself. Starting Xvfb after the driver therefore + leaves headed Chromium without DISPLAY on Linux even though Python can see + it. + """ + if headless is None: + headless = resolve_headless(default=False) + await ensure_browser_display(headless) + return await async_playwright().start(), headless + + async def _launch_chromium(pw, args: list[str], headless: Optional[bool] = None): """统一的 Chromium 启动入口:自动处理无头/有头、虚拟显示与住宅代理。 @@ -76,10 +92,14 @@ class DouyinWorker: login_mode: str = "auto", *, credential_prevalidated: bool = False, + relogin_hook: Optional[Callable[[int], Awaitable[None]]] = None, ): self.account_id = account_id self.login_mode = login_mode # auto | im_direct | browser self.credential_prevalidated = bool(credential_prevalidated) + # 登录态失效(KICK/INVALID_REQUEST/用户未登录)时通知上层自动重登录; + # 由 WorkerManager 注入,worker 自身不感知 manager,避免循环依赖。 + self.relogin_hook: Optional[Callable[[int], Awaitable[None]]] = relogin_hook self.browser = None self.context = None self.page = None @@ -118,7 +138,11 @@ class DouyinWorker: self._last_refresh_ts = 0.0 self._refresh_cooldown = 90.0 self._user_agent: str = "" + # 账号表显式配置的伪装 UA(区别于 resolve_user_agent 的默认值): + # 为空表示用户没配置,应保留凭证采集时写入的真实浏览器 UA。 + self._raw_user_agent: str = "" self._sec_user_id_missing_fired = False + self._douyin_logged_out_reported = False # Lightweight follow-welcome configuration. Disabled accounts refresh # infrequently, so 500 idle workers do not query Account + sec_user_id # every minute merely to discover that the feature is still off. @@ -128,6 +152,11 @@ class DouyinWorker: self._follow_welcome_enabled = False self._follow_welcome_content = "" self._follow_welcome_sec_user_id = "" + # 登录态保活:定时用已保存登录态访问抖音首页,触发 passport 滑动续期, + # 把「30 天必失效」的 sessionid 变成「持续活跃基本不失效」。 + self._keepalive_task: asyncio.Task | None = None + self._keepalive_lock = asyncio.Lock() + self._keepalive_last_result: str = "" async def _load_user_agent(self) -> str: """读取账号配置的伪装设备头,用于浏览器与 IM 全链路一致。""" @@ -143,6 +172,30 @@ class DouyinWorker: await db.close() return self._user_agent + async def _load_raw_user_agent(self) -> str: + """读取账号表显式配置的伪装 UA;未配置返回空串。 + + 与 _load_user_agent 的区别:后者在账号未配置时回退到默认 Chrome/120, + 而凭证采集工具/浏览器模式登录会把“真实采集浏览器”的 UA 写入 + storage_state(im_session_data 的 user_agent)。若用默认 UA 去签名一套 + 真实浏览器(如 Chrome/148)采集的凭证,抖音安全网关会返回 7911/KICK。 + """ + if self._raw_user_agent: + return self._raw_user_agent + db = await self.get_db() + raw = "" + try: + result = await db.execute( + select(Account.user_agent).where(Account.id == self.account_id) + ) + raw = str(result.scalar_one_or_none() or "").strip() + finally: + await db.close() + self._raw_user_agent = raw + if raw: + self._user_agent = raw + return raw + def _browser_context_options(self, storage_state: dict | None = None) -> dict: opts = { "user_agent": self._user_agent or resolve_user_agent(None), @@ -340,6 +393,10 @@ class DouyinWorker: sec_user_id = str(detail.get("sec_user_id") or "").strip() if not sec_user_id: + if detail.get("logged_out"): + # 抖音已判定登录失效:托管继续跑也收不到、发不出任何私信, + # 必须显式告警,不能只当成一次「资料接口抖动」。 + await self._report_douyin_logged_out(str(detail.get("message") or "")) if detail.get("sec_user_id_status") == "unknown": raise RuntimeError( detail.get("message") @@ -447,6 +504,27 @@ class DouyinWorker: finally: await db.close() + async def _report_douyin_logged_out(self, message: str) -> None: + """抖音判定登录失效时告警一次(每轮托管只报一次)。""" + if self._douyin_logged_out_reported: + return + self._douyin_logged_out_reported = True + reason = message or ( + "抖音返回「用户未登录」:Cookie 仍在但服务端登录态已失效," + "请停止托管后重新扫码登录该账号。" + ) + logger.warning("Account %s is logged out on Douyin: %s", self.account_id, reason) + system_logger.record( + "抖音登录态已失效,需重新扫码登录", + detail=( + f"{reason} 当前托管既收不到新私信,也无法发送自动回复;" + "账号卡片上的「Cookie 有效」只表示本地还存着 sessionid。" + ), + level="error", + category="auth", + account_id=self.account_id, + ) + async def _stop_for_missing_sec_user_id(self, stage: str) -> None: """Stop hosting once when the account identity lacks sec_user_id.""" if self._sec_user_id_missing_fired: @@ -491,6 +569,33 @@ class DouyinWorker: await self._stop_for_missing_sec_user_id(stage) return "" + async def _best_effort_sec_user_id( + self, + *, + refresh_if_missing: bool = False, + refresh_if_stale: bool = False, + force_refresh: bool = False, + ) -> str: + """Resolve sec_user_id without making optional profile data block IM.""" + try: + sec_user_id = str(await self._load_sec_user_id() or "").strip() + should_refresh = force_refresh or ( + not sec_user_id and refresh_if_missing + ) + if not should_refresh and refresh_if_stale: + should_refresh = await self._sec_user_id_is_stale() + if should_refresh: + sec_user_id = str(await self._refresh_sec_user_id() or "").strip() + return sec_user_id + except Exception as exc: + logger.warning( + "Account %s could not refresh optional sec_user_id; " + "IM hosting will continue: %s", + self.account_id, + exc, + ) + return "" + def _is_browser_alive(self) -> bool: return bool( self.page @@ -607,20 +712,111 @@ class DouyinWorker: ) -> DouyinImSession: db = await self.get_db() saved_im = None + account_uid = "" # accounts.douyin_uid:拿 cookie 从抖音拉取的权威账号 UID + profile_uid = "" + profile_updated_at = None + cookie_updated_at = None try: result = await db.execute( - select(Account.im_session_data).where(Account.id == self.account_id) + select( + Account.im_session_data, + Account.cookie_updated_at, + Account.douyin_uid, + AccountProfileDetail.uid, + AccountProfileDetail.updated_at.label("profile_updated_at"), + ) + .outerjoin( + AccountProfileDetail, + AccountProfileDetail.account_id == Account.id, + ) + .where(Account.id == self.account_id) ) - saved_im = result.scalar_one_or_none() + row = result.one_or_none() + if row: + saved_im = row.im_session_data + account_uid = str(getattr(row, "douyin_uid", None) or "").strip() + profile_uid = str(getattr(row, "uid", None) or "").strip() + profile_updated_at = getattr(row, "profile_updated_at", None) + cookie_updated_at = getattr(row, "cookie_updated_at", None) finally: await db.close() session = build_im_session_from_storage(storage or {}, saved_im) - ua = await self._load_user_agent() - session.user_agent = ua + # 权威 UID 覆盖:accounts.douyin_uid 是拿当前 cookie 从抖音接口拉取后写入的 + # 账号标识,最可靠,无条件覆盖。account_profile_details.uid 有串号风险 + # (如账号 9 的 profile 里存了别的账号的 UID),仅在资料不早于 cookie 更新 + # 时才可信(保留时间戳保护)。 + # 之前对 douyin_uid 也套时间戳条件:混合登录态下 tea 解析出的 my_uid 可能 + # 是 web_id(device_id != my_uid -> KICK 循环),而资料同步往往滞后于 + # cookie 落库,时间戳条件会让错误 UID 一直带病运行。 + verified_uid = account_uid if account_uid.isdigit() else "" + if not verified_uid and profile_uid.isdigit(): + profile_fresh = ( + cookie_updated_at is None + or ( + profile_updated_at is not None + and profile_updated_at >= cookie_updated_at + ) + ) + if profile_fresh: + verified_uid = profile_uid + if verified_uid: + verified_uid = int(verified_uid) + old_uid = int(session.my_uid or 0) + if old_uid and old_uid != verified_uid: + logger.warning( + "Account %s replaced collected IM uid %s with current " + "uid %s (account.douyin_uid=%s profile.uid=%s " + "profile_updated_at=%s cookie_updated_at=%s)", + self.account_id, + old_uid, + verified_uid, + account_uid, + profile_uid, + profile_updated_at, + cookie_updated_at, + ) + session.my_uid = verified_uid + # device_id 必须与 my_uid 指向同一账号:protobuf/frontier 的 device_id + # 优先取 session.device_id(见 resolve_proto_device_id),若凭证里残留 + # 旧设备号(如 www 域 web_runtime_security_uid),发送时 device_id != + # my_uid 会被安全网关判为设备指纹异常 -> decision=KICK。 + if str(session.device_id or "") != str(verified_uid): + if session.device_id: + logger.info( + "Account %s synced device_id %s -> %s to match verified uid", + self.account_id, + session.device_id, + verified_uid, + ) + session.device_id = str(verified_uid) + # Keep the browser runtime device_id for frontier. The protobuf + # sender uses the verified IM UID separately in DouyinAuth. + session.uid_verified = True + # UA 全链路一致原则:a_bogus 签名、IM 请求头、Protobuf body 必须与 + # 凭证采集环境(storage_state.user_agent)使用同一 UA,否则安全网关 + # 判定设备指纹不一致 -> 7911 / decision=KICK。 + # 账号表显式配置的 UA(浏览器登录上下文用它)优先;未配置时保留 + # storage_state/im_session_data 里采集写入的真实浏览器 UA, + # 绝不用默认 Chrome/120 去签名一套 Chrome/148 环境采集的凭证。 + raw_ua = await self._load_raw_user_agent() + if raw_ua: + session.user_agent = resolve_user_agent(raw_ua) + else: + logger.info( + "Account %s: 未显式配置 UA,保留采集 UA=%s", + self.account_id, + session.user_agent[:60] + "…" if len(session.user_agent or "") > 60 else session.user_agent, + ) if extra: - if extra.get("ws_urls") and not session.ws_urls: - session.ws_urls = list(dict.fromkeys(extra.get("ws_urls") or [])) + # 浏览器本次真实建连的 frontier 地址必须排在缓存地址之前。 + # 之前写成 "and not session.ws_urls":DB 里那条我们自己拼出来的 + # frontier-im 地址永远非空,于是每次重新登录抓到的真实地址都被丢弃, + # 长连接一直用推导出的 token/access_key,收不到抖音下发的私信。 + if extra.get("ws_urls"): + session.ws_urls = list( + dict.fromkeys(list(extra["ws_urls"]) + list(session.ws_urls)) + ) # 这些是浏览器实时 localStorage 读取到的“最新”签名凭证(含时效性的 ts_sign), # 必须覆盖来自 DB 的旧值,否则重新登录也刷新不了凭证,导致一直 7911。 if extra.get("keys_str"): @@ -716,12 +912,16 @@ class DouyinWorker: ) return False, reason - if not await self._require_sec_user_id( - "启动托管时", + sec_user_id = await self._best_effort_sec_user_id( refresh_if_missing=True, refresh_if_stale=True, - ): - return False, "缺少 sec_user_id,托管已自动退出" + ) + if not sec_user_id: + logger.warning( + "Account %s has no verified sec_user_id; continuing IM hosting " + "with follow-welcome polling temporarily unavailable", + self.account_id, + ) logger.info( f"IM session validated for account {self.account_id} " @@ -970,13 +1170,18 @@ class DouyinWorker: # 实时解析冷却时间(账号专属优先,否则全局),改设置无需重启托管 cooldown_resolver=self.resolve_cooldown_seconds, # 不在发送链路上自动开浏览器刷新:实测重载页面并不会重生 web_protect, - # 反而每次失败阻塞 ~22s(“反应特别慢”),且无法解决 7911 风控。 + # 反而每次失败阻塞 ~22s("反应特别慢"),且无法解决 7911 风控。 refresh_credentials=None, + # 第二套发送方案:HTTP 签名发送被 KICK/7911/INVALID_REQUEST 拒绝时, + # 用浏览器页面上下文重发(真实 JS 签名,可自愈被踢会话)。 + send_fallback=self.send_im_via_browser_page, ) self._im_service = im_service from rpa_engine.douyin_im import hosted_registry if session.my_uid: hosted_registry.register(session.my_uid) + # 托管运行期间定期活跃抖音首页,给 passport 登录态滑动续期 + await self._start_keepalive() try: await im_service.run() finally: @@ -985,6 +1190,497 @@ class DouyinWorker: if self._im_service is im_service: await im_service.stop() self._im_service = None + await self._stop_keepalive() + + # ---------- 登录态保活(keepalive) ---------- + # 抖音 web 登录态(sessionid/passport)有有效期且无 refresh token 可自动换新, + # 但服务端对「持续活跃」的账号做滑动续期。IM 通道(imapi + frontier WS)的活跃 + # 并不刷新 passport 登录态,所以托管期间需要定期用已保存登录态打开一次抖音首页, + # 让页面自带 JS 触发 passport 活跃请求,把登录态从「30 天必失效」延长为 + # 「持续活跃基本不失效」。行为等同真人打开网页,风险低。 + + def _keepalive_interval(self) -> float: + try: + return max(300.0, float(os.getenv("KEFU_KEEPALIVE_INTERVAL", "21600"))) + except (TypeError, ValueError): + return 21600.0 + + def _keepalive_disabled(self) -> bool: + return os.getenv("KEFU_KEEPALIVE_DISABLED", "").strip().lower() in ( + "1", "true", "yes", + ) + + @staticmethod + def _cookie_expires_map(cookies: list) -> dict: + """提取 passport 关键 cookie 的过期时间(epoch 秒),用于观测是否滑动续期。""" + names = ( + "sid_guard", "sessionid", "sessionid_ss", + "sid_tt", "sid_tt_ss", "uid_tt", "uid_tt_ss", + ) + out: dict = {} + for c in cookies or []: + name = (c.get("name") or "").lower() + if name in names and c.get("value"): + try: + exp = int(float(c.get("expires") or 0)) + except (TypeError, ValueError): + exp = 0 + out[name] = exp if exp > 0 else 0 + return out + + @staticmethod + def _fmt_expires_map(m: dict) -> str: + from datetime import datetime as _dt + + parts = [] + for name, exp in sorted(m.items()): + if exp: + parts.append( + f"{name}={_dt.fromtimestamp(exp).strftime('%m-%d %H:%M')}" + ) + else: + parts.append(f"{name}=session") + return ", ".join(parts) if parts else "(none)" + + async def _start_keepalive(self) -> None: + if self._keepalive_task and not self._keepalive_task.done(): + return + if self._keepalive_disabled(): + logger.info( + f"Account {self.account_id}: keepalive disabled by KEFU_KEEPALIVE_DISABLED" + ) + return + self._keepalive_task = asyncio.create_task( + self._keepalive_loop(), + name=f"douyin-keepalive-{self.account_id}", + ) + + async def _stop_keepalive(self) -> None: + task = self._keepalive_task + self._keepalive_task = None + if task and task is not asyncio.current_task() and not task.done(): + task.cancel() + try: + await task + except (asyncio.CancelledError, Exception): + pass + + async def _keepalive_loop(self) -> None: + """周期保活:让服务端认为账号持续活跃,滑动续期 passport 登录态。""" + interval = self._keepalive_interval() + logger.info( + f"Account {self.account_id}: keepalive loop started " + f"(every {interval / 3600:.1f}h, timeout-based, low risk)" + ) + while not self.stopping and self.is_running: + await asyncio.sleep(interval) + if self.stopping or not self.is_running: + break + try: + ok, detail = await self._keepalive_touch() + self._keepalive_last_result = detail + if ok: + logger.info(f"Account {self.account_id}: keepalive ok - {detail}") + else: + # 保活发现登录态失效:IM 通道很快也会报错并触发 + # on_im_session_invalid → relogin_hook 自动重登录,这里不重复处理。 + logger.warning( + f"Account {self.account_id}: keepalive failed - {detail}" + ) + except asyncio.CancelledError: + raise + except Exception as exc: + logger.warning( + f"Account {self.account_id}: keepalive exception: {exc}" + ) + logger.info(f"Account {self.account_id}: keepalive loop stopped") + + async def _keepalive_touch(self) -> tuple[bool, str]: + """打开抖音首页触发 passport 活跃续期,并重新持久化 cookie。 + + 在全局 browser_slot 内执行,与扫码登录/凭证刷新等浏览器操作互斥, + 保证同一时刻只有一个有头浏览器实例。 + """ + if self._keepalive_lock.locked(): + return False, "上一次保活仍在进行" + async with self._keepalive_lock: + storage_state = await self._load_storage_state() + if not storage_state: + return False, "未找到已保存的登录态" + before_exp = self._cookie_expires_map(storage_state.get("cookies") or []) + controller = get_traffic_controller() + async with controller.browser_slot(self.account_id, "keepalive"): + pw = None + browser = None + context = None + page = None + saved_browser_refs = ( + self.playwright, + self.browser, + self.context, + self.page, + ) + try: + pw, browser_headless = await _start_playwright_for_browser() + import sys + + args = [ + "--disable-blink-features=AutomationControlled", + "--no-sandbox", + "--disable-setuid-sandbox", + ] + if sys.platform == "win32": + args.append("--start-minimized") + browser = await _launch_chromium( + pw, args, headless=browser_headless + ) + ua = self._user_agent or resolve_user_agent(None) + context = await browser.new_context( + storage_state=storage_state, + user_agent=ua, + viewport={"width": 1280, "height": 800}, + locale="zh-CN", + ) + await context.add_init_script( + "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})" + ) + page = await context.new_page() + # 临时挂到 self,复用 _has_visible_login_prompt / _persist_cookies; + # browser_slot 全局串行保证不会与登录流程并发争抢这些字段。 + ( + self.playwright, + self.browser, + self.context, + self.page, + ) = (pw, browser, context, page) + # 默认访问私信页(更贴近真实活跃,触发 IM 域请求);可用 + # KEFU_KEEPALIVE_URL 覆盖,/im 异常时回退首页。 + import random as _random + + target_url = os.getenv( + "KEFU_KEEPALIVE_URL", "https://www.douyin.com/im" + ) + try: + await page.goto( + target_url, + wait_until="domcontentloaded", + timeout=30000, + ) + except Exception: + await page.goto( + "https://www.douyin.com/", + wait_until="domcontentloaded", + timeout=30000, + ) + # 随机停留 + 轻微滚动,避免固定机械节奏 + await asyncio.sleep(_random.uniform(4, 8)) + try: + await page.mouse.wheel(0, 600) + await asyncio.sleep(_random.uniform(0.5, 1.5)) + except Exception: + pass + if await self._has_visible_login_prompt(): + return False, ( + "页面显示未登录(服务端登录态已失效,将触发自动重登录)" + ) + # 观测 passport cookie 是否发生滑动续期(expires 变大) + after_exp = self._cookie_expires_map(await self.context.cookies()) + if before_exp: + renewed = [ + k for k in before_exp + if before_exp.get(k) and after_exp.get(k) + and after_exp[k] > before_exp[k] + ] + logger.info( + f"Account {self.account_id}: keepalive passport expires " + f"before[{self._fmt_expires_map(before_exp)}] " + f"after[{self._fmt_expires_map(after_exp)}] " + f"renewed={','.join(renewed) or 'none'}" + ) + # 活跃访问后 cookie(msToken 等)可能更新,重新落库 + try: + await self._persist_cookies() + except Exception as exc: + logger.warning( + f"Account {self.account_id}: keepalive persist cookies " + f"failed: {exc}" + ) + return True, f"已访问 {target_url} 并刷新登录态" + except asyncio.CancelledError: + raise + except Exception as exc: + return False, f"保活访问失败:{exc}" + finally: + ( + self.playwright, + self.browser, + self.context, + self.page, + ) = saved_browser_refs + if page is not None: + try: + await page.close() + except Exception: + pass + if context is not None: + try: + await context.close() + except Exception: + pass + if browser is not None: + try: + await browser.close() + except Exception: + pass + if pw is not None: + try: + await pw.stop() + except Exception: + pass + + async def send_im_via_browser_page( + self, + conversation_id: str, + content: str, + ) -> tuple[bool, str]: + """第二套发送方案:浏览器页面上下文内重发私信。 + + HTTP 签名发送被抖音安全网关拒绝(decision=KICK / 7911 / INVALID_REQUEST) + 时的兜底:用已保存的登录态打开抖音页面,由页面自带 security-sdk 在真实 + 浏览器环境里生成 a_bogus / bd-ticket-guard 并完成发送——绕开 Node execjs + 的签名模拟;浏览器重新加载页面也会重建安全会话,可自愈被服务端踢掉的 + 登录态。仅文本/表情/卡片内容可用,图片需先走 HTTP 上传链路。 + + 返回 (是否成功, 详情)。失败不会抛异常,只记录日志。 + """ + from rpa_engine.douyin_im.auth import DouyinAuth + from rpa_engine.douyin_im.conv_util import normalize_conversation_id, resolve_peer_uid + from rpa_engine.douyin_im.pb_decode import analyze_send_response + from rpa_engine.douyin_im.proto_builder import ProtoBuilder + from rpa_engine.douyin_im.reply_payload import build_msg_payload, parse_reply_content + + timeout = float(os.getenv("KEFU_BROWSER_SEND_TIMEOUT", "45")) + + async def _attempt() -> tuple[bool, str]: + try: + session = await self._build_im_session() + except Exception as exc: + return False, f"无法构建 IM 会话:{exc}" + if not session.can_direct_im(): + return False, "Cookie 缺失,浏览器兜底无法发送" + + auth = DouyinAuth.from_im_session(session) + my_uid = int(session.my_uid or 0) + if not my_uid: + my_uid = int(auth.get_uid() or 0) + if not my_uid: + return False, "无法获取 my_uid" + + conv_id = normalize_conversation_id(conversation_id, my_uid) + peer_uid = resolve_peer_uid(conv_id, my_uid) + if not peer_uid: + return False, "无法从会话 ID 解析对方用户 ID" + + # 1) 解析新鲜会话票据(unsigned 接口,发送被踢后依然可用) + try: + async with DouyinImHttpClient(session, account_id=self.account_id) as http: + resolved_id, short_id, ticket = await http.resolve_conversation_meta( + auth, conv_id, my_uid, peer_uid + ) + except Exception as exc: + return False, f"解析会话票据失败:{exc}" + if not short_id or not ticket: + return False, "未拿到会话 ticket/short_id" + if resolved_id: + conv_id = resolved_id + + # 2) 构造与 HTTP 发送一致的 protobuf 报文 + reply_spec = parse_reply_content(content) + if reply_spec.get("type") == "image": + return False, "浏览器兜底暂不支持图片回复,请改用纯文字" + try: + request_proto = await asyncio.to_thread( + ProtoBuilder.build_send_message_request, + auth, + conv_id, + short_id, + ticket, + *build_msg_payload(reply_spec), + ) + except Exception as exc: + return False, f"构造发送报文失败:{exc}" + body_b64 = base64.b64encode(request_proto.SerializeToString()).decode("ascii") + + s_v_web_id = session.cookies.get("s_v_web_id", "") + ms_token = session.cookies.get("msToken", "") + params = { + "verifyFp": s_v_web_id, + "fp": s_v_web_id, + } + if ms_token: + params["msToken"] = ms_token + + # 3) 打开页面:让 security-sdk 加载并接管签名。与登录/凭证刷新一致, + # 用非 headless + 最小化(headless 易被抖音安全 SDK 判定而生成无效签名)。 + pw = None + browser = None + context = None + page = None + try: + pw, browser_headless = await _start_playwright_for_browser() + # 浏览器页面 UA 必须与会话发送 UA 完全一致(a_bogus 绑定 UA), + # 直接使用 session.user_agent——它已被 _build_im_session 修正为 + # 凭证采集环境的真实 UA,而不是账号表默认值。 + context_ua = session.user_agent or resolve_user_agent(None) + import sys + + token_args = [ + "--disable-blink-features=AutomationControlled", + "--no-sandbox", + "--disable-setuid-sandbox", + ] + if sys.platform == "win32": + token_args.append("--start-minimized") + browser = await _launch_chromium( + pw, + token_args, + headless=browser_headless, + ) + storage_state = await self._load_storage_state() + context = await browser.new_context( + storage_state=storage_state or {}, + user_agent=context_ua, + viewport={"width": 1280, "height": 800}, + locale="zh-CN", + ) + await context.add_init_script( + "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})" + ) + page = await context.new_page() + await page.goto( + "https://www.douyin.com/message", + wait_until="domcontentloaded", + timeout=30000, + ) + # 等待安全 SDK 初始化(与 _reharvest_security_tokens 相同的轮询节奏) + sdk_ready = False + for _ in range(15): + try: + sdk_ready = bool( + await page.evaluate( + 'Boolean(localStorage["security-sdk/s_sdk_crypt_sdk"])' + ) + ) + except Exception: + sdk_ready = False + if sdk_ready: + break + await asyncio.sleep(1) + if not sdk_ready: + return False, "页面未加载 security-sdk,无法进行真实签名发送" + await asyncio.sleep(1.5) + + # 4) 页面上下文内 fetch:SDK 注入 a_bogus/bd-ticket-guard,携带同域 Cookie + js_result = await page.evaluate( + """async (args) => { + const bin = atob(args.bodyB64); + const buf = new Uint8Array(bin.length); + for (let i = 0; i < bin.length; i++) buf[i] = bin.charCodeAt(i); + const qs = new URLSearchParams(args.params).toString(); + const url = args.url + (qs ? '?' + qs : ''); + const ctl = new AbortController(); + const timer = setTimeout(() => ctl.abort(), 15000); + try { + const r = await fetch(url, { + method: 'POST', + headers: { + 'Content-Type': 'application/x-protobuf', + 'Accept': 'application/x-protobuf', + 'Referer': 'https://www.douyin.com/', + 'Origin': 'https://www.douyin.com', + }, + body: buf, + credentials: 'include', + signal: ctl.signal, + }); + const ab = await r.arrayBuffer(); + const bytes = new Uint8Array(ab); + let b64 = ''; + const chunk = 0x8000; + for (let i = 0; i < bytes.length; i += chunk) { + b64 += String.fromCharCode.apply(null, bytes.subarray(i, i + chunk)); + } + return { http: r.status, bodyB64: btoa(b64) }; + } catch (e) { + return { error: String(e && e.message || e) }; + } finally { clearTimeout(timer); } + }""", + { + "url": "https://imapi.douyin.com/v1/message/send", + "params": params, + "bodyB64": body_b64, + }, + ) + if not isinstance(js_result, dict) or js_result.get("error"): + return False, f"页面内发送请求失败:{js_result}" + body_bytes = base64.b64decode(js_result.get("bodyB64") or "") + http_status = js_result.get("http") + if not body_bytes: + return False, f"页面内发送无响应体(http={http_status})" + + result = analyze_send_response(body_bytes) + if result.get("ok"): + self._im_conv_meta[conv_id] = { + "conversation_short_id": short_id, + "ticket": ticket, + } + return True, ( + f"页面内发送成功 server_message_id={result.get('server_message_id')} " + f"resp[{result.get('summary')}]" + ) + decision = str(result.get("decision") or "").strip().upper() + if decision: + return False, f"页面内发送仍被安全网关拒绝 decision={decision}" + sc = result.get("status_code") + if sc is not None and sc != 0: + return False, f"页面内发送被拒绝 status_code={sc} {result.get('summary') or ''}" + return False, f"页面内发送未确认投递(http={http_status} resp[{result.get('summary')}])" + finally: + if page is not None: + try: + await page.close() + except Exception: + pass + if context is not None: + try: + await context.close() + except Exception: + pass + if browser is not None: + try: + await browser.close() + except Exception: + pass + if pw is not None: + try: + await pw.stop() + except Exception: + pass + + try: + ok, detail = await asyncio.wait_for(_attempt(), timeout=timeout) + except asyncio.TimeoutError: + return False, f"浏览器兜底发送超时({int(timeout)}s)" + except Exception as exc: + return False, f"浏览器兜底发送异常:{exc}" + if not ok: + logger.warning( + "Account %s browser fallback send failed for %s: %s", + self.account_id, + conversation_id, + detail, + ) + return ok, detail async def refresh_im_credentials(self) -> bool: """后台用一次性 headless 浏览器重新采集最新 web_protect/keys(含新鲜 ts_sign), @@ -1068,7 +1764,7 @@ class DouyinWorker: web_protect_str = "" keys_str = "" try: - pw = await async_playwright().start() + pw, browser_headless = await _start_playwright_for_browser() # 与登录流程一致使用非 headless + 最小化:headless 易被抖音安全 SDK 判定, # 可能生成无效 ts_sign,反而刷新无效。 await self._load_user_agent() @@ -1080,7 +1776,11 @@ class DouyinWorker: ] if sys.platform == "win32": token_args.append("--start-minimized") - browser = await _launch_chromium(pw, token_args) + browser = await _launch_chromium( + pw, + token_args, + headless=browser_headless, + ) context = await browser.new_context( storage_state=storage_state, user_agent=self._user_agent, @@ -1290,7 +1990,13 @@ class DouyinWorker: return [] async def on_im_session_invalid(self, reason: str): - """IM 登录失效时自动下线:标记账号 offline 并停止托管循环。""" + """IM 登录失效时自动下线:标记账号 offline 并停止托管循环。 + + 若配置了 relogin_hook(WorkerManager 注入),同步通知上层自动重登录: + 上层用 browser 模式重新启动 worker——浏览器流程会探测页面登录态, + 未登录则自动弹二维码(前端账号卡片展示),扫码成功后自动采集凭证 + 并恢复托管。用户只需扫码,无需手动停止/启动。 + """ logger.warning(f"Account {self.account_id} IM 登录失效,自动下线:{reason}") self.stopping = True self.is_running = False @@ -1301,8 +2007,18 @@ class DouyinWorker: pass await self.update_account_status( "offline", - error_msg=f"IM 登录已失效({reason}),托管已自动下线,请重新登录后再启动托管", + error_msg=( + f"IM 登录已失效({reason}),正在自动重登录," + "请留意账号卡片上的登录二维码并扫码" + ), ) + if self.relogin_hook: + try: + await self.relogin_hook(self.account_id) + except Exception as exc: + logger.error( + f"Account {self.account_id}: relogin_hook failed: {exc}" + ) async def follow_welcome_tick(self): """检测新粉丝,给「已互相关注」且未发送过的新粉丝发送一次关注欢迎语。 @@ -1314,38 +2030,30 @@ class DouyinWorker: return from rpa_engine.douyin_im.follower_poll import fetch_recent_followers - # The direct-service startup normally preloads the lightweight cache. - # Keep a guard-first fallback for legacy/tests/partial initialization: - # identity safety must not depend on follow-welcome configuration. - guarded_sec_user_id = "" - if not self._follow_config_loaded: - guarded_sec_user_id = await self._require_sec_user_id("托管运行中") - if not guarded_sec_user_id: - return - try: enabled, content, sec_user_id = ( await self._refresh_follow_welcome_config() ) except Exception: - if not guarded_sec_user_id: - # Even when the optional config read fails, execute the - # hosting identity guard before surfacing the transient error. - await self._require_sec_user_id("托管运行中") raise - sec_user_id = str(sec_user_id or guarded_sec_user_id or "").strip() - # sec_user_id remains a hosting invariant. The lightweight cached - # refresh detects a later database removal without making every - # disabled account query the database once per minute. - if not sec_user_id: - sec_user_id = await self._require_sec_user_id("托管运行中") - if not sec_user_id: - return - self._follow_welcome_sec_user_id = sec_user_id if not enabled or not content: return + sec_user_id = str(sec_user_id or "").strip() + if not sec_user_id: + sec_user_id = await self._best_effort_sec_user_id( + refresh_if_missing=True, + ) + if not sec_user_id: + logger.warning( + "Account %s skipped follow-welcome polling because sec_user_id " + "is temporarily unavailable", + self.account_id, + ) + return + self._follow_welcome_sec_user_id = sec_user_id + # 1) 功能已启用时才读取已处理过的粉丝集合 db = await self.get_db() try: @@ -1480,6 +2188,7 @@ class DouyinWorker: self.stopping = True self.is_running = False self._mark_startup_failed("托管初始化已取消") + await self._stop_keepalive() if self._im_service: await self._im_service.stop() task = self._task @@ -1586,11 +2295,25 @@ class DouyinWorker: im_session, im_ok, im_reason = prepared # 资料接口不依赖浏览器,必须在释放全局 browser slot 后核验, # 避免资料接口波动长期占住其他账号的浏览器登录通道。 - if not await self._require_sec_user_id( - "浏览器登录后", - force_refresh=True, - ): - return + # sec_user_id 只服务于「关注欢迎语」轮询,收私信与自动回复都不需要它。 + # 这里过去用 _require_sec_user_id 直接退出托管,导致资料接口拿不到 + # sec_user_id 的账号浏览器登录后立刻下线、永远不会自动回复。 + if not await self._best_effort_sec_user_id(force_refresh=True): + logger.warning( + "Account %s has no verified sec_user_id after browser login; " + "continuing IM hosting with follow-welcome polling unavailable", + self.account_id, + ) + system_logger.record( + "未能核验 sec_user_id,关注欢迎语暂不可用", + detail=( + "私信接收与自动回复不依赖 sec_user_id,托管继续运行;" + "如需「关注后自动欢迎语」,请在账号管理中同步资料。" + ), + level="warning", + category="auth", + account_id=self.account_id, + ) if im_ok: await self._persist_im_session( @@ -1643,7 +2366,7 @@ class DouyinWorker: ) storage_state = None - self.playwright = await async_playwright().start() + self.playwright, browser_headless = await _start_playwright_for_browser() import sys args = [ @@ -1655,7 +2378,11 @@ class DouyinWorker: args.append("--start-minimized") try: - self.browser = await _launch_chromium(self.playwright, args) + self.browser = await _launch_chromium( + self.playwright, + args, + headless=browser_headless, + ) logger.info(f"Account {self.account_id}: opening browser for IM setup (minimized)") except RuntimeError: # 无虚拟显示等带操作指引的错误原样抛出,避免被通用提示覆盖 @@ -1745,13 +2472,17 @@ class DouyinWorker: async def _verify_login_state(self) -> bool: """判断账号是否已登录(不要求私信页已打开)""" - if await self.check_logged_in_by_cookie(): - return True + # A stale sessionid can survive a server-side KICK. Page-level login + # prompts are authoritative negative evidence and must win over the + # mere presence of that cookie, otherwise browser refresh skips QR + # login and gets stuck behind the message-center login dialog. + if await self._has_visible_login_prompt(): + return False if self._is_404_page(): return False if await self.check_homepage_login_status(): return True - return await self.check_login_status() + return await self.check_logged_in_by_cookie() async def _probe_existing_login(self) -> bool: """打开浏览器后从首页探测登录态""" @@ -1764,32 +2495,39 @@ class DouyinWorker: logger.warning(f"Homepage probe skipped: {e}") return False - async def check_homepage_login_status(self) -> bool: - """检查抖音首页是否处于已登录状态""" + async def _has_visible_login_prompt(self) -> bool: + """Return True when the current page visibly asks the user to log in.""" try: - # 抖音登录后右上角有头像;未登录则有醒目的「登录」按钮/登录弹窗 login_modal = await self.page.query_selector( "#login-pannel, [class*='login-guide'], [class*='login-mask'], [class*='account_login']" ) if login_modal and await login_modal.is_visible(): + return True + + return bool(await self.page.evaluate("""() => { + const labels = new Set(['登录', '登录/注册', '立即登录']); + return [...document.querySelectorAll('button, a, [role="button"], p')] + .some((el) => labels.has((el.innerText || '').trim()) && el.offsetParent); + }""")) + except Exception as e: + logger.debug(f"Visible login prompt check failed: {e}") + return False + + async def check_homepage_login_status(self) -> bool: + """检查抖音首页是否处于已登录状态""" + try: + # 抖音登录后右上角有头像;未登录则有醒目的「登录」按钮/登录弹窗 + if await self._has_visible_login_prompt(): return False avatar = await self.page.query_selector( - "[class*='avatar'] img, img[class*='avatar'], [class*='Avatar']" + "header [class*='avatar'] img, header img[class*='avatar'], " + "[data-e2e*='user-avatar'] img, [data-e2e='user-avatar']" ) if avatar and await avatar.is_visible(): return True # 抖音登录后导航栏会出现「私信」入口 - has_dm_entry = await self.page.evaluate("""() => { - const nodes = [...document.querySelectorAll('a, span, div, button')]; - return nodes.some((el) => { - const t = (el.innerText || '').trim(); - return (t === '私信' || t === '消息') && el.offsetParent; - }); - }""") - if has_dm_entry: - return True except Exception as e: logger.debug(f"Homepage login check failed: {e}") return False @@ -1937,6 +2675,7 @@ class DouyinWorker: # 抖音登录二维码相关选择器(class 名为 hash,尽量用通用属性匹配) _QR_SELECTORS = [ + # Douyin 新版登录弹窗常见结构 "[class*='qrcode'] img", "[class*='QrCode'] img", "[class*='qr-code'] img", @@ -1948,6 +2687,18 @@ class DouyinWorker: "canvas[class*='qrcode']", "[class*='scan'] img", "[class*='scan'] canvas", + # 登录弹窗内最可能的 img / canvas + "[class*='login-guide'] img", + "[class*='login-guide'] canvas", + "[class*='login-panel'] img", + "[class*='login-panel'] canvas", + "[class*='login_pannel'] img", + "[class*='login_pannel'] canvas", + "[class*='account_login'] img", + "[class*='account_login'] canvas", + "#login-pannel img", + "#login-pannel canvas", + "#login-pannel [class*='qrcode']", ] _QR_CONTAINER_SELECTORS = [ "[class*='qrcode-container']", @@ -1956,6 +2707,11 @@ class DouyinWorker: "[class*='QrCode']", "[class*='login-scan']", "[class*='scan-code']", + "[class*='login-guide']", + "[class*='login-panel']", + "[class*='login_pannel']", + "[class*='account_login']", + "#login-pannel", ] # 兜底:抓不到二维码元素时,截取整块登录面板 / 登录 iframe,用户仍可扫描其中的码 _LOGIN_PANEL_SELECTORS = [ @@ -1966,7 +2722,10 @@ class DouyinWorker: "iframe[src*='sso']", "[class*='login_panel']", "[class*='login-panel']", + "[class*='login_pannel']", "[class*='account_login']", + "[class*='login-guide']", + "[class*='login-mask']", ] def _all_frames(self) -> list: @@ -1976,6 +2735,18 @@ class DouyinWorker: except Exception: return [self.page] + @staticmethod + def _looks_like_qr_box(box: dict) -> bool: + """根据尺寸/长宽比判断一个元素是否像二维码区域。""" + if not box: + return False + w = box.get("width", 0) + h = box.get("height", 0) + if w < 80 or h < 80 or w > 600 or h > 600: + return False + ratio = min(w, h) / max(w, h) + return ratio >= 0.75 + async def _grab_qr_in_frames(self) -> Optional[str]: """在所有 frame 内查找二维码 / 并转为 data URL。""" for frame in self._all_frames(): @@ -1989,47 +2760,198 @@ class DouyinWorker: try: if not await el.is_visible(): continue + box = await el.bounding_box() or {} + if not self._looks_like_qr_box(box): + logger.debug(f"QR selector matched but box unlikely: {sel} {box}") + continue src = await el.get_attribute("src") or "" if src.startswith("data:image"): - return src + logger.info(f"Captured QR via data-src in frame: {sel}") + return self._upscale_qr_image(src) if src.startswith("http"): try: resp = await self.page.request.get(src) if resp.ok: b64 = base64.b64encode(await resp.body()).decode("utf-8") - return f"data:image/png;base64,{b64}" + logger.info(f"Captured QR via http-src in frame: {sel}") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") except Exception: pass + # canvas 直接转 data URL + tag = await el.evaluate("e => e.tagName.toLowerCase()") + if tag == "canvas": + data_url = await el.evaluate("e => e.toDataURL('image/png')") + if data_url and data_url.startswith("data:image"): + logger.info(f"Captured QR via canvas.toDataURL in frame: {sel}") + return self._upscale_qr_image(data_url) shot = await el.screenshot(type="png") b64 = base64.b64encode(shot).decode("utf-8") - logger.info(f"Captured QR via element in frame: {sel}") - return f"data:image/png;base64,{b64}" + logger.info(f"Captured QR via element screenshot in frame: {sel}") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") except Exception as e: if "Execution context was destroyed" in str(e): return None + logger.debug(f"QR selector {sel} failed in frame: {e}") continue return None + async def _grab_qr_generic_in_frames(self) -> Optional[str]: + """泛化查找:在所有 frame 中找登录弹窗内最大的方型 img/canvas。""" + best_el = None + best_score = 0 + best_frame = None + login_container_sels = " ".join(self._QR_CONTAINER_SELECTORS + self._LOGIN_PANEL_SELECTORS) + + for frame in self._all_frames(): + try: + # 优先只在登录容器内查找 + candidates = await frame.query_selector_all( + f"{login_container_sels} img, {login_container_sels} canvas" + ) + if not candidates: + # 兜底:扫描全页 img/canvas + candidates = await frame.query_selector_all("img, canvas") + for el in candidates: + try: + if not await el.is_visible(): + continue + box = await el.bounding_box() or {} + if not self._looks_like_qr_box(box): + continue + # 优先选择长宽比接近 1:1 的 + w, h = box.get("width", 0), box.get("height", 0) + ratio_score = min(w, h) / max(w, h) + area = w * h + score = area * ratio_score + if score > best_score: + best_score = score + best_el = el + best_frame = frame + except Exception: + continue + except Exception as e: + logger.debug(f"Generic QR scan failed in frame {frame.url}: {e}") + continue + + if not best_el: + return None + + try: + tag = await best_el.evaluate("e => e.tagName.toLowerCase()") + if tag == "canvas": + data_url = await best_el.evaluate("e => e.toDataURL('image/png')") + if data_url and data_url.startswith("data:image"): + logger.info(f"Captured QR via generic canvas.toDataURL in frame {best_frame.url[:60]}") + return self._upscale_qr_image(data_url) + src = await best_el.get_attribute("src") or "" + if src.startswith("data:image"): + logger.info("Captured QR via generic data-src") + return self._upscale_qr_image(src) + if src.startswith("http"): + try: + resp = await self.page.request.get(src) + if resp.ok: + b64 = base64.b64encode(await resp.body()).decode("utf-8") + logger.info("Captured QR via generic http-src") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") + except Exception: + pass + shot = await best_el.screenshot(type="png") + b64 = base64.b64encode(shot).decode("utf-8") + logger.info("Captured QR via generic element screenshot") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") + except Exception as e: + if "Execution context was destroyed" in str(e): + return None + logger.debug(f"Generic QR capture failed: {e}") + return None + async def _grab_login_panel_shot(self) -> Optional[str]: """兜底:截取登录面板 / 登录 iframe 整块(含其中的二维码),用户仍可扫描。""" - for sel in self._QR_CONTAINER_SELECTORS + self._LOGIN_PANEL_SELECTORS: - try: - container = await self.page.query_selector(sel) - if not container or not await container.is_visible(): + # 所有 frame 都可能是登录面板(尤其是 passport iframe) + for frame in self._all_frames(): + for sel in self._QR_CONTAINER_SELECTORS + self._LOGIN_PANEL_SELECTORS: + try: + container = await frame.query_selector(sel) + if not container or not await container.is_visible(): + continue + box = await container.bounding_box() + if not box or box.get("width", 0) < 80 or box.get("height", 0) < 80: + continue + # 如果面板太小(只是容器),截图出来 QR 也会小,尝试放大 viewport 再截 + await self._ensure_panel_fits(box) + shot = await container.screenshot(type="png") + b64 = base64.b64encode(shot).decode("utf-8") + logger.info(f"Captured QR via panel screenshot: {sel} in frame {frame.url[:60]}") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") + except Exception as e: + if "Execution context was destroyed" in str(e): + return None + logger.debug(f"Panel screenshot {sel} failed: {e}") continue - box = await container.bounding_box() - if not box or box.get("width", 0) < 80 or box.get("height", 0) < 80: - continue - shot = await container.screenshot(type="png") - b64 = base64.b64encode(shot).decode("utf-8") - logger.info(f"Captured QR via panel screenshot: {sel}") - return f"data:image/png;base64,{b64}" - except Exception as e: - if "Execution context was destroyed" in str(e): - return None - continue return None + async def _ensure_panel_fits(self, box: dict): + """如果登录面板尺寸较大,临时放大 viewport 以保证截图清晰。""" + try: + needed_w = int(box.get("x", 0) + box.get("width", 0) + 50) + needed_h = int(box.get("y", 0) + box.get("height", 0) + 50) + cur = await self.page.viewport_size() + if cur and (needed_w > cur.get("width", 0) or needed_h > cur.get("height", 0)): + await self.page.set_viewport_size({ + "width": max(needed_w, cur.get("width", 1280)), + "height": max(needed_h, cur.get("height", 900)), + }) + await asyncio.sleep(0.3) + except Exception as e: + logger.debug(f"ensure_panel_fits skipped: {e}") + + async def _crop_center_viewport_shot(self) -> Optional[str]: + """截取视口中心区域,通常登录弹窗在此。""" + try: + viewport = await self.page.viewport_size() + vw, vh = viewport.get("width", 1280), viewport.get("height", 900) + # 中心 700x800 区域,覆盖常见登录弹窗 + cw, ch = min(700, vw), min(800, vh) + x = max(0, (vw - cw) // 2) + y = max(0, (vh - ch) // 2) + shot = await self.page.screenshot( + type="png", + clip={"x": x, "y": y, "width": cw, "height": ch}, + timeout=15000, + ) + b64 = base64.b64encode(shot).decode("utf-8") + logger.info(f"Captured QR via center viewport clip: {x},{y} {cw}x{ch}") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") + except Exception as e: + logger.debug(f"Center viewport clip failed: {e}") + return None + + def _upscale_qr_image(self, data_url: str, min_size: int = 280) -> str: + """如果二维码图像小于 min_size,使用 Pillow 放大,提高手机扫描成功率。""" + try: + if not data_url.startswith("data:image"): + return data_url + header, b64data = data_url.split(",", 1) + raw = base64.b64decode(b64data) + img = Image.open(io.BytesIO(raw)) + w, h = img.size + if w >= min_size and h >= min_size: + return data_url + scale = max(min_size / w, min_size / h) + new_size = (int(w * scale), int(h * scale)) + # 二维码用最近邻放大更锐利 + upscaled = img.resize(new_size, Image.NEAREST) + buf = io.BytesIO() + fmt = "PNG" if "png" in header else "JPEG" + upscaled.save(buf, format=fmt) + new_b64 = base64.b64encode(buf.getvalue()).decode("utf-8") + logger.info(f"Upscaled QR image from {w}x{h} to {new_size[0]}x{new_size[1]}") + return f"data:image/{fmt.lower()};base64,{new_b64}" + except Exception as e: + logger.debug(f"QR upscale failed: {e}") + return data_url + async def _check_and_grab_captcha(self) -> Optional[str]: """检测页面是否显示了验证码(滑块/点击等),如果显示了,则对验证码区域或整页截图。""" captcha_selectors = [ @@ -2086,7 +3008,12 @@ class DouyinWorker: return false; }""") if has_visible_captcha_text: - shot = await self.page.screenshot(type="png") + # 验证码通常出现在视口中央,截取中央区域避免整页字体加载超时 + center = await self._crop_center_viewport_shot() + if center: + logger.info("Captured center viewport screenshot due to detected captcha text") + return center + shot = await self.page.screenshot(type="png", timeout=15000) b64 = base64.b64encode(shot).decode("utf-8") logger.info("Captured full-page screenshot due to detected captcha text") return f"data:image/png;base64,{b64}" @@ -2096,34 +3023,49 @@ class DouyinWorker: return None async def _grab_qr_data_url(self) -> Optional[str]: - """统一二维码抓取:先检测验证码并截图,失败再找二维码,接着回退到登录面板,最后如果都失败,直接截取整页。""" + """统一二维码抓取:先检测验证码并截图,失败再找二维码,接着回退到登录面板 / 中心区域,最后才截整页。""" if not self._is_browser_alive(): return None - + # 1. 优先检测并截图验证码 captcha_img = await self._check_and_grab_captcha() if captcha_img: + logger.info("QR capture: returned captcha image") return captcha_img - - # 2. 正常获取二维码元素 + + # 2. 精确选择器获取二维码元素 qr = await self._grab_qr_in_frames() if qr: + logger.info("QR capture: returned via precise selector") return qr - - # 3. 登录面板截图 + + # 3. 泛化查找登录弹窗内的方型 img/canvas + generic_qr = await self._grab_qr_generic_in_frames() + if generic_qr: + logger.info("QR capture: returned via generic scan") + return generic_qr + + # 4. 登录面板 / iframe 截图 panel = await self._grab_login_panel_shot() if panel: + logger.info("QR capture: returned via login panel screenshot") return panel - # 4. 终极兜底:直接截取整个网页视口(保证无论如何都有画面,而不是转圈) + # 5. 中心区域兜底(比整页截图更聚焦,二维码不会太小) + center = await self._crop_center_viewport_shot() + if center: + logger.info("QR capture: returned via center viewport clip") + return center + + # 6. 终极兜底:直接截取整个网页视口 try: - shot = await self.page.screenshot(type="png") + shot = await self.page.screenshot(type="png", timeout=15000) b64 = base64.b64encode(shot).decode("utf-8") - logger.info("Captured QR via full-viewport screenshot fallback") - return f"data:image/png;base64,{b64}" + logger.warning("QR capture: fell back to full-viewport screenshot") + return self._upscale_qr_image(f"data:image/png;base64,{b64}") except Exception as e: logger.warning(f"Full-page screenshot fallback failed: {e}") - + return None async def _refresh_qr_image(self): diff --git a/backend/rpa_engine/runtime_config.py b/backend/rpa_engine/runtime_config.py index b78f8f7..71163b2 100644 --- a/backend/rpa_engine/runtime_config.py +++ b/backend/rpa_engine/runtime_config.py @@ -1,129 +1,145 @@ -"""运行时环境配置:住宅代理 + 浏览器显示。 - -用于解决「部署到云服务器后」两类常见问题: - 1. 机房 IP 触发抖音风控(7911)—— 通过 KEFU_DOUYIN_PROXY 让抖音请求走住宅代理。 - 2. 无图形界面的 Linux 起不来有头浏览器 —— 自动拉起 Xvfb 虚拟显示。 - -全部通过环境变量控制,无需改代码: - - KEFU_DOUYIN_PROXY 抖音 IM HTTP 请求与浏览器登录走的代理,绕开机房 IP 风控。 - 形如 http://user:pass@host:port 或 socks5://host:port - KEFU_BROWSER_HEADLESS 是否使用无头浏览器(1/true 开启)。默认 false:抖音安全 SDK - 对 headless 判定严格,无头易生成无效 ts_sign,反而刷新无效。 -""" -import asyncio -import logging -import os -from typing import Optional -from urllib.parse import urlparse - -logger = logging.getLogger("rpa_engine.runtime") - -_TRUE = {"1", "true", "yes", "on"} -_FALSE = {"0", "false", "no", "off"} - -_NO_DISPLAY_HINT = ( - "当前是无图形界面的 Linux 服务器,且无法启动虚拟显示来运行有头浏览器。" - "抖音安全 SDK 对 headless 判定严格,扫码登录 / 刷新凭证需要有头 Chromium。请任选其一:\n" - " 1) 安装 Xvfb + pyvirtualdisplay,让程序自动拉起虚拟显示:\n" - " Debian/Ubuntu: apt install -y xvfb && pip install pyvirtualdisplay\n" - " CentOS/Rocky : yum install -y xorg-x11-server-Xvfb && pip install pyvirtualdisplay\n" - " 2) 或用 xvfb-run 启动后端:xvfb-run -a ./start_web.sh\n" - " 3) 或设置 KEFU_BROWSER_HEADLESS=1 强制无头(更易触发抖音风控,不推荐)。" -) - - -def get_douyin_proxy() -> Optional[str]: - """读取抖音请求代理 URL(未配置返回 None)。""" - val = (os.getenv("KEFU_DOUYIN_PROXY") or "").strip() - return val or None - - -def httpx_proxy() -> Optional[str]: - """供 httpx.AsyncClient(proxy=...) 使用的代理 URL。""" - return get_douyin_proxy() - - -def requests_proxies() -> Optional[dict]: - """供 requests.get(proxies=...) 使用的代理字典。""" - url = get_douyin_proxy() - if not url: - return None - return {"http": url, "https": url} - - -def playwright_proxy() -> Optional[dict]: - """转成 Playwright launch(proxy=...) 所需结构(未配置或无法解析返回 None)。""" - url = get_douyin_proxy() - if not url: - return None - parsed = urlparse(url) - if not parsed.hostname: - logger.warning("KEFU_DOUYIN_PROXY 格式无法解析,已忽略:%s", url) - return None - server = f"{parsed.scheme or 'http'}://{parsed.hostname}" - if parsed.port: - server += f":{parsed.port}" - proxy: dict[str, str] = {"server": server} - if parsed.username: - proxy["username"] = parsed.username - if parsed.password: - proxy["password"] = parsed.password - return proxy - - -def resolve_headless(default: bool = False) -> bool: - """根据 KEFU_BROWSER_HEADLESS 决定是否无头;未设置时用 default。""" - val = (os.getenv("KEFU_BROWSER_HEADLESS") or "").strip().lower() - if val in _TRUE: - return True - if val in _FALSE: - return False - return default - - -# 进程内仅启动一次的虚拟显示(Xvfb)句柄 -_virtual_display = None -_virtual_display_failed = False - - -def _start_virtual_display_sync() -> Optional[str]: - """在无 DISPLAY 的 Linux 上启动一次 Xvfb 虚拟显示(阻塞,需放线程执行)。""" - global _virtual_display, _virtual_display_failed - - # 仅 Linux 且无 DISPLAY 时才需要虚拟显示;Windows/macOS 有桌面,直接返回。 - if os.name != "posix": - return os.environ.get("DISPLAY") - if os.environ.get("DISPLAY"): - return os.environ["DISPLAY"] - if _virtual_display is not None: - return os.environ.get("DISPLAY") - if _virtual_display_failed: - raise RuntimeError(_NO_DISPLAY_HINT) - - try: - from pyvirtualdisplay import Display - except ImportError as e: - _virtual_display_failed = True - raise RuntimeError(_NO_DISPLAY_HINT) from e - - try: - disp = Display(visible=False, size=(1280, 800)) - disp.start() # 设置 os.environ['DISPLAY'] - except Exception as e: - _virtual_display_failed = True - raise RuntimeError(_NO_DISPLAY_HINT) from e - - _virtual_display = disp - logger.info("已启动 Xvfb 虚拟显示 DISPLAY=%s 供有头浏览器使用", os.environ.get("DISPLAY")) - return os.environ.get("DISPLAY") - - -async def ensure_browser_display(headless: bool) -> None: - """有头模式在无 DISPLAY 的 Linux 上自动拉起 Xvfb 虚拟显示。 - - headless=True 时无需显示,直接返回;启动失败抛出带操作指引的 RuntimeError。 - """ - if headless: - return - await asyncio.to_thread(_start_virtual_display_sync) +"""运行时环境配置:住宅代理 + 浏览器显示。 + +用于解决「部署到云服务器后」两类常见问题: + 1. 机房 IP 触发抖音风控(7911)—— 通过 KEFU_DOUYIN_PROXY 让抖音请求走住宅代理。 + 2. 无图形界面的 Linux 起不来有头浏览器 —— 自动拉起 Xvfb 虚拟显示。 + +全部通过环境变量控制,无需改代码: + + KEFU_DOUYIN_PROXY 抖音 IM HTTP 请求与浏览器登录走的代理,绕开机房 IP 风控。 + 形如 http://user:pass@host:port 或 socks5://host:port + KEFU_BROWSER_HEADLESS 是否使用无头浏览器(1/true 开启)。默认 false:抖音安全 SDK + 对 headless 判定严格,无头易生成无效 ts_sign,反而刷新无效。 +""" +import asyncio +import logging +import os +from typing import Optional +from urllib.parse import urlparse + +logger = logging.getLogger("rpa_engine.runtime") + +_TRUE = {"1", "true", "yes", "on"} +_FALSE = {"0", "false", "no", "off"} + +_NO_DISPLAY_HINT = ( + "当前是无图形界面的 Linux 服务器,且无法启动虚拟显示来运行有头浏览器。" + "抖音安全 SDK 对 headless 判定严格,扫码登录 / 刷新凭证需要有头 Chromium。请任选其一:\n" + " 1) 安装 Xvfb + pyvirtualdisplay,让程序自动拉起虚拟显示:\n" + " Debian/Ubuntu: apt install -y xvfb && pip install pyvirtualdisplay\n" + " CentOS/Rocky : yum install -y xorg-x11-server-Xvfb && pip install pyvirtualdisplay\n" + " 2) 或用 xvfb-run 启动后端:xvfb-run -a ./start_web.sh\n" + " 3) 或设置 KEFU_BROWSER_HEADLESS=1 强制无头(更易触发抖音风控,不推荐)。" +) + + +def get_douyin_proxy() -> Optional[str]: + """读取抖音请求代理 URL(未配置返回 None)。""" + val = (os.getenv("KEFU_DOUYIN_PROXY") or "").strip() + return val or None + + +def httpx_proxy() -> Optional[str]: + """供 httpx.AsyncClient(proxy=...) 使用的代理 URL。""" + return get_douyin_proxy() + + +def requests_proxies() -> Optional[dict]: + """供 requests.get(proxies=...) 使用的代理字典。""" + url = get_douyin_proxy() + if not url: + return None + return {"http": url, "https": url} + + +def playwright_proxy() -> Optional[dict]: + """转成 Playwright launch(proxy=...) 所需结构(未配置或无法解析返回 None)。""" + url = get_douyin_proxy() + if not url: + return None + parsed = urlparse(url) + if not parsed.hostname: + logger.warning("KEFU_DOUYIN_PROXY 格式无法解析,已忽略:%s", url) + return None + server = f"{parsed.scheme or 'http'}://{parsed.hostname}" + if parsed.port: + server += f":{parsed.port}" + proxy: dict[str, str] = {"server": server} + if parsed.username: + proxy["username"] = parsed.username + if parsed.password: + proxy["password"] = parsed.password + return proxy + + +def resolve_headless(default: bool = False) -> bool: + """根据 KEFU_BROWSER_HEADLESS 决定是否无头;未设置时用 default。""" + val = (os.getenv("KEFU_BROWSER_HEADLESS") or "").strip().lower() + if val in _TRUE: + return True + if val in _FALSE: + return False + return default + + +def ui_conversation_page_budget(default: int = 3) -> int: + """用户点开会话列表时允许翻的收件箱页数(KEFU_UI_CONVERSATION_PAGES)。 + + 抖音收件箱按游标分页,一次请求只给一页(实测每页约 100-500KB);某账号翻 + 6 页拿到 35 个会话仍未翻完。所以必须有预算:只拿一页会把「其中一页」当成 + 完整会话列表,不设上限又可能为一次点击拉下好几 MB。默认 3 页只是折中, + 花多少流量换多完整的列表属于业务取舍,用环境变量调整即可。 + 无论调到多少,翻不完时都会并入本地历史,不会把残缺列表伪装成完整列表。 + """ + try: + value = int(os.getenv("KEFU_UI_CONVERSATION_PAGES", str(default)) or default) + except (TypeError, ValueError): + value = default + return max(1, min(20, value)) + + +# 进程内仅启动一次的虚拟显示(Xvfb)句柄 +_virtual_display = None +_virtual_display_failed = False + + +def _start_virtual_display_sync() -> Optional[str]: + """在无 DISPLAY 的 Linux 上启动一次 Xvfb 虚拟显示(阻塞,需放线程执行)。""" + global _virtual_display, _virtual_display_failed + + # 仅 Linux 且无 DISPLAY 时才需要虚拟显示;Windows/macOS 有桌面,直接返回。 + if os.name != "posix": + return os.environ.get("DISPLAY") + if os.environ.get("DISPLAY"): + return os.environ["DISPLAY"] + if _virtual_display is not None: + return os.environ.get("DISPLAY") + if _virtual_display_failed: + raise RuntimeError(_NO_DISPLAY_HINT) + + try: + from pyvirtualdisplay import Display + except ImportError as e: + _virtual_display_failed = True + raise RuntimeError(_NO_DISPLAY_HINT) from e + + try: + disp = Display(visible=False, size=(1280, 800)) + disp.start() # 设置 os.environ['DISPLAY'] + except Exception as e: + _virtual_display_failed = True + raise RuntimeError(_NO_DISPLAY_HINT) from e + + _virtual_display = disp + logger.info("已启动 Xvfb 虚拟显示 DISPLAY=%s 供有头浏览器使用", os.environ.get("DISPLAY")) + return os.environ.get("DISPLAY") + + +async def ensure_browser_display(headless: bool) -> None: + """有头模式在无 DISPLAY 的 Linux 上自动拉起 Xvfb 虚拟显示。 + + headless=True 时无需显示,直接返回;启动失败抛出带操作指引的 RuntimeError。 + """ + if headless: + return + await asyncio.to_thread(_start_virtual_display_sync) diff --git a/backend/tests/test_conversation_poll_bandwidth.py b/backend/tests/test_conversation_poll_bandwidth.py index fe272fc..2c8e165 100644 --- a/backend/tests/test_conversation_poll_bandwidth.py +++ b/backend/tests/test_conversation_poll_bandwidth.py @@ -30,67 +30,69 @@ class ConversationPollBandwidthTests(unittest.IsolatedAsyncioTestCase): account_id=9, ) - async def test_terminal_token_error_does_not_probe_other_payloads(self): + async def test_inbox_is_fetched_with_exactly_one_protobuf_request(self): + """imapi 只认 protobuf;轮询一次就只该发一个请求。""" client = self._make_client() - client._request = AsyncMock( - return_value={ - "status_code": 500, - "error_desc": "empty token", - "body": {}, - } - ) + client.fetch_inbox_messages = AsyncMock(return_value=[]) + client._request = AsyncMock() self.assertEqual(await client.get_conversations(), []) - client._request.assert_awaited_once() - self.assertEqual(client._request.await_args.args[0], "POST") + client.fetch_inbox_messages.assert_awaited_once() + # 不能再退回 JSON 的 /v1/conversation/list:那个请求恒被抖音拒绝。 + client._request.assert_not_awaited() - async def test_successful_empty_response_stops_after_first_payload(self): + async def test_transport_failure_is_recorded_and_returns_empty(self): client = self._make_client() - client._request = AsyncMock( - return_value={"status_code": 0, "body": {"conversation_list": []}} - ) - - self.assertEqual(await client.get_conversations(), []) - client._request.assert_awaited_once() - - async def test_parameter_error_can_fall_through_to_compatible_payload(self): - client = self._make_client() - client._request = AsyncMock( - side_effect=[ - {"status_code": 400, "error_desc": "invalid parameter"}, - {"status_code": 0, "body": {"conversation_list": []}}, - ] - ) - - self.assertEqual(await client.get_conversations(), []) - self.assertEqual(client._request.await_count, 2) - self.assertTrue( - all(call.args[0] == "POST" for call in client._request.await_args_list) - ) - - async def test_get_fallback_only_runs_after_transport_failure(self): - client = self._make_client() - client._request = AsyncMock( - side_effect=[None, {"status_code": 0, "body": {}}] - ) - - self.assertEqual(await client.get_conversations(), []) - self.assertEqual( - [call.args[0] for call in client._request.await_args_list], - ["POST", "GET"], - ) - - async def test_transport_outage_stops_after_one_post_and_get_pair(self): - client = self._make_client() - client._request = AsyncMock(return_value=None) + client.fetch_inbox_messages = AsyncMock(side_effect=RuntimeError("boom")) with self.assertLogs("douyin_im.http", level="WARNING"): self.assertEqual(await client.get_conversations(), []) - self.assertEqual(client._request.await_count, 2) + self.assertIn("boom", client.last_error) + + async def test_inbox_messages_group_into_one_row_per_conversation(self): + client = self._make_client() + client.fetch_inbox_messages = AsyncMock( + return_value=[ + { + "conversation_id": "0:1:10001:20001", + "server_message_id": "700", + "conversation_short_id": "555", + "message_type": 7, + "sender": "20001", + "content": '{"text":"旧"}', + }, + { + "conversation_id": "0:1:10001:20001", + "server_message_id": "900", + "conversation_short_id": "555", + "message_type": 7, + "sender": "20001", + "content": '{"text":"新"}', + }, + { + "conversation_id": "0:1:10001:20002", + "server_message_id": "800", + "message_type": 7, + "sender": "20002", + "content": '{"text":"另一个"}', + }, + ] + ) + + rows = await client.get_conversations(enrich_profiles=False) + + by_id = {r["conversation_id"]: r for r in rows} + self.assertEqual(len(rows), 2) + # 同一会话只保留 server_message_id 最大的那条 + self.assertEqual(by_id["0:1:10001:20001"]["server_message_id"], "900") + self.assertIn("新", by_id["0:1:10001:20001"]["content"]) + # peer_uid 由 conversation_id 推导,不能直接取 sender(可能是自己) + self.assertEqual(by_id["0:1:10001:20002"]["peer_uid"], "20002") + # 顺手缓存 short_id,发送时就不必再 create 一次会话 self.assertEqual( - [call.args[0] for call in client._request.await_args_list], - ["POST", "GET"], + client.session.conv_meta["0:1:10001:20001"]["conversation_short_id"], + "555", ) def test_websocket_reconciliation_is_slow_and_http_fallback_stays_fast(self): @@ -156,6 +158,7 @@ class ConversationPollBandwidthTests(unittest.IsolatedAsyncioTestCase): class _HttpClient: def __init__(self): self.get_conversations = AsyncMock(return_value=[]) + self.conversation_list_unsupported = False async def __aenter__(self): return self @@ -254,6 +257,7 @@ class ConversationPollBandwidthTests(unittest.IsolatedAsyncioTestCase): class _HttpClient: def __init__(self): self.get_conversations = AsyncMock(side_effect=snapshots) + self.conversation_list_unsupported = False self.enter_count = 0 self.exit_count = 0 diff --git a/backend/tests/test_egress_channels.py b/backend/tests/test_egress_channels.py index e810348..708d71c 100644 --- a/backend/tests/test_egress_channels.py +++ b/backend/tests/test_egress_channels.py @@ -8,6 +8,8 @@ from pathlib import Path from unittest.mock import AsyncMock, patch from sqlalchemy import create_engine, inspect, text +from sqlalchemy.dialects import mysql +from sqlalchemy.schema import CreateTable BACKEND_DIR = Path(__file__).resolve().parents[1] @@ -24,6 +26,7 @@ from rpa_engine.egress_channels import ( resolve_send_channels, ) from models.db_migrate import migrate_accounts_table +from models.models import Account class EgressChannelTests(unittest.IsolatedAsyncioTestCase): @@ -98,6 +101,13 @@ class EgressChannelTests(unittest.IsolatedAsyncioTestCase): class EgressMigrationTests(unittest.TestCase): + def test_mysql_accounts_uses_longtext_for_browser_payloads(self): + ddl = str(CreateTable(Account.__table__).compile(dialect=mysql.dialect())) + + self.assertIn("cookie_data LONGTEXT", ddl) + self.assertIn("im_session_data LONGTEXT", ddl) + self.assertIn("qr_code_base64 LONGTEXT", ddl) + def test_old_accounts_table_receives_egress_columns(self): engine = create_engine("sqlite:///:memory:") with engine.begin() as connection: diff --git a/backend/tests/test_im_receive_path.py b/backend/tests/test_im_receive_path.py new file mode 100644 index 0000000..609c48f --- /dev/null +++ b/backend/tests/test_im_receive_path.py @@ -0,0 +1,711 @@ +"""接收私信链路的回归测试。 + +覆盖三个曾让「托管中收不到抖音下发的私信」的缺陷: +1. frontier 长连接地址用了账号 UID 而不是设备号,握手成功却订阅错地址; +2. 浏览器本次登录抓到的真实 frontier 地址被 DB 里的旧地址挤掉; +3. PushFrame 负载是 gzip / payloadType 不是 'pb' 时整帧被丢弃。 +另外覆盖会话列表接口被抖音拒绝时不能再伪装成「收件箱为空」。 +""" +from __future__ import annotations + +import gzip +import json +import os +import sys +import unittest +from datetime import datetime +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + + +BACKEND_DIR = Path(__file__).resolve().parents[1] +os.environ.setdefault("KEFU_DB_TYPE", "sqlite") +os.environ.setdefault("KEFU_DATABASE_URL", "") +os.environ.setdefault("KEFU_DB_PATH", str(BACKEND_DIR / "kefu.db")) +if str(BACKEND_DIR) not in sys.path: + sys.path.insert(0, str(BACKEND_DIR)) + +from rpa_engine.douyin_im import frontier as frontier_module +from rpa_engine.douyin_im.auth import DouyinAuth +from rpa_engine.douyin_im.frontier import ensure_frontier_ws +from rpa_engine.douyin_im.http_client import DouyinImHttpClient +from rpa_engine.douyin_im.protocol import parse_ws_payload +from rpa_engine.douyin_im.session import DouyinImSession +from rpa_engine.douyin_im.static import Live_pb2, Response_pb2 +from rpa_engine.playwright_worker import DouyinWorker + + +def _build_push_frame( + *, + conversation_id: str, + sender: int, + content: str, + message_type: int = 7, + server_message_id: int = 7665317099296081465, + encoding: str = "", + payload_type: str = "pb", +) -> bytes: + response = Response_pb2.Response() + notify = response.body.new_message_notify + notify.conversation_id = conversation_id + message = notify.message + message.conversation_id = conversation_id + message.conversation_type = 1 + message.server_message_id = server_message_id + message.message_type = message_type + message.sender = sender + message.content = content + + body = response.SerializeToString() + if encoding == "gzip": + body = gzip.compress(body) + + frame = Live_pb2.PushFrame() + frame.seqId = 1 + frame.service = 6 + frame.method = 2 + frame.payloadEncoding = encoding + frame.payloadType = payload_type + frame.payload = body + return frame.SerializeToString() + + +class FrontierAddressTests(unittest.TestCase): + """frontier 按 device_id 寻址,不能用账号 UID 顶替。""" + + def _session(self) -> DouyinImSession: + return DouyinImSession( + cookies={"sessionid": "6313fec013ec0000000000000000abcd"}, + # query/user 返回的 id:本浏览器的设备注册号 + device_id="7678285795559818786", + web_id="7678286623234475535", + my_uid=2609567359568155, + uid_verified=True, + ) + + def test_built_url_uses_device_id_not_account_uid(self): + session = self._session() + + url = ensure_frontier_ws(session) + + self.assertIsNotNone(url) + self.assertIn("device_id=7678285795559818786", url) + self.assertNotIn("device_id=2609567359568155", url) + + def test_missing_device_id_falls_back_to_query_user_lookup(self): + session = self._session() + session.device_id = "" + session.web_id = "" + + with patch.object( + frontier_module, + "fetch_device_id", + return_value="7678285795559818786", + ) as fetch: + url = ensure_frontier_ws(session) + + fetch.assert_called_once() + self.assertIn("device_id=7678285795559818786", url) + + def test_proto_auth_keeps_device_id_for_verified_uid(self): + session = self._session() + + auth = DouyinAuth.from_im_session(session) + + self.assertEqual(auth.device_id, "7678285795559818786") + + +class CapturedFrontierUrlTests(unittest.IsolatedAsyncioTestCase): + """浏览器本次抓到的真实地址必须压过 DB 里的旧地址。""" + + async def test_fresh_browser_ws_url_wins_over_cached_url(self): + cached = ( + "wss://frontier-im.douyin.com/ws/v2?aid=6383&device_platform=douyin_pc" + "&fpid=9&device_id=7678285795559818786&token=stale&access_key=stale" + ) + captured = ( + "wss://frontier31-normal.zijieapi.com/ws/v2?aid=6383&fpid=9" + "&device_id=7678285795559818786&access_key=realkey&token=realtoken" + ) + saved = DouyinImSession( + cookies={"sessionid": "6313fec013ec0000000000000000abcd"}, + ws_urls=[cached], + device_id="7678285795559818786", + my_uid=2609567359568155, + ) + row = SimpleNamespace( + im_session_data=json.dumps(saved.to_dict()), + cookie_updated_at=datetime.utcnow(), + uid=None, + profile_updated_at=None, + ) + result = MagicMock() + result.one_or_none.return_value = row + db = SimpleNamespace(execute=AsyncMock(return_value=result), close=AsyncMock()) + + worker = DouyinWorker(account_id=400, login_mode="browser") + worker.get_db = AsyncMock(return_value=db) + worker._load_raw_user_agent = AsyncMock(return_value="test-agent") + + session = await worker._build_im_session_from_storage( + {"cookies": [{"name": "sessionid", "value": "6313fec013ec0000000000000000abcd"}]}, + {"ws_urls": [captured]}, + ) + + self.assertEqual(session.ws_urls[0], captured) + self.assertEqual(ensure_frontier_ws(session), captured) + + +class PushFramePayloadTests(unittest.TestCase): + """PushFrame 负载的编码/类型不能再决定整帧被不被丢弃。""" + + def test_plain_protobuf_frame_is_parsed(self): + raw = _build_push_frame( + conversation_id="0:1:869032150442612:2609567359568155", + sender=869032150442612, + content=json.dumps({"text": "你好", "aweType": 700}, ensure_ascii=False), + ) + + messages = parse_ws_payload(raw) + + self.assertEqual(len(messages), 1) + self.assertEqual(messages[0]["sender_uid"], "869032150442612") + self.assertEqual( + messages[0]["conversation_id"], "0:1:869032150442612:2609567359568155" + ) + + def test_gzip_encoded_frame_is_parsed(self): + raw = _build_push_frame( + conversation_id="0:1:869032150442612:2609567359568155", + sender=869032150442612, + content=json.dumps({"text": "在吗", "aweType": 700}, ensure_ascii=False), + encoding="gzip", + ) + + messages = parse_ws_payload(raw) + + self.assertEqual(len(messages), 1) + self.assertEqual(messages[0]["sender_uid"], "869032150442612") + + def test_non_pb_payload_type_is_still_parsed(self): + # 现网 frontier 帧会带 payloadType='text/json';只认 'pb' 会整帧丢弃。 + raw = _build_push_frame( + conversation_id="0:1:869032150442612:2609567359568155", + sender=869032150442612, + content=json.dumps({"text": "在吗", "aweType": 700}, ensure_ascii=False), + payload_type="text/json", + ) + + messages = parse_ws_payload(raw) + + self.assertEqual(len(messages), 1) + self.assertEqual(messages[0]["sender_uid"], "869032150442612") + + def test_empty_payload_control_frame_yields_no_message(self): + frame = Live_pb2.PushFrame() + frame.service = 6 + frame.method = 2 + frame.payloadEncoding = "utf-8" + frame.payloadType = "text/json" + + self.assertEqual(parse_ws_payload(frame.SerializeToString()), []) + + +class InboxProtobufTests(unittest.TestCase): + """imapi 只认 protobuf:解析真实响应形状,而不是 JSON。""" + + @staticmethod + def _response(*, cmd=200, status=0, message="OK", messages=()): + from rpa_engine.douyin_im.http_client import _pb_int, _pb_msg, _pb_str + + entries = b"" + for m in messages: + entries += _pb_msg( + 1, + _pb_str(1, m["conversation_id"]) + + _pb_int(3, m["server_message_id"]) + + _pb_int(5, m.get("conversation_short_id", 0)) + + _pb_int(6, m.get("message_type", 7)) + + _pb_int(7, m["sender"]) + + _pb_str(8, m.get("content", "")), + ) + return ( + _pb_int(1, cmd) + + _pb_int(3, status) + + _pb_str(4, message) + + _pb_msg(6, _pb_msg(cmd, entries)) + ) + + def test_status_is_read_from_the_protobuf_envelope(self): + from rpa_engine.douyin_im.http_client import _pb_response_status + + ok = self._response() + self.assertEqual(_pb_response_status(ok), (0, "OK")) + + rejected = self._response(status=1, message="unexepcted session length") + self.assertEqual( + _pb_response_status(rejected), + (1, "unexepcted session length"), + ) + + def test_message_bodies_are_extracted_from_the_inbox_response(self): + from rpa_engine.douyin_im.http_client import _pb_parse_inbox_messages + + raw = self._response( + messages=[ + { + "conversation_id": "0:1:2609567359568155:869032150442612", + "server_message_id": 7678140298052355621, + "conversation_short_id": 7654765893796266545, + "message_type": 7, + "sender": 869032150442612, + "content": '{"text":"你好"}', + } + ] + ) + + parsed = _pb_parse_inbox_messages(raw, 200) + + self.assertEqual(len(parsed), 1) + self.assertEqual( + parsed[0]["conversation_id"], + "0:1:2609567359568155:869032150442612", + ) + self.assertEqual(parsed[0]["server_message_id"], "7678140298052355621") + self.assertEqual(parsed[0]["sender"], "869032150442612") + self.assertIn("你好", parsed[0]["content"]) + + def test_unrelated_protobuf_is_not_mistaken_for_a_message(self): + from rpa_engine.douyin_im.http_client import ( + _pb_int, _pb_msg, _pb_parse_inbox_messages, _pb_str, + ) + + # 一段带字符串字段 1 但不是 conversation_id 的子消息 + noise = _pb_msg(6, _pb_msg(200, _pb_msg(1, _pb_str(1, "not-a-conv") + _pb_int(3, 5)))) + self.assertEqual(_pb_parse_inbox_messages(noise, 200), []) + + def test_empty_inbox_yields_no_messages(self): + from rpa_engine.douyin_im.http_client import _pb_parse_inbox_messages + + self.assertEqual(_pb_parse_inbox_messages(self._response(), 200), []) + + +class InboxCursorAndListTests(unittest.IsolatedAsyncioTestCase): + """轮询用小窗口,用户点开列表用全量——同一个 cmd,只是游标不同。""" + + def _client(self) -> DouyinImHttpClient: + session = DouyinImSession( + cookies={"sessionid": "s", "x_tt_token": "00" + "a" * 353}, + device_id="7678285795559818786", + my_uid=2609567359568155, + ) + return DouyinImHttpClient(session, account_id=405) + + @staticmethod + def _cursor_from_payload(payload: bytes) -> int: + from rpa_engine.douyin_im.http_client import _pb_parse_fields + + for fn, wt, val in _pb_parse_fields(payload): + if fn != 8 or wt != 2: + continue + for bfn, bwt, bval in _pb_parse_fields(val): + if bfn != 200 or bwt != 2: + continue + for cfn, cwt, cval in _pb_parse_fields(bval): + if cfn == 1 and cwt == 0: + return int(cval) + return -1 + + async def _capture_cursor(self, **kwargs) -> int: + from rpa_engine.douyin_im.http_client import _pb_int, _pb_str + + client = self._client() + captured: dict = {} + + async def fake_post(url, auth, payload, **_kw): + captured["payload"] = payload + return SimpleNamespace( + content=_pb_int(1, 200) + _pb_int(3, 0) + _pb_str(4, "OK"), + raise_for_status=lambda: None, + ) + + with patch.object(client, "_post_protobuf", fake_post): + await client.fetch_inbox_messages(**kwargs) + return self._cursor_from_payload(captured["payload"]) + + async def test_poll_window_sends_a_recent_microsecond_cursor(self): + import time as _time + + cursor = await self._capture_cursor(lookback_seconds=1800) + now_us = int(_time.time() * 1_000_000) + + self.assertGreater(cursor, 0) + # 游标应落在「大约半小时前」,允许几秒误差 + self.assertLess(now_us - cursor, int(1810 * 1_000_000)) + self.assertGreater(now_us - cursor, int(1790 * 1_000_000)) + + async def test_zero_lookback_means_no_cursor_not_now(self): + # lookback=0 若被算成 now,就只要「比此刻更新」的消息,永远是空列表。 + self.assertEqual(await self._capture_cursor(lookback_seconds=0), 0) + + @staticmethod + def _page(*, entries=(), next_cursor=0, has_more=False, cmd=200): + from rpa_engine.douyin_im.http_client import _pb_int, _pb_msg, _pb_str + + inner = b"" + for short_id, conv_id in entries: + inner += _pb_msg(6, _pb_int(1, short_id) + _pb_str(4, conv_id)) + inner += _pb_int(2, next_cursor) + _pb_int(3, 1 if has_more else 0) + return ( + _pb_int(1, cmd) + + _pb_int(3, 0) + + _pb_str(4, "OK") + + _pb_msg(6, _pb_msg(cmd, inner)) + ) + + async def test_paging_follows_the_cursor_and_dedupes_conversations(self): + client = self._client() + pages = [ + self._page( + entries=[(1, "0:1:10001:20001"), (2, "0:1:10001:20002")], + next_cursor=111, + has_more=True, + ), + self._page( + # 第二页重复一个、新增一个 + entries=[(2, "0:1:10001:20002"), (3, "0:1:10001:20003")], + next_cursor=222, + has_more=True, + ), + ] + cursors: list[int] = [] + + async def fake_post(url, auth, payload, **_kw): + cursors.append(self._cursor_from_payload(payload)) + return SimpleNamespace( + content=pages[len(cursors) - 1], raise_for_status=lambda: None + ) + + with patch.object(client, "_post_protobuf", fake_post): + await client.fetch_inbox_messages(lookback_seconds=0, max_pages=2) + + self.assertEqual(cursors, [0, 111]) + self.assertEqual( + [c["conversation_id"] for c in client._last_inbox_conversations], + ["0:1:10001:20001", "0:1:10001:20002", "0:1:10001:20003"], + ) + # 预算用完但抖音还说 has_more:必须承认列表不完整 + self.assertTrue(client.inbox_truncated) + + async def test_last_page_is_not_reported_as_truncated(self): + client = self._client() + page = self._page(entries=[(1, "0:1:10001:20001")], has_more=False) + + async def fake_post(url, auth, payload, **_kw): + return SimpleNamespace(content=page, raise_for_status=lambda: None) + + with patch.object(client, "_post_protobuf", fake_post): + await client.fetch_inbox_messages(lookback_seconds=0, max_pages=5) + + self.assertFalse(client.inbox_truncated) + + async def test_a_stalled_cursor_stops_paging(self): + client = self._client() + # 抖音回 has_more=1 但游标不前进:不能无限翻同一页 + page = self._page( + entries=[(1, "0:1:10001:20001")], next_cursor=0, has_more=True + ) + calls = {"n": 0} + + async def fake_post(url, auth, payload, **_kw): + calls["n"] += 1 + return SimpleNamespace(content=page, raise_for_status=lambda: None) + + with patch.object(client, "_post_protobuf", fake_post): + await client.fetch_inbox_messages(lookback_seconds=0, max_pages=10) + + self.assertEqual(calls["n"], 1) + + async def test_conversations_without_recent_messages_still_listed(self): + client = self._client() + client.fetch_inbox_messages = AsyncMock(return_value=[]) + client._last_inbox_conversations = [ + {"conversation_id": "0:1:10001:20001", "conversation_short_id": "555"}, + {"conversation_id": "0:1:10001:20002", "conversation_short_id": "666"}, + ] + + rows = await client.get_conversations( + enrich_profiles=False, lookback_seconds=0 + ) + + self.assertEqual( + {r["conversation_id"] for r in rows}, + {"0:1:10001:20001", "0:1:10001:20002"}, + ) + self.assertEqual( + client.session.conv_meta["0:1:10001:20002"]["conversation_short_id"], + "666", + ) + + async def test_control_frames_never_become_a_conversation_preview(self): + client = self._client() + client.fetch_inbox_messages = AsyncMock( + return_value=[ + { + "conversation_id": "0:1:10001:20001", + "server_message_id": "100", + "message_type": 7, + "sender": "20001", + "content": '{"text":"真实消息"}', + }, + { + "conversation_id": "0:1:10001:20001", + "server_message_id": "200", + "message_type": 50001, + "sender": "20001", + "content": '{"command_type":6,"conversation_id":"0:1:10001:20001"}', + }, + ] + ) + + rows = await client.get_conversations(enrich_profiles=False) + + # 控制帧 server_message_id 更大,但不能顶掉真实消息成为预览, + # 否则 _handle_incoming 会拿它去匹配自动回复。 + self.assertEqual(len(rows), 1) + self.assertIn("真实消息", rows[0]["content"]) + self.assertEqual(rows[0]["server_message_id"], "100") + + +class ReadRequestTokenTests(unittest.TestCase): + """读接口的 Request.token 必须是 x_tt_token。 + + 带 auth.ticket 时抖音照样回 status_code=0 "OK",但把调用方当匿名用户, + 正文恒为空——和「收件箱没有消息」完全无法区分,是最难发现的那类故障。 + 实测同一请求只换 token:auth.ticket 73 字节 0 条,x_tt_token 113KB 47 条。 + """ + + def _auth(self): + session = DouyinImSession( + cookies={ + "sessionid": "6313fec013ec0000000000000000abcd", + "x_tt_token": "00" + "a" * 353, + }, + device_id="7678285795559818786", + my_uid=2609567359568155, + ) + return DouyinAuth.from_im_session(session) + + def test_read_request_uses_x_tt_token(self): + from rpa_engine.douyin_im.proto_builder import ProtoBuilder + + auth = self._auth() + request = ProtoBuilder.build_read_request(auth, 200) + + self.assertEqual(request.token, "00" + "a" * 353) + self.assertNotEqual(request.token, auth.ticket) + + def test_read_request_keeps_ticket_when_cookie_missing(self): + from rpa_engine.douyin_im.proto_builder import ProtoBuilder + + session = DouyinImSession( + cookies={"sessionid": "6313fec013ec0000000000000000abcd"}, + device_id="7678285795559818786", + my_uid=2609567359568155, + ) + auth = DouyinAuth.from_im_session(session) + + request = ProtoBuilder.build_read_request(auth, 200) + + self.assertEqual(request.token, auth.ticket or "") + + def test_send_request_is_left_on_the_normal_envelope(self): + from rpa_engine.douyin_im.proto_builder import ProtoBuilder + + # 发送接口另有 bd-ticket-guard 签名且线上可用,不能顺手改掉它的 token。 + auth = self._auth() + request = ProtoBuilder.build_normal_request(auth, 100) + + self.assertEqual(request.token, auth.ticket or "") + + +class AuthoritativeUidTests(unittest.TestCase): + """imapi 响应字段 13 是抖音认定的本账号 IM uid。""" + + def test_response_uid_corrects_a_wrong_my_uid(self): + from rpa_engine.douyin_im.http_client import _pb_int, _pb_str + + session = DouyinImSession( + cookies={"sessionid": "s"}, + my_uid=7678285795559818786, # 误把 frontier 设备号当成了 IM uid + ) + client = DouyinImHttpClient(session, account_id=404) + content = ( + _pb_int(1, 200) + + _pb_int(3, 0) + + _pb_str(4, "OK") + + _pb_int(13, 2609567359568155) + ) + + client._adopt_authoritative_uid(content) + + self.assertEqual(session.my_uid, 2609567359568155) + self.assertTrue(session.uid_verified) + + def test_matching_uid_is_left_alone(self): + from rpa_engine.douyin_im.http_client import _pb_int, _pb_str + + session = DouyinImSession(cookies={"sessionid": "s"}, my_uid=2609567359568155) + client = DouyinImHttpClient(session, account_id=404) + content = _pb_int(1, 200) + _pb_int(3, 0) + _pb_str(4, "OK") + _pb_int( + 13, 2609567359568155 + ) + + client._adopt_authoritative_uid(content) + + self.assertEqual(session.my_uid, 2609567359568155) + self.assertFalse(session.uid_verified) + + +class ConversationListRejectionTests(unittest.IsolatedAsyncioTestCase): + """接口被拒不能再伪装成「收件箱为空」。""" + + def _client(self) -> DouyinImHttpClient: + session = DouyinImSession( + cookies={"sessionid": "6313fec013ec0000000000000000abcd"}, + device_id="7678285795559818786", + my_uid=2609567359568155, + ) + return DouyinImHttpClient(session, account_id=401) + + async def test_rejected_protobuf_response_is_reported(self): + client = self._client() + raw = InboxProtobufTests._response( + status=1, message="unexepcted session length" + ) + post = AsyncMock( + return_value=SimpleNamespace( + content=raw, raise_for_status=lambda: None + ) + ) + + with ( + patch.object(client, "_post_protobuf", post), + patch( + "rpa_engine.douyin_im.auth.DouyinAuth.from_im_session", + return_value=SimpleNamespace(source_ip=""), + ), + patch( + "rpa_engine.douyin_im.proto_builder.ProtoBuilder.build_normal_request", + return_value=SimpleNamespace(SerializeToString=lambda: b""), + ), + patch.object(client, "_report_conversation_list_rejected") as report, + ): + self.assertEqual(await client.fetch_inbox_messages(), []) + + post.assert_awaited_once() + report.assert_called_once_with("unexepcted session length") + + async def test_rejection_marks_the_endpoint_unsupported(self): + client = self._client() + client._report_conversation_list_rejected("unexepcted session length") + + self.assertTrue(client.conversation_list_unsupported) + self.assertIn("unexepcted session length", client.last_error) + + async def test_empty_but_successful_inbox_is_not_reported_as_failure(self): + client = self._client() + client.fetch_inbox_messages = AsyncMock(return_value=[]) + + with patch.object( + client, "_report_conversation_list_rejected" + ) as report: + self.assertEqual( + await client.get_conversations(enrich_profiles=False), [] + ) + + report.assert_not_called() + self.assertEqual(client.last_error, "") + self.assertFalse(client.conversation_list_unsupported) + + +class LoggedOutDetectionTests(unittest.IsolatedAsyncioTestCase): + """抖音回「用户未登录」时必须明确报出来,不能当成资料接口抖动。""" + + def test_status_code_8_is_reported_as_logged_out(self): + from rpa_engine import account_profile as ap + + auth = SimpleNamespace(cookie={}, msToken="t", get_uid=lambda: "938334054809296") + payloads = [ + {"status_code": 0, "user_uid": "938334054809296"}, + {"status_code": 8, "status_msg": "用户未登录", "user": None}, + {"status_code": 8, "status_msg": "用户未登录", "user": None}, + ] + responses = [SimpleNamespace(json=lambda v=v: v) for v in payloads] + + with ( + patch.object(ap, "_build_auth", return_value=(auth, "ua")), + patch.object(ap.requests, "get", side_effect=responses), + patch.object(ap, "generate_a_bogus", return_value="a-bogus"), + patch.object(ap, "generate_webid", return_value="web-id"), + patch.object(ap, "_requests_proxies", return_value=None), + ): + detail = ap.fetch_douyin_profile_detail_sync("cookie-json", "ua") + + self.assertTrue(detail["logged_out"]) + self.assertFalse(detail["fetched"]) + self.assertIn("用户未登录", detail["message"]) + + async def test_hosting_reports_logged_out_once(self): + worker = DouyinWorker(account_id=403, login_mode="im_direct") + with patch( + "rpa_engine.playwright_worker.system_logger.record" + ) as record: + await worker._report_douyin_logged_out("抖音返回「用户未登录」") + await worker._report_douyin_logged_out("抖音返回「用户未登录」") + + record.assert_called_once() + self.assertEqual(record.call_args.kwargs["level"], "error") + + +class ReconciliationBackoffTests(unittest.IsolatedAsyncioTestCase): + """被抖音拒绝过的接口不能每 120 秒再白打一次。""" + + def _service(self): + from rpa_engine.douyin_im.service import DouyinImService + + return DouyinImService( + session=DouyinImSession( + cookies={"sessionid": "6313fec013ec0000000000000000abcd"}, + device_id="7678285795559818786", + my_uid=2609567359568155, + ), + match_reply=AsyncMock(return_value=[]), + log_fn=AsyncMock(), + account_id=402, + ) + + async def test_second_poll_skips_a_rejected_endpoint(self): + service = self._service() + + client = MagicMock() + client.get_conversations = AsyncMock(return_value=[]) + client.conversation_list_unsupported = True + client.__aenter__ = AsyncMock(return_value=client) + client.__aexit__ = AsyncMock(return_value=False) + + with patch( + "rpa_engine.douyin_im.service.DouyinImHttpClient", + return_value=client, + ): + self.assertEqual(await service._poll_conversations(), []) + self.assertTrue(service._conversation_list_unsupported) + # 第二轮完全不再构造 HTTP 客户端 / 发请求 + self.assertEqual(await service._poll_conversations(), []) + + client.get_conversations.assert_awaited_once() + + +if __name__ == "__main__": + unittest.main() diff --git a/backend/tests/test_sec_user_id_guard.py b/backend/tests/test_sec_user_id_guard.py index 314507b..5337d4e 100644 --- a/backend/tests/test_sec_user_id_guard.py +++ b/backend/tests/test_sec_user_id_guard.py @@ -1,9 +1,11 @@ from __future__ import annotations +import json import os import sys import unittest from contextlib import asynccontextmanager +from datetime import datetime, timedelta from pathlib import Path from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock, patch @@ -18,13 +20,114 @@ if str(BACKEND_DIR) not in sys.path: from rpa_engine.douyin_im.session import DouyinImSession from rpa_engine import account_profile as account_profile_module +from rpa_engine.credential import build_im_session_from_storage from rpa_engine.playwright_worker import DouyinWorker class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): - async def test_im_direct_missing_sec_user_id_exits_before_online_state(self): + async def test_current_profile_uid_overrides_browser_runtime_uid(self): + now = datetime.utcnow() + row = SimpleNamespace( + im_session_data=None, + cookie_updated_at=now, + uid="2609567359568155", + profile_updated_at=now, + ) + result = MagicMock() + result.one_or_none.return_value = row + db = SimpleNamespace( + execute=AsyncMock(return_value=result), + close=AsyncMock(), + ) + worker = DouyinWorker(account_id=300, login_mode="im_direct") + worker.get_db = AsyncMock(return_value=db) + worker._load_raw_user_agent = AsyncMock(return_value="test-agent") + storage = { + "cookies": [{"name": "sessionid", "value": "test-session"}], + "my_uid": 7678285795559818786, + "origins": [ + { + "localStorage": [ + { + "name": "web_runtime_security_uid", + "value": "7678285795559818786", + } + ] + } + ], + } + + session = await worker._build_im_session_from_storage(storage) + + self.assertEqual(session.my_uid, 2609567359568155) + # device_id 必须与权威 UID 同步(protobuf 发送时 device_id 优先取 + # session.device_id,残留的浏览器采集值会导致 device_id != my_uid -> KICK) + self.assertEqual(session.device_id, "2609567359568155") + self.assertTrue(session.uid_verified) + db.close.assert_awaited_once() + + async def test_stale_profile_uid_does_not_override_new_cookie(self): + now = datetime.utcnow() + row = SimpleNamespace( + im_session_data=None, + cookie_updated_at=now, + uid="2609567359568155", + profile_updated_at=now - timedelta(seconds=1), + ) + result = MagicMock() + result.one_or_none.return_value = row + db = SimpleNamespace( + execute=AsyncMock(return_value=result), + close=AsyncMock(), + ) + worker = DouyinWorker(account_id=300, login_mode="im_direct") + worker.get_db = AsyncMock(return_value=db) + worker._load_raw_user_agent = AsyncMock(return_value="test-agent") + + session = await worker._build_im_session_from_storage( + { + "cookies": [{"name": "sessionid", "value": "test-session"}], + "my_uid": 7678285795559818786, + } + ) + + self.assertEqual(session.my_uid, 7678285795559818786) + self.assertFalse(session.uid_verified) + + def test_persisted_verified_uid_wins_in_generic_session_builder(self): + saved = DouyinImSession( + cookies={"sessionid": "test-session"}, + my_uid=2609567359568155, + device_id="7678285795559818786", + uid_verified=True, + ) + + session = build_im_session_from_storage( + { + "cookies": [{"name": "sessionid", "value": "test-session"}], + "my_uid": 7678285795559818786, + "origins": [ + { + "localStorage": [ + { + "name": "web_runtime_security_uid", + "value": "7678285795559818786", + } + ] + } + ], + }, + json.dumps(saved.to_dict()), + ) + + self.assertEqual(session.my_uid, 2609567359568155) + # device_id 同步为已核验 UID,避免凭证残留设备号导致 device_id != my_uid + self.assertEqual(session.device_id, "2609567359568155") + self.assertTrue(session.uid_verified) + + async def test_im_direct_missing_sec_user_id_continues_im_hosting(self): worker = DouyinWorker(account_id=301, login_mode="im_direct") - worker._require_sec_user_id = AsyncMock(return_value=False) + worker._best_effort_sec_user_id = AsyncMock(return_value="") worker._load_user_agent = AsyncMock(return_value="test-agent") im_session = DouyinImSession( cookies={"sessionid": "test-session"}, @@ -42,37 +145,34 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): {"cookies": [{"name": "sessionid", "value": "test-session"}]} ) - self.assertFalse(started) - self.assertIn("sec_user_id", reason) - worker._require_sec_user_id.assert_awaited_once() - require_call = worker._require_sec_user_id.await_args - self.assertTrue(require_call.kwargs["refresh_if_missing"]) - self.assertTrue(require_call.kwargs["refresh_if_stale"]) + self.assertTrue(started) + self.assertEqual(reason, "") + worker._best_effort_sec_user_id.assert_awaited_once_with( + refresh_if_missing=True, + refresh_if_stale=True, + ) worker._build_im_session_from_storage.assert_awaited_once() validate.assert_awaited_once_with(im_session) - worker._persist_im_session.assert_not_awaited() - worker._run_im_direct_service.assert_not_awaited() + worker._persist_im_session.assert_awaited_once() + worker._run_im_direct_service.assert_awaited_once_with(im_session) - async def test_running_guard_precedes_disabled_follow_welcome_setting(self): + async def test_disabled_follow_welcome_does_not_require_identity(self): worker = DouyinWorker(account_id=302, login_mode="im_direct") worker.is_running = True worker._im_service = SimpleNamespace(_running=True) - worker._require_sec_user_id = AsyncMock(return_value=False) - worker._refresh_follow_welcome_config = AsyncMock() + worker._best_effort_sec_user_id = AsyncMock() + worker._refresh_follow_welcome_config = AsyncMock( + return_value=(False, "", "") + ) worker.get_db = AsyncMock() await worker.follow_welcome_tick() - worker._require_sec_user_id.assert_awaited_once() - require_call = worker._require_sec_user_id.await_args - self.assertFalse(require_call.kwargs.get("refresh_if_missing", False)) - # The identity guard must run before the account's follow-welcome flag - # is queried. Otherwise accounts with that feature disabled could stay - # hosted indefinitely without a sec_user_id. - worker._refresh_follow_welcome_config.assert_not_awaited() + worker._refresh_follow_welcome_config.assert_awaited_once_with() + worker._best_effort_sec_user_id.assert_not_awaited() worker.get_db.assert_not_awaited() - async def test_cached_disabled_follow_setting_still_guards_missing_identity(self): + async def test_cached_disabled_follow_setting_skips_missing_identity(self): worker = DouyinWorker(account_id=311, login_mode="im_direct") worker.is_running = True worker._im_service = SimpleNamespace(_running=True) @@ -80,12 +180,12 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): worker._refresh_follow_welcome_config = AsyncMock( return_value=(False, "", "") ) - worker._require_sec_user_id = AsyncMock(return_value="") + worker._best_effort_sec_user_id = AsyncMock(return_value="") await worker.follow_welcome_tick() worker._refresh_follow_welcome_config.assert_awaited_once_with() - worker._require_sec_user_id.assert_awaited_once_with("托管运行中") + worker._best_effort_sec_user_id.assert_not_awaited() async def test_blank_sec_user_id_is_missing_and_stops_hosting(self): worker = DouyinWorker(account_id=303, login_mode="im_direct") @@ -297,6 +397,141 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): self.assertIn("托管已自动退出", status_call.kwargs["error_msg"]) record_system_log.assert_called_once() + def test_profile_payload_uid_outranks_query_user_and_cookie_uid(self): + """资料接口的 UID 必须压过 query/user 的 user_uid 和 cookie 兜底。 + + 实测同一个 Cookie:query/user 返回 user_uid=938334054809296, + 而账号资料 UID 是 2609567359568155。以前前者先占位,导致账号卡片 + 一直显示「用户938334054809296」,反查 sec_user_id 也永远失败。 + """ + auth = SimpleNamespace( + cookie={}, + msToken="test-ms-token", + get_uid=lambda: "938334054809296", + ) + payloads = [ + # 1) query/user:只有 user_uid,属于弱兜底 + {"status_code": 0, "user_uid": "938334054809296"}, + # 2) user/profile/self:权威账号资料 + { + "status_code": 0, + "user": { + "uid": "2609567359568155", + "nickname": "凤的心", + "sec_uid": "MS4wLjABAAAA-real-sec-user-id", + "unique_id": "39688250979", + "aweme_count": 50, + "follower_count": 154, + }, + }, + ] + responses = [SimpleNamespace(json=lambda value=value: value) for value in payloads] + + with ( + patch.object( + account_profile_module, "_build_auth", + return_value=(auth, "test-agent"), + ), + patch.object( + account_profile_module.requests, "get", side_effect=responses, + ), + patch.object( + account_profile_module, "generate_a_bogus", return_value="a-bogus", + ), + patch.object( + account_profile_module, "generate_webid", return_value="web-id", + ), + patch.object( + account_profile_module, "_requests_proxies", return_value=None, + ), + ): + detail = account_profile_module.fetch_douyin_profile_detail_sync( + "cookie-json", "test-agent", + ) + + self.assertEqual(detail["uid"], "2609567359568155") + self.assertEqual(detail["nickname"], "凤的心") + self.assertTrue(detail["fetched"]) + + async def test_sync_writes_one_identity_to_both_account_and_profile(self): + """卡片与详细资料必须来自同一次抓取,不能各抓一次抓出两个身份。""" + detail = { + "uid": "2609567359568155", + "nickname": "凤的心", + "avatar_url": "https://example.test/avatar.png", + "unique_id": "39688250979", + "signature": "", + "sec_user_id": "MS4wLjABAAAA-real-sec-user-id", + "sec_user_id_status": "found", + "video_count": 50, + "follower_count": 154, + "following_count": 162, + "total_favorited": 612, + "favoriting_count": 0, + "fetched": True, + "message": "", + } + account = SimpleNamespace( + id=1, + username="用户938334054809296", + douyin_uid="938334054809296", + avatar_url=None, + user_agent="test-agent", + ) + profile = SimpleNamespace( + account_id=1, uid=None, nickname=None, avatar_url=None, + unique_id=None, signature=None, sec_user_id=None, + follower_count=None, following_count=None, total_favorited=None, + favoriting_count=None, video_count=None, synced_at=None, + sync_message=None, + ) + profile_result = MagicMock() + profile_result.scalar_one_or_none.return_value = profile + username_result = MagicMock() + username_result.scalar_one_or_none.return_value = None + db = SimpleNamespace( + execute=AsyncMock(side_effect=lambda stmt: ( + username_result if "accounts.username" in str(stmt).lower() + or "username" in str(stmt).lower() else profile_result + )), + add=MagicMock(), + commit=AsyncMock(), + refresh=AsyncMock(), + ) + + with ( + patch.object( + account_profile_module, + "fetch_douyin_profile_detail_with_sec_user_id", + new=AsyncMock(return_value=detail), + ), + patch.object( + account_profile_module, + "fetch_douyin_user_videos", + new=AsyncMock(return_value={"videos": [], "message": "无作品"}), + ), + patch.object( + account_profile_module, + "fetch_douyin_profile", + new=AsyncMock(side_effect=AssertionError("must not re-fetch")), + ), + patch.object( + account_profile_module, + "load_account_profile_from_db", + new=AsyncMock(return_value={}), + ), + ): + await account_profile_module.sync_account_profile_to_db( + db, account, "cookie-json", + ) + + # 同一个身份同时写进 accounts 行和 account_profile_details 行 + self.assertEqual(account.douyin_uid, "2609567359568155") + self.assertEqual(account.username, "凤的心") + self.assertEqual(account.avatar_url, "https://example.test/avatar.png") + self.assertEqual(profile.uid, "2609567359568155") + self.assertEqual(profile.nickname, "凤的心") + async def test_cookie_uid_without_valid_profile_payload_stays_unknown(self): auth = SimpleNamespace( cookie={}, @@ -490,7 +725,7 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): ), patch.object( account_profile_module, - "apply_douyin_profile", + "apply_profile_to_account", apply_profile, ), patch.object( @@ -518,19 +753,19 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): "profile endpoint temporarily unavailable", ) - async def test_browser_login_missing_sec_user_id_closes_browser_and_skips_im(self): + async def test_browser_login_missing_sec_user_id_still_starts_im(self): worker = DouyinWorker(account_id=305, login_mode="browser") worker.is_running = True events: list[str] = [] async def reject_identity(*_args, **_kwargs): events.append("require-sec-user-id") - return False + return "" worker._load_user_agent = AsyncMock(return_value="test-agent") worker._probe_existing_login = AsyncMock(return_value=True) worker._finalize_login_session = AsyncMock() - worker._require_sec_user_id = AsyncMock(side_effect=reject_identity) + worker._best_effort_sec_user_id = AsyncMock(side_effect=reject_identity) worker._setup_im_network_listener = AsyncMock() worker._navigate_to_message_center = AsyncMock() worker._harvest_im_credentials = AsyncMock() @@ -595,8 +830,8 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): ) worker._finalize_login_session.assert_awaited_once_with() - worker._require_sec_user_id.assert_awaited_once() - require_call = worker._require_sec_user_id.await_args + worker._best_effort_sec_user_id.assert_awaited_once() + require_call = worker._best_effort_sec_user_id.await_args self.assertTrue(require_call.kwargs["force_refresh"]) self.assertEqual( events, @@ -607,8 +842,10 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): worker._harvest_im_credentials.assert_awaited_once_with(timeout=25) worker._persist_cookies.assert_awaited_once_with() worker._build_im_session.assert_awaited_once_with() - worker._persist_im_session.assert_not_awaited() - worker._run_im_direct_service.assert_not_awaited() + # sec_user_id 只服务关注欢迎语;缺它不能阻断私信托管, + # 否则浏览器登录后账号立刻下线、永远不会自动回复。 + worker._persist_im_session.assert_awaited_once() + worker._run_im_direct_service.assert_awaited_once_with(im_session) worker._close_browser_only.assert_awaited_once_with() async def test_browser_login_with_sec_user_id_continues_to_im(self): @@ -617,7 +854,7 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): worker._load_user_agent = AsyncMock(return_value="test-agent") worker._probe_existing_login = AsyncMock(return_value=True) worker._finalize_login_session = AsyncMock() - worker._require_sec_user_id = AsyncMock(return_value=True) + worker._best_effort_sec_user_id = AsyncMock(return_value="sec-uid-306") worker._setup_im_network_listener = AsyncMock() worker._navigate_to_message_center = AsyncMock() worker._harvest_im_credentials = AsyncMock() @@ -674,8 +911,10 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase): }, ) - worker._require_sec_user_id.assert_awaited_once() - self.assertTrue(worker._require_sec_user_id.await_args.kwargs["force_refresh"]) + worker._best_effort_sec_user_id.assert_awaited_once() + self.assertTrue( + worker._best_effort_sec_user_id.await_args.kwargs["force_refresh"] + ) worker._setup_im_network_listener.assert_awaited_once_with() worker._navigate_to_message_center.assert_awaited_once_with() worker._harvest_im_credentials.assert_awaited_once_with(timeout=25) diff --git a/backend/tests/test_send_entry_and_worker_lifecycle.py b/backend/tests/test_send_entry_and_worker_lifecycle.py index 9af7375..dff39a3 100644 --- a/backend/tests/test_send_entry_and_worker_lifecycle.py +++ b/backend/tests/test_send_entry_and_worker_lifecycle.py @@ -18,9 +18,12 @@ if str(BACKEND_DIR) not in sys.path: sys.path.insert(0, str(BACKEND_DIR)) from rpa_engine.douyin_im import http_client as http_client_module +from rpa_engine.douyin_im.auth import DouyinAuth +from rpa_engine.douyin_im.frontier import ensure_frontier_ws from rpa_engine.douyin_im.http_client import DouyinImHttpClient from rpa_engine.douyin_im.session import DouyinImSession from rpa_engine.egress_channels import EgressChannel +from rpa_engine import playwright_worker as playwright_worker_module from rpa_engine.playwright_worker import DouyinWorker @@ -33,6 +36,75 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase): ) return DouyinImHttpClient(session, account_id=account_id) + def test_query_user_does_not_replace_existing_im_uid(self): + client = self._make_client() + auth = SimpleNamespace( + source_ip="", + get_uid=MagicMock(return_value=938334054809296), + ) + + resolved = client._resolve_authoritative_uid(auth) + + self.assertEqual(resolved, 10001) + self.assertEqual(client.session.my_uid, 10001) + auth.get_uid.assert_not_called() + + def test_verified_uid_does_not_replace_the_runtime_device_id(self): + # device_id 是 query/user 返回的设备注册号,my_uid 是账号 UID。 + # 拿 my_uid 顶替 device_id 会让 frontier 订阅到另一个地址:握手照样 + # 成功,却永远收不到这个账号的私信。 + session = DouyinImSession( + cookies={"sessionid": "test-session"}, + my_uid=2609567359568155, + device_id="7678285795559818786", + web_id="7678286623234475535", + uid_verified=True, + ) + + auth = DouyinAuth.from_im_session(session) + + self.assertEqual(auth.device_id, "7678285795559818786") + self.assertEqual(session.device_id, "7678285795559818786") + + def test_captured_tokenless_browser_frontier_url_is_kept(self): + url = ( + "wss://frontier100-normal.zijieapi.com/ws/v2?aid=6383&" + "device_platform=web&fpid=9&device_id=7678285795559818786&" + "access_key=0123456789abcdef0123456789abcdef&version_code=fws_1.0.0" + ) + session = DouyinImSession( + cookies={"sessionid": "test-session"}, + ws_urls=[url], + my_uid=2609567359568155, + device_id="7678285795559818786", + ) + + resolved = ensure_frontier_ws(session) + + self.assertEqual(resolved, url) + self.assertEqual(session.ws_urls, [url]) + + def test_non_im_frontier_product_is_rejected(self): + url = ( + "wss://frontier100-normal.zijieapi.com/ws/v2?aid=6383&" + "device_platform=web&fpid=971&device_id=7678285795559818786&" + "access_key=0123456789abcdef0123456789abcdef" + ) + session = DouyinImSession( + cookies={"sessionid": "test-session"}, + ws_urls=[url], + my_uid=2609567359568155, + device_id="7678285795559818786", + uid_verified=True, + ) + + resolved = ensure_frontier_ws(session) + + self.assertIn("frontier-im.douyin.com", resolved) + self.assertIn("fpid=9", resolved) + self.assertIn("device_id=7678285795559818786", resolved) + self.assertNotIn("fpid=971", resolved) + async def test_public_entry_submits_the_whole_send_to_outbound_queue(self): client = self._make_client(account_id=73) submit = AsyncMock(return_value=True) @@ -194,6 +266,59 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase): class WorkerLifecycleTests(unittest.IsolatedAsyncioTestCase): + async def test_virtual_display_starts_before_playwright_driver(self): + ensure_display = AsyncMock() + + async def start_driver(): + ensure_display.assert_awaited_once_with(False) + return "playwright-driver" + + manager = SimpleNamespace(start=AsyncMock(side_effect=start_driver)) + with ( + patch.object( + playwright_worker_module, + "resolve_headless", + return_value=False, + ), + patch.object( + playwright_worker_module, + "ensure_browser_display", + ensure_display, + ), + patch.object( + playwright_worker_module, + "async_playwright", + return_value=manager, + ), + ): + driver, headless = await playwright_worker_module._start_playwright_for_browser() + + self.assertEqual(driver, "playwright-driver") + self.assertFalse(headless) + manager.start.assert_awaited_once_with() + + async def test_visible_login_prompt_overrides_stale_sessionid(self): + worker = DouyinWorker(account_id=917, login_mode="browser") + worker._has_visible_login_prompt = AsyncMock(return_value=True) + worker.check_homepage_login_status = AsyncMock(return_value=True) + worker.check_logged_in_by_cookie = AsyncMock(return_value=True) + + logged_in = await worker._verify_login_state() + + self.assertFalse(logged_in) + worker.check_homepage_login_status.assert_not_awaited() + worker.check_logged_in_by_cookie.assert_not_awaited() + + async def test_homepage_message_entry_is_not_login_evidence(self): + worker = DouyinWorker(account_id=918, login_mode="browser") + worker._has_visible_login_prompt = AsyncMock(return_value=False) + worker.page = SimpleNamespace(query_selector=AsyncMock(return_value=None)) + + logged_in = await worker.check_homepage_login_status() + + self.assertFalse(logged_in) + worker.page.query_selector.assert_awaited_once() + async def test_start_saves_task_and_stop_waits_until_it_is_done(self): worker = DouyinWorker(account_id=919, login_mode="im_direct") loop_started = asyncio.Event() diff --git a/backend/tests/test_worker_scale_controls.py b/backend/tests/test_worker_scale_controls.py index d22dc32..a5d8d35 100644 --- a/backend/tests/test_worker_scale_controls.py +++ b/backend/tests/test_worker_scale_controls.py @@ -109,7 +109,7 @@ class WorkerScaleControlTests(unittest.IsolatedAsyncioTestCase): ) worker._load_user_agent = AsyncMock(return_value="test-agent") worker._build_im_session_from_storage = AsyncMock(return_value=session) - worker._require_sec_user_id = AsyncMock(return_value="sec-user") + worker._best_effort_sec_user_id = AsyncMock(return_value="sec-user") worker._persist_im_session = AsyncMock() worker._run_im_direct_service = AsyncMock() @@ -124,7 +124,10 @@ class WorkerScaleControlTests(unittest.IsolatedAsyncioTestCase): self.assertTrue(started) self.assertEqual(reason, "") validate.assert_not_awaited() - worker._require_sec_user_id.assert_awaited_once() + worker._best_effort_sec_user_id.assert_awaited_once_with( + refresh_if_missing=True, + refresh_if_stale=True, + ) worker._run_im_direct_service.assert_awaited_once_with(session) async def test_disabled_follow_welcome_uses_cached_lightweight_config(self): @@ -171,20 +174,18 @@ class WorkerScaleControlTests(unittest.IsolatedAsyncioTestCase): worker.get_db.assert_not_awaited() worker._require_sec_user_id.assert_not_awaited() - async def test_missing_cached_sec_user_id_stops_hosting(self): + async def test_disabled_follow_welcome_ignores_missing_cached_sec_user_id(self): worker = DouyinWorker(account_id=505) worker._im_service = SimpleNamespace(session=object()) worker._follow_config_loaded = True worker._refresh_follow_welcome_config = AsyncMock( return_value=(False, "", "") ) - worker._require_sec_user_id = AsyncMock(return_value="") + worker._best_effort_sec_user_id = AsyncMock(return_value="") await worker.follow_welcome_tick() - worker._require_sec_user_id.assert_awaited_once_with( - "托管运行中" - ) + worker._best_effort_sec_user_id.assert_not_awaited() if __name__ == "__main__": diff --git a/backend/tests/test_ws_client_scaling.py b/backend/tests/test_ws_client_scaling.py index c5f1177..1ec3854 100644 --- a/backend/tests/test_ws_client_scaling.py +++ b/backend/tests/test_ws_client_scaling.py @@ -20,7 +20,10 @@ from rpa_engine.douyin_im.session import DouyinImSession from rpa_engine.douyin_im.ws_client import DouyinImWsClient, _reconnect_delay -TEST_WS_URL = "wss://frontier-im.douyin.com/ws/v2?token=test-token-value" +TEST_WS_URL = ( + "wss://frontier-im.douyin.com/ws/v2?fpid=9&device_id=10001&" + "token=test-token-value" +) class _FakeWebSocket: @@ -108,6 +111,25 @@ class WebSocketScalingTests(unittest.IsolatedAsyncioTestCase): self.assertFalse(client.connected) self.assertIsNone(client._connection) + async def test_browser_frontier_uses_text_heartbeat_and_filters_ack(self): + client = self._make_client() + client._running = True + websocket = AsyncMock() + + heartbeat = asyncio.create_task(client._run_browser_heartbeat(websocket)) + for _ in range(20): + if websocket.send.await_count: + break + await asyncio.sleep(0) + heartbeat.cancel() + with self.assertRaises(asyncio.CancelledError): + await heartbeat + + websocket.send.assert_awaited_once_with("hi") + with patch.object(ws_module, "parse_ws_payload") as parse: + await client._dispatch("hi") + parse.assert_not_called() + async def test_starting_500_clients_does_not_create_os_threads(self): parked = asyncio.Event() diff --git a/backend/utils/cookie_store.py b/backend/utils/cookie_store.py index 88ed8f8..55e7fad 100644 --- a/backend/utils/cookie_store.py +++ b/backend/utils/cookie_store.py @@ -128,6 +128,10 @@ def _parse_tea_from_ls(origins: list) -> tuple[str, str]: parsed = json.loads(entry.get("value") or "{}") except Exception: continue + # __tea_cache_first_6383 等条目的值是纯数字(如 1),json.loads 返回 int, + # 不是 dict,必须跳过,否则 parsed.get() 抛 AttributeError 导致保存凭证 500。 + if not isinstance(parsed, dict): + continue uid = str(parsed.get("user_unique_id") or "").strip() wid = str(parsed.get("web_id") or "").strip() if uid and wid and uid == wid and len(uid) > 12: @@ -233,6 +237,25 @@ def validate_cookie_json(cookie_data: str) -> dict: return normalize_storage_state_for_im(data) +def extract_user_agent_from_cookie_data(cookie_data: Optional[str]) -> str: + """从已保存的 Cookie JSON 提取顶层 user_agent(登录头)。 + + Playwright storage_state 导出时顶层带 user_agent(采集浏览器当前 UA); + DYCRED 凭证经 convert_dycred_to_storage_state 后也统一落到 user_agent。 + 返回完整 UA 字符串;缺失/无效时返回空串,由调用方决定是否回填账号。 + """ + if not cookie_data: + return "" + try: + data = json.loads(cookie_data) + except Exception: + return "" + if not isinstance(data, dict): + return "" + ua = str(data.get("user_agent") or data.get("ua") or "").strip() + return ua + + def write_cookie_file(account_id: int, cookie_data: str) -> str: data = validate_cookie_json(cookie_data) path = get_cookie_path(account_id)