12 Commits
Author SHA1 Message Date
Your Name 89cae5b8cc 更新 2026-09-01 15:58:34 +08:00
Your Name 2fc864cc00 更新 2026-09-01 15:31:05 +08:00
Your Name 1f3addcf79 更新 2026-08-27 18:32:03 +08:00
Your Name 4ac6990efe 更新 2026-08-26 17:18:09 +08:00
Your Name 327a0bc42f 新增 2026-08-07 17:51:57 +08:00
Your Name 6119fdd767 更新 2026-08-07 15:35:02 +08:00
Your Name 3fc94c4a89 更新 2026-07-30 10:06:53 +08:00
Your Name 8f68af1c2c 更新 2026-07-28 15:04:17 +08:00
Your Name ac406a5f99 更新 2026-07-28 11:49:36 +08:00
Your Name f99a4edf83 更新 2026-07-28 09:13:07 +08:00
Your Name 153db97dc7 更新 2026-07-28 09:00:19 +08:00
Your Name 8ba13a8ff9 更新 2026-07-27 15:20:15 +08:00
107 changed files with 20336 additions and 2740 deletions
Binary file not shown.

After

Width:  |  Height:  |  Size: 350 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 341 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 399 KiB

+1
View File
@@ -10,6 +10,7 @@ backend/**/__pycache__/
backend/kefu.db
backend/kefu.db-journal
backend/sessions/
/douyin.zip
*.env
.env.*
+221
View File
@@ -0,0 +1,221 @@
# 2026-08-27 工作日志
## 抖音私信发送 decision=KICK 修复 + 第二套发送方案
### 根因诊断(已完成)
- 现象:账号1my_uid=2609567359568155)收消息正常(UA=Chrome/148),发消息被安全网关 decision=KICK。
- 根因:**UA 全链路不一致**——凭证采集时写入 storage_state/im_session_data 的 UA 是 Chrome/148,但 worker 构建 IM 会话时 `_load_user_agent()``resolve_user_agent` 回退默认 Chrome/120 覆盖了采集 UA。a_bogus 签名、IM 请求头、Protobuf body 用 Chrome/120,浏览器上下文/凭证却是 Chrome/148,设备指纹不一致触发 KICK。
- 数据库证据:`account.user_agent=None``im_session_data.user_agent=Chrome/120`(被覆盖后写入)、`cookie_data.user_agent=Chrome/148`
### 修改(backend/rpa_engine/playwright_worker.py
1. `__init__` 新增 `self._raw_user_agent = ""`
2. 新增 `_load_raw_user_agent()`:读账号表显式配置的 UA(未配置返回空串)。
3. `_build_im_session_from_storage`:账号表显式配置 UA 时用 `resolve_user_agent(raw_ua)`;否则**保留** storage_state 采集的 UA(不再覆盖),加日志。
4. 新增 `send_im_via_browser_page(conversation_id, content)`:第二套发送方案——构建 IM session → `DouyinImHttpClient.resolve_conversation_meta` 解析 ticket/short_id → `ProtoBuilder.build_send_message_request` 构造 protobuf → 打开抖音 /message 页面(非 headless 最小化)→ 页面 JS fetch `/v1/message/send`security-sdk 注入 a_bogus/bd-ticket-guard)→ `analyze_send_response` 判定。受 `KEFU_BROWSER_SEND_TIMEOUT`(默认45s)控制。
5. `_run_im_direct_service` 注入 `send_fallback=self.send_im_via_browser_page`
### 修改(backend/rpa_engine/douyin_im/service.py
- `__init__` 新增 `send_fallback` 参数。
- `_send_text` 失败分支:`last_error``DECISION=KICK`/`STATUS_CODE=7911`/`INVALID_REQUEST` 且存在 `send_fallback` 时调用兜底;成功则重置 `_session_invalid_strikes/_fired` 并返回 True;失败继续走 `_note_session_invalid` 自动下线。
### 方法签名验证(全部通过,py_compile OK
- `resolve_conversation_meta(auth, conv_id, my_uid, peer_uid) → (conv_id, short_id, ticket)`
- `ProtoBuilder.build_send_message_request(auth, conv_id, short_id, ticket, msg_content, message_type=7)` 静态方法;`build_msg_payload(spec) → (msg_content, message_type)` 展开正确
- `parse_reply_content(raw) → dict`(含 `type` 键,`"image"` 时兜底拒绝)
- `analyze_send_response(bytes) → dict`ok/decision/status_code/summary/server_message_id
- `DouyinAuth.from_im_session` / `ImSession.can_direct_im/cookies/my_uid/user_agent` / `DouyinImHttpClient(session, account_id)` async with 均匹配
### 待办/注意
- 后端当前未运行;下次启动自动生效(无需重启)。
- 浏览器兜底暂不支持图片回复(返回"请改用纯文字")。
- 运行时验证未执行:需要账号重新托管后实际触发发送观察;KICK 后兜底页面需等待 security-sdk 加载(15s 轮询)。
## 登录态常驻方案(A 保活 + B 自动重登录)✅ 已实施并通过编译/导入验证
背景:sessionid/passport 无 refresh token 可自动换新;IM 通道活跃**不刷新** passport 登录态,托管约 30 天失效 → 抖音返回「用户未登录」。
### A 保活(backend/rpa_engine/playwright_worker.py
- 配置:`KEFU_KEEPALIVE_INTERVAL`(默认 21600s=6h,最小 300s)、`KEFU_KEEPALIVE_DISABLED`(设为 1 关闭)。
- `_keepalive_loop`:每间隔调用 `_keepalive_touch`
- `_keepalive_touch``controller.browser_slot(account_id,"keepalive")` 串行 → 非 headless 最小化 Chromium → storage_state + `self._user_agent` 建 context → **临时挂载 `(self.playwright/browser/context/page)`** 复用 `_has_visible_login_prompt/_persist_cookies` → goto douyin.com → sleep 6s → 检测登录提示(有则返回失败"将触发自动重登录")→ `_persist_cookies()` → finally 恢复引用并关浏览器。
- 挂载点:`_run_im_direct_service` run() 前 `_start_keepalive()`finally 后 `_stop_keepalive()``stop()` 开头也 `_stop_keepalive()`(防悬挂)。
### B 失效自动重登录(backend/main.py + worker 侧)
- worker`__init__` 新增 `relogin_hook: Optional[Callable[[int], Awaitable[None]]]``on_im_session_invalid` 末尾 await hookmanager 层驱动,避免 worker 自 cancel 扭曲 + browser_slot 竞争)。
- manager`_AUTO_RELOGIN_COOLDOWN`(默认 1800s=30min 防抖,env `KEFU_AUTO_RELOGIN_COOLDOWN`);`_schedule_auto_relogin` 快速返回(防抖→查重→create_task);`_auto_relogin_account`:轮询等旧 worker 退出(100×0.2s)→ DB 置 `logging_in`、清 qr/error → `start_worker(login_mode="browser", wait_until_ready=False)` → 浏览器弹码 → 前端账号卡片轮询展示 qr_code_base64 → 扫码自动恢复托管;异常置 offline+error_message。
- 前端无需改(Accounts.vue 已支持 logging_in + 二维码)。
### 验证记录
- `py_compile playwright_worker.py main.py` → COMPILE_OK。
- AST 检查修正版(须同时匹配 `ast.AsyncFunctionDef`!)确认:DouyinWorker 含 `_keepalive_interval(sync)/_keepalive_disabled(sync)/_start_keepalive/_stop_keepalive/_keepalive_loop/_keepalive_touch/on_im_session_invalid`(均 async);WorkerManager 含 `_schedule_auto_relogin/_auto_relogin_account`(均 async)。
- venv 导入冒烟(`.venv/Scripts/python.exe`)→ IMPORT_SMOKE_OK`DouyinWorker.__init__` 参数含 `relogin_hook`
### 运行时验证(待后端启动后)
1. 托管后 6h 内观察保活日志(打开 douyin.com + 持久化 cookie)。
2. 手动清 passport cookie 或等失效 → 观察自动重登录链路:offline→logging_in→弹码→扫码→恢复托管;确认 30min 防抖生效(扫码失败不会死循环)。
3. 唯一绕不开的人工动作是扫码(抖音无免扫码续期通道)。
## 二维码截图太小/无法扫描 ✅ 已修复
### 问题
自动重登录时前端弹窗显示的是**整页截图兜底**,二维码被包在整个网页截图中,尺寸过小,手机无法扫描。
### 诊断
- 后端 `_grab_qr_data_url()` 的精确选择器未命中当前 Douyin 二维码元素,导致一直回退到 `page.screenshot()`
- 使用 Playwright 诊断脚本 `backend/debug_qr_inspect.py` 观察:headless 环境触发的是验证码 iframe(`lf-rc1.yhgfb-cn-static.com/.../rmc-nocaptcha`),与真实有头浏览器弹出的扫码登录弹窗不同,因此重点改为增强二维码抓取鲁棒性。
### 修改(backend/rpa_engine/playwright_worker.py
1. 新增 `from PIL import Image``import io`
2. 扩充 `_QR_SELECTORS`:新增登录弹窗内 `img`/`canvas` 选择器(`#login-pannel``*login-guide*``*login-panel*``*account_login*` 等)。
3. 扩充 `_QR_CONTAINER_SELECTORS` / `_LOGIN_PANEL_SELECTORS`
4. `_grab_qr_in_frames`:匹配元素后增加 `_looks_like_qr_box()` 校验(80~600px、长宽比≥0.75);对 `<canvas>` 优先用 `canvas.toDataURL()` 提取;对 `<img>` 优先读 data-src/http-src;返回前统一经 `_upscale_qr_image()` 放大。
5. 新增 `_grab_qr_generic_in_frames()`:在所有 frame 中泛化扫描登录容器内最大方型 `img`/`canvas`,作为精确选择器未命中时的兜底。
6. `_grab_login_panel_shot()`:改为遍历所有 frame(含 iframe),并调用 `_ensure_panel_fits()` 临时放大 viewport 保证截图清晰。
7. 新增 `_crop_center_viewport_shot()`:截取视口中央 700x800 区域,替代直接整页截图,避免二维码过小。
8. 新增 `_upscale_qr_image()`:对小于 280px 的二维码用 Pillow 最近邻放大,提高手机扫描成功率。
9. `_grab_qr_data_url()` 增加分阶段日志;把「中心区域截图」放在「整页截图」之前;整页截图加 15s timeout 防字体加载卡死。
10. `_check_and_grab_captcha()` 的整页兜底改为先 `_crop_center_viewport_shot()`
### 验证
- `py_compile rpa_engine/playwright_worker.py` → COMPILE_OK。
- venv 导入冒烟 → IMPORT_OK;新增方法列表:`_looks_like_qr_box``_grab_qr_generic_in_frames``_ensure_panel_fits``_crop_center_viewport_shot``_upscale_qr_image`
### 待验证
- 重启后端后触发自动重登录/首次托管,观察前端二维码是否清晰可扫;查看日志应出现 `Captured QR via precise selector` / `generic scan` / `login panel screenshot` 等字样,而不是 `fell back to full-viewport screenshot`
## 排查:KICK 是否由代码主动退出登录触发 ✅ 结论:否
### 用户疑问
收到 `decision=KICK` 错误,怀疑代码里有主动退出登录的逻辑。
### 排查结果
全库搜索 `logout / 退出登录 / clear_cookie / sessionid.*None / passport.*logout` 等,**没有发现任何主动调用抖音退出登录接口或自动清空 session cookie 的代码**。
- `clear_cookie_file()` 仅在 3 个**手动 API** 中被调用:
- `DELETE /api/accounts/{id}`(删除账号)
- `POST /api/accounts/{id}/clear-cookie`(手动清除 Cookie
- `_reset_account_credentials`(重置账号凭证接口)
- `on_im_session_invalid()` 只停止 worker、更新 DB 状态为 offline、触发 `relogin_hook` 自动重登录,**不会删除 cookie / session**。
- `_keepalive_touch()` 仅访问 `https://www.douyin.com/` 并持久化 cookie,不会登出。
`decision=KICK` 是**抖音服务端安全网关返回的**,常见原因:
1. passport/sessionid 自然过期;
2. 账号在其它设备/浏览器登录,挤掉当前会话;
3. 设备指纹/签名不一致触发风控;
4. 服务端主动下线。
### 同步更新提示文案
发现 `http_client.py``service.py` 中 KICK 提示仍写着"请停止托管后用浏览器模式重新登录...",与已实施的自动重登录方案矛盾。已修改为"系统正在自动重登录,请留意账号卡片上的登录二维码并扫码"。
### 修改文件
- `backend/rpa_engine/douyin_im/http_client.py`
- `backend/rpa_engine/douyin_im/service.py`
### 验证
- `py_compile` 两个文件 → COMPILE_OK。
## UA 全链路一致性修复(接收链路)✅ 已实施并通过编译/导入验证
### 需求
用户明确要求:接收消息与发送消息使用同一 User-Agent,账号配置里改了 UA,发送、接收都要同步生效。
### 背景
- 发送链路上一轮已一致:`_build_im_session_from_storage` 保留采集 UA → `session.user_agent``DouyinAuth.from_im_session(session)``auth.user_agent`
- 接收链路存在 2 个硬编码漏网点(Chrome/120 DEFAULT_USER_AGENT+ 多处无参 `DouyinAuth()` 构造导致 `self.user_agent` 未设置。
### 修改
1. **`douyin_im/auth.py`**
- `__init__` 增加 `self.user_agent = None`
- `perepare_auth` 增加 `user_agent: str = ""` 参数,非空时保存 `self.user_agent`(避免覆盖 from_im_session 已设值)。
- `query_my_uid()` 的请求头与 `generate_a_bogus` 改用 `ua = self.user_agent or DEFAULT_USER_AGENT`(消除硬编码)。
- `from_im_session` 在 perepare_auth 时直接传 `user_agent=session.user_agent or DEFAULT_USER_AGENT`
2. **`douyin_im/dy_util.py`**`generate_webid(auth=None, url="", user_agent="")` 新增参数;内部 UA 优先级:显式参数 > `auth.user_agent` > DEFAULT(消除硬编码)。
3. **调用点全部显式传 UA**`session.user_agent or DEFAULT_USER_AGENT`):
- `frontier.py fetch_device_id``follower_poll.py``main.py:2911`conversations 兜底)、`service.py send_message`uid 兜底)
- `peer_profile.py _build_auth``image_upload.py``account_profile.py _build_auth`(传 `ua` 变量)
### 保留原样(非风险点)
- `image_upload.py:496/599``DEFAULT_USER_AGENT`:VOD 存储上传(腾讯云 VODAWS SigV4/JWT 独立鉴权,非抖音 web API、无 a_bogus),不会触发 7911。
- `proto_builder.py _ua_headers``http_client.py:740`:已走 `auth.user_agent`,自动生效。
- `device_profiles.py` DEFAULT_USER_AGENT:仅作为无显式配置时的默认 profile。
### 验证
- `py_compile` 9 个文件 → PY_COMPILE_OK。
- venv 导入冒烟:perepare_auth 签名含 user_agent、设置/不设置路径正确、generate_webid 签名与缓存命中正确 → IMPORT_SMOKE_OK。
- Grep 确认接收链路(douyin_im)无 `user_agent=DEFAULT_USER_AGENT` / `"User-Agent": DEFAULT_USER_AGENT` 残留。
### 运行时验证(待重启后端)
配置账号 UAaccount.user_agent)→ 重启后端 → 观察日志确认 a_bogus 签名与请求头 UA 一致(Chrome/148 或其他配置值,而非 Chrome/120)。
## 凭证登录头(user_agent)自动回填账号 ✅ 已实施并通过编译/冒烟测试
### 需求
用户问:「获取登录凭证里有没有登录头,如果有的话保存账户的时候直接填上去」。确认采集器导出的 storage_state 顶层带 `user_agent`(如 Chrome/148),保存 Cookie 时之前只存进 `cookie_data``account.user_agent` 列不回填。
### 修改
1. **`backend/utils/cookie_store.py`**:新增 `extract_user_agent_from_cookie_data(cookie_data) -> str`——解析 JSON 取顶层 `user_agent`(兼容 DYCRED 的 `ua` 键),无效/缺失返回空串。
2. **`backend/main.py`**
- 新增 `_backfill_user_agent_from_cookie(account, standard_json_str)`:仅当 `account.user_agent` 为空且凭证含 UA 时回填;**已有自定义 UA 保持不变**(用户配置优先)。
- `create_account`POST /api/accounts)与 `update_account_cookie`PUT /api/accounts/{id}/cookie)保存 Cookie 后调用回填。
- `AccountCookieResponse` 增加 `user_agent` / `user_agent_label` 字段,`_build_cookie_response` 填充,前端保存凭证后可确认回填结果。
### 验证
- `py_compile main.py utils/cookie_store.py` → PY_COMPILE_OK。
- 冒烟测试:storage_state(顶层 user_agent)→ validate_cookie_json 保留 → 提取 Chrome/148 → 空列回填成功;已有自定义 UA 不被覆盖;DYCRED 输入(ua 键)也能提取 → ALL_SMOKE_OK。
### 行为说明
- 保存/更新凭证时自动回填,发送+接收链路经 `_build_account_im_session` 统一走该 UA。
- 若用户在配置页显式改过 UA,凭证里的登录头不会覆盖它。
- 清空凭证(DELETE cookie)不会清 `account.user_agent` 列(保留设备指纹配置,可在配置页手动改)。
### 待验证
重启后端 → 保存一份带 user_agent 的凭证 → 观察账号响应中 `user_agent` 已回填(非空)且 `user_agent_label` 正确。
## 咨询:能否用抖音 APP 凭证登录 → 结论:不能,已改为优化保活 ✅
### 用户提问
能否搞到抖音 APP 登录凭证(APP sessionid)用来登录/托管。
### 结论(已给用户,含对比图)
- APP sessionid 与 Web sessionid 分属不同端,当前架构(frontier WS aid=2906 + web API + a_bogus)只认 Web 登录态 + security-sdk 密钥,APP 凭证直接喂入 → 设备指纹/签名不匹配 → 风控/限号/封号。
- 获取 APP 凭证本身要 root + 抓包 + fridassl pinning 反抓包),且绑定设备指纹,跨环境使用高风控;等于重写整套 APP 协议层,不推荐。
- 用户动机 = 摆脱频繁扫码/登录失效 → 方向改为优化 Web 保活。
### 保活增强(backend/rpa_engine/playwright_worker.py
1. `_keepalive_touch` 访问目标从首页改为 `https://www.douyin.com/im`(私信页,更贴近真实活跃、触发 IM 域请求),可用 `KEFU_KEEPALIVE_URL` 覆盖;/im 异常时回退首页。
2. 随机节奏:停留 4~8s + 一次 `mouse.wheel` 滚动,避免固定机械行为。
3. 新增 `_cookie_expires_map` / `_fmt_expires_map` 静态方法:保活前后读取 sid_guard/sessionid/sid_tt 等 passport cookie 的 expires 并打日志(before/after/renewed),用于实测抖音 Web 是否对持续活跃账号滑动续期。
### 验证
- py_compile → PY_COMPILE_OK。
- AST + venv 冒烟:辅助方法存在且为 staticexpires 提取(含 session cookie -1→0、空值跳过、非 passport 键过滤)、格式化、renewed 判定逻辑全部通过 → KEEPALIVE_SMOKE_OK。
### 待观察(重启后端后)
- 保活日志出现 `keepalive passport expires before[...] after[...] renewed=...`:若 renewed 有值 → 保活确实续期,可继续调优频率;若持续 renewed=none → 抖音 Web passport 不滑动续期,30 天到期仍需扫码一次(自动重登录已保证只扫一次)。
## KICK 循环修复(发送被踢下线 → 重登 → 又被踢)
### 根因(实锤)
- `DouyinImSession.from_storage_state` 把新版 `__tea_cache_tokens_6383.user_unique_id`(实际是 web_id,如 7678646545793812008)误当账号 UID → my_uid=web_id,而 device_id 取 `web_runtime_security_uid`(真实 UID)→ **device_id != my_uid** → 安全网关 decision=KICK → 自动重登 → 新旧登录态混合 → 循环。
- 重登后 cookie 落库仍混合(normalize 只修顶层 my_uid 与 web_runtime_security_uid,不清理 localStorage 混合 tea)。
- worker 的 `profile_matches_cookie` 要求 profile_updated_at >= cookie_updated_at:资料同步滞后时错误 UID 带病运行。
### 修复(4 个文件)
1. `session.py from_storage_state`:改两遍扫描收集(ls_sec_uid/ls_web_id/ls_tea_pairs),my_uid 优先级:extra/顶层 > web_runtime_security_uid > tea(user_unique_id!=web_id 才可信)device_idextra/cookies > ls_sec_uid > my_uid;末尾加一致性收敛(my_uid 与 device_id 不一致时以 my_uid 为准)。tea 解析对非 dict 值(如 int 1)类型保护。
2. `cookie_store.py _parse_tea_from_ls``if not isinstance(parsed, dict): continue`,修复 `__tea_cache_first_*` 值为 1 时的 AttributeError 崩溃(保存凭证 500)。
3. `playwright_worker.py _build_im_session_from_storage`:权威 UID 覆盖改为 `accounts.douyin_uid` 无条件优先(拿 cookie 拉的,最可靠),`profile.uid` 仅在资料不早于 cookie 更新时可信(防串号,账号 9 的 profile 表就存了账号 8 的 UID);覆盖 my_uid 时同步 device_id=verified_uid。
4. `credential.py build_im_session_from_storage`saved.uid_verified 覆盖 my_uid 时同步 device_id(之前只改 my_uid 不改 device_id,凭证残留设备号导致不一致)。
### 数据修复
- 备份:backend/.bak/kefu*.db.before_uid_fix_20260827_*
- 账号 1im_session_data.my_uid 7678646545793812008(web_id) → 2609567359568155uid_verified=True
- 账号 8/9:仅补 uid_verified=True
- 验证:3 账号均 device_id==my_uid==douyin_uidALL_CONSISTENT
### 测试
- test_sec_user_id_guard + test_im_receive_path 共 52 个用例全过(更新 3 处断言:device_id 同步为新行为;3 处 mock 从 _load_user_agent 改为 _load_raw_user_agent)。
- 全套 228 个用例剩 6 个失败为**预先存在的测试 mock 签名过期**get_owned_account 新增 write_permission 参数,测试 mock 未更新),与本次改动无关。
### 待观察(重启后端后)
- 账号 1 再发送是否还 KICK(预期不再循环);日志出现 "replaced collected IM uid ..." / "synced device_id" 即覆盖生效。
- 账号 9 的 localStorage 仍混合(www 域残留 7657072144060941834),等重新登录后由 normalize + 新解析逻辑自然清洗。
+21
View File
@@ -0,0 +1,21 @@
# 2026-08-28 工作日志
## 抖音 IM「decision=KICK」排查(服务器 116.62.23.103
**结论**:不是抖音改了 IM 规则/签名算法,是账号被安全网关风控踢下线。签名链路正常(同服务器另一账号可正常发送、读接口正常返回、凭证完整)。
**关键证据**(服务器 MySQL `kefu` 库 + `/www/wwwlogs/python/douyin/error.log`):
- 活跃账号:id=11「随安尔乐」my_uid=7670159096859706425、id=12「抖音账号_12」my_uid=7670157997767050299,均为 Chrome/148 UAkeys/web_protect 凭证完整(len 533/459)。
- 两个账号反复 KICK,且都在回复同一测试号「Huhao」(peer_uid=66578464308),回复内容只是 "ss"/"jjj" 测试文本(非导流话术)。
- 时序:KICK → create_conversation INVALID_REQUEST → 「用户未登录」→ 自动重登录 → 恢复 → 再发 → 再 KICK,形成死循环(约每 10~30 分钟一次)。
- 读接口(get_by_user_init)正常,只有 signed 的发送(message/send)被 KICK → 签名没问题,是账号级风控。
**根因判断**:抖音 2026 风控收紧(内容+设备+IP+行为+账号五维)。触发点最可能是:多账号同服务器 IP + 对同一陌生 peer 的自动回复行为;且 KICK→重登→再发 的紧循环本身会加重风控。
**处理方向**
1. 停掉受影响账号的托管/自动回复,冷却几小时~一天。
2. 浏览器模式手动重登,先用互关好友或「对方先发」的真实用户测发送,别再用小号对冷门 peer 反复自动回。
3. 确认账号没在手机端同时登录(并发登录会吊销 web ticket)。
4. 后续可选代码改进:KICK 后对同一会话加冷却(重登后暂不重发),打断紧循环。
**环境备注**:后端用 MySQL(`KEFU_DB_TYPE=mysql`)`kefu.db`/`kefu1.db` 是遗留 SQLite(kefu.db 已损坏,非活跃库,可忽略)。
Binary file not shown.
Binary file not shown.
+82 -7
View File
@@ -6,7 +6,26 @@ from sqlalchemy.ext.asyncio import AsyncSession
from models.database import get_db
from models.models import User
from .jwt_utils import decode_access_token
from .roles import can_manage_users, can_write, is_admin
from .permissions import (
ACCOUNTS_COOKIE,
ACCOUNTS_CREATE,
ACCOUNTS_DELETE,
ACCOUNTS_START,
ACCOUNTS_STOP,
ACCOUNTS_UPDATE,
ACCOUNTS_WRITE,
ACCOUNTS_WRITE_GRANULAR,
LINK_CARDS_WRITE,
MESSAGES_WRITE,
ORDERS_CREATE,
ROLES_MANAGE,
RULES_WRITE,
SETTINGS_DATABASE,
SYSTEM_LOGS_CLEAR,
USERS_MANAGE,
WRITE_PERMISSIONS,
)
from .roles import can_write, has_permission, is_admin
bearer_scheme = HTTPBearer(auto_error=False)
@@ -32,6 +51,20 @@ async def get_current_user(
return user
def require_permission(permission: str):
"""FastAPI dependency factory that checks a single permission code."""
async def _checker(user: User = Depends(get_current_user)) -> User:
if has_permission(user.role, permission):
return user
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"缺少权限:{permission}",
)
return _checker
async def require_admin(user: User = Depends(get_current_user)) -> User:
if not is_admin(user.role):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="需要管理员权限")
@@ -39,12 +72,54 @@ async def require_admin(user: User = Depends(get_current_user)) -> User:
async def require_write(user: User = Depends(get_current_user)) -> User:
if not can_write(user.role):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="当前角色只读,无法执行此操作")
return user
"""Any write-capable permission (accounts/messages/rules) or legacy can_write."""
if is_admin(user.role) or can_write(user.role):
return user
if any(has_permission(user.role, code) for code in WRITE_PERMISSIONS):
return user
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="当前角色只读,无法执行此操作",
)
def _require_any(*codes: str, detail: str):
async def _checker(user: User = Depends(get_current_user)) -> User:
if any(has_permission(user.role, code) for code in codes):
return user
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=detail)
return _checker
# Backward-compatible: any account write capability.
require_accounts_write = _require_any(
ACCOUNTS_WRITE,
*ACCOUNTS_WRITE_GRANULAR,
detail="缺少账号写权限",
)
require_accounts_create = require_permission(ACCOUNTS_CREATE)
require_accounts_update = require_permission(ACCOUNTS_UPDATE)
require_accounts_delete = require_permission(ACCOUNTS_DELETE)
require_accounts_start = require_permission(ACCOUNTS_START)
require_accounts_stop = require_permission(ACCOUNTS_STOP)
require_accounts_cookie = require_permission(ACCOUNTS_COOKIE)
require_messages_write = require_permission(MESSAGES_WRITE)
require_rules_write = require_permission(RULES_WRITE)
require_link_cards_write = require_permission(LINK_CARDS_WRITE)
require_system_logs_clear = require_permission(SYSTEM_LOGS_CLEAR)
require_settings_database = require_permission(SETTINGS_DATABASE)
require_orders_create = require_permission(ORDERS_CREATE)
async def require_user_manager(user: User = Depends(get_current_user)) -> User:
if not can_manage_users(user.role):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="需要管理员权限")
return user
if has_permission(user.role, USERS_MANAGE):
return user
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="需要用户管理权限")
async def require_role_manager(user: User = Depends(get_current_user)) -> User:
if has_permission(user.role, ROLES_MANAGE):
return user
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="需要角色管理权限")
+378
View File
@@ -0,0 +1,378 @@
"""Fixed permission catalog: menus, button actions, and data scope.
UI only selects from this list; new codes must be added in code.
"""
from __future__ import annotations
from typing import Any
# ---------------------------------------------------------------------------
# Menu visibility
# ---------------------------------------------------------------------------
MENU_DASHBOARD = "menu.dashboard"
MENU_ACCOUNTS = "menu.accounts"
MENU_MESSAGES = "menu.messages"
MENU_RULES = "menu.rules"
MENU_LOGS = "menu.logs"
MENU_RECEIVED_MESSAGES = "menu.received_messages"
MENU_SYSTEM_LOGS = "menu.system_logs"
MENU_DOWNLOAD = "menu.download"
MENU_HELP = "menu.help"
MENU_USERS = "menu.users"
MENU_ROLES = "menu.roles"
MENU_SETTINGS = "menu.settings"
MENU_DESKTOP_UPDATE = "menu.desktop_update"
MENU_PAYMENT_SETTINGS = "menu.payment_settings"
MENU_PAYMENT_ORDERS = "menu.payment_orders"
# ---------------------------------------------------------------------------
# Button / action permissions
# ---------------------------------------------------------------------------
# Accounts (granular). Legacy ``accounts.write`` expands to the full set.
ACCOUNTS_CREATE = "accounts.create"
ACCOUNTS_UPDATE = "accounts.update"
ACCOUNTS_DELETE = "accounts.delete"
ACCOUNTS_START = "accounts.start"
ACCOUNTS_STOP = "accounts.stop"
ACCOUNTS_COOKIE = "accounts.cookie"
ACCOUNTS_WRITE = "accounts.write" # legacy bundle
MESSAGES_WRITE = "messages.write"
RULES_WRITE = "rules.write"
LINK_CARDS_WRITE = "link_cards.write"
LOGS_READ = "logs.read"
RECEIVED_MESSAGES_READ = "received_messages.read"
SYSTEM_LOGS_READ = "system_logs.read"
SYSTEM_LOGS_CLEAR = "system_logs.clear"
USERS_MANAGE = "users.manage"
ROLES_MANAGE = "roles.manage"
SETTINGS_MANAGE = "settings.manage"
SETTINGS_DATABASE = "settings.database"
DESKTOP_MANAGE = "desktop.manage"
PAYMENTS_MANAGE = "payments.manage"
ORDERS_READ = "orders.read"
ORDERS_CREATE = "orders.create"
# ---------------------------------------------------------------------------
# Data scope
# ---------------------------------------------------------------------------
# Without this (and without is_admin), users only see their own data.
DATA_SCOPE_ALL = "data.scope_all"
# ---------------------------------------------------------------------------
# Bundles / aliases expanded on save & when checking permissions
# ---------------------------------------------------------------------------
ACCOUNTS_WRITE_GRANULAR: tuple[str, ...] = (
ACCOUNTS_CREATE,
ACCOUNTS_UPDATE,
ACCOUNTS_DELETE,
ACCOUNTS_START,
ACCOUNTS_STOP,
ACCOUNTS_COOKIE,
)
LEGACY_BUNDLES: dict[str, tuple[str, ...]] = {
ACCOUNTS_WRITE: ACCOUNTS_WRITE_GRANULAR,
}
ALL_PERMISSIONS: tuple[str, ...] = (
MENU_DASHBOARD,
MENU_ACCOUNTS,
MENU_MESSAGES,
MENU_RULES,
MENU_LOGS,
MENU_RECEIVED_MESSAGES,
MENU_SYSTEM_LOGS,
MENU_DOWNLOAD,
MENU_HELP,
MENU_USERS,
MENU_ROLES,
MENU_SETTINGS,
MENU_DESKTOP_UPDATE,
MENU_PAYMENT_SETTINGS,
MENU_PAYMENT_ORDERS,
ACCOUNTS_CREATE,
ACCOUNTS_UPDATE,
ACCOUNTS_DELETE,
ACCOUNTS_START,
ACCOUNTS_STOP,
ACCOUNTS_COOKIE,
ACCOUNTS_WRITE,
MESSAGES_WRITE,
RULES_WRITE,
LINK_CARDS_WRITE,
LOGS_READ,
RECEIVED_MESSAGES_READ,
SYSTEM_LOGS_READ,
SYSTEM_LOGS_CLEAR,
USERS_MANAGE,
ROLES_MANAGE,
SETTINGS_MANAGE,
SETTINGS_DATABASE,
DESKTOP_MANAGE,
PAYMENTS_MANAGE,
ORDERS_READ,
ORDERS_CREATE,
DATA_SCOPE_ALL,
)
PERMISSION_SET = frozenset(ALL_PERMISSIONS)
WRITE_PERMISSIONS = frozenset(
{
ACCOUNTS_WRITE,
*ACCOUNTS_WRITE_GRANULAR,
MESSAGES_WRITE,
RULES_WRITE,
LINK_CARDS_WRITE,
}
)
_PERMISSION_META: dict[str, dict[str, str]] = {
MENU_DASHBOARD: {"group": "menu", "label": "数据概览"},
MENU_ACCOUNTS: {"group": "menu", "label": "账号管理"},
MENU_MESSAGES: {"group": "menu", "label": "私信收发"},
MENU_RULES: {"group": "menu", "label": "自动回复规则"},
MENU_LOGS: {"group": "menu", "label": "回复日志面板"},
MENU_RECEIVED_MESSAGES: {"group": "menu", "label": "接收消息日志"},
MENU_SYSTEM_LOGS: {"group": "menu", "label": "系统诊断日志"},
MENU_DOWNLOAD: {"group": "menu", "label": "软件下载"},
MENU_HELP: {"group": "menu", "label": "帮助中心"},
MENU_USERS: {"group": "menu", "label": "用户管理"},
MENU_ROLES: {"group": "menu", "label": "角色设定"},
MENU_SETTINGS: {"group": "menu", "label": "系统设置"},
MENU_DESKTOP_UPDATE: {"group": "menu", "label": "桌面端升级"},
MENU_PAYMENT_SETTINGS: {"group": "menu", "label": "支付配置"},
MENU_PAYMENT_ORDERS: {"group": "menu", "label": "我的订单"},
ACCOUNTS_CREATE: {"group": "action", "label": "新增抖音账号"},
ACCOUNTS_UPDATE: {"group": "action", "label": "编辑账号信息"},
ACCOUNTS_DELETE: {"group": "action", "label": "删除账号"},
ACCOUNTS_START: {"group": "action", "label": "启动托管 / 批量启动"},
ACCOUNTS_STOP: {"group": "action", "label": "停止托管"},
ACCOUNTS_COOKIE: {"group": "action", "label": "查看/修改 Cookie 与凭证"},
ACCOUNTS_WRITE: {"group": "action", "label": "账号全部写操作(兼容旧版,等同下列细项)"},
MESSAGES_WRITE: {"group": "action", "label": "发送私信 / 队列立即发送"},
RULES_WRITE: {"group": "action", "label": "编辑自动回复规则"},
LINK_CARDS_WRITE: {"group": "action", "label": "上传素材 / 生成链接卡片"},
LOGS_READ: {"group": "action", "label": "查看回复日志"},
RECEIVED_MESSAGES_READ: {"group": "action", "label": "查看接收消息日志"},
SYSTEM_LOGS_READ: {"group": "action", "label": "查看系统诊断日志"},
SYSTEM_LOGS_CLEAR: {"group": "action", "label": "清空系统诊断日志"},
USERS_MANAGE: {"group": "action", "label": "管理用户"},
ROLES_MANAGE: {"group": "action", "label": "管理角色"},
SETTINGS_MANAGE: {"group": "action", "label": "管理系统设置"},
SETTINGS_DATABASE: {"group": "action", "label": "管理数据库配置与迁移"},
DESKTOP_MANAGE: {"group": "action", "label": "管理桌面端升级"},
PAYMENTS_MANAGE: {"group": "action", "label": "管理支付配置与全部订单"},
ORDERS_READ: {"group": "action", "label": "查看我的订单"},
ORDERS_CREATE: {"group": "action", "label": "购买额度 / 创建订单"},
DATA_SCOPE_ALL: {"group": "data", "label": "查看全部用户数据(全局数据范围)"},
}
OPERATOR_PERMISSIONS: tuple[str, ...] = (
MENU_DASHBOARD,
MENU_ACCOUNTS,
MENU_MESSAGES,
MENU_RULES,
MENU_LOGS,
MENU_RECEIVED_MESSAGES,
MENU_DOWNLOAD,
MENU_HELP,
MENU_PAYMENT_ORDERS,
*ACCOUNTS_WRITE_GRANULAR,
MESSAGES_WRITE,
RULES_WRITE,
LINK_CARDS_WRITE,
LOGS_READ,
RECEIVED_MESSAGES_READ,
ORDERS_READ,
ORDERS_CREATE,
)
VIEWER_PERMISSIONS: tuple[str, ...] = (
MENU_DASHBOARD,
MENU_ACCOUNTS,
MENU_MESSAGES,
MENU_RULES,
MENU_LOGS,
MENU_RECEIVED_MESSAGES,
MENU_DOWNLOAD,
MENU_HELP,
LOGS_READ,
RECEIVED_MESSAGES_READ,
)
def normalize_permissions(codes: list[str] | tuple[str, ...] | None) -> list[str]:
if not codes:
return []
seen: set[str] = set()
result: list[str] = []
for code in codes:
value = str(code or "").strip()
if not value or value not in PERMISSION_SET or value in seen:
continue
seen.add(value)
result.append(value)
return result
def expand_legacy_bundles(codes: set[str]) -> set[str]:
"""Expand legacy bundle codes into granular permissions."""
expanded = set(codes)
for bundle, parts in LEGACY_BUNDLES.items():
if bundle in expanded:
expanded.update(parts)
return expanded
# Menu → required action. Selecting a menu always grants the action.
MENU_REQUIRED_ACTIONS: dict[str, str | tuple[str, ...]] = {
MENU_USERS: USERS_MANAGE,
MENU_ROLES: ROLES_MANAGE,
MENU_SETTINGS: SETTINGS_MANAGE,
MENU_DESKTOP_UPDATE: DESKTOP_MANAGE,
MENU_PAYMENT_SETTINGS: PAYMENTS_MANAGE,
MENU_PAYMENT_ORDERS: (ORDERS_READ, ORDERS_CREATE),
MENU_LOGS: LOGS_READ,
MENU_RECEIVED_MESSAGES: RECEIVED_MESSAGES_READ,
MENU_SYSTEM_LOGS: SYSTEM_LOGS_READ,
MENU_ACCOUNTS: (), # page access only; buttons are separate
MENU_MESSAGES: (),
MENU_RULES: (),
}
# Action → primary menu only.
ACTION_PRIMARY_MENU: dict[str, str] = {
USERS_MANAGE: MENU_USERS,
ROLES_MANAGE: MENU_ROLES,
SETTINGS_MANAGE: MENU_SETTINGS,
SETTINGS_DATABASE: MENU_SETTINGS,
DESKTOP_MANAGE: MENU_DESKTOP_UPDATE,
PAYMENTS_MANAGE: MENU_PAYMENT_SETTINGS,
ORDERS_READ: MENU_PAYMENT_ORDERS,
ORDERS_CREATE: MENU_PAYMENT_ORDERS,
LOGS_READ: MENU_LOGS,
RECEIVED_MESSAGES_READ: MENU_RECEIVED_MESSAGES,
SYSTEM_LOGS_READ: MENU_SYSTEM_LOGS,
SYSTEM_LOGS_CLEAR: MENU_SYSTEM_LOGS,
ACCOUNTS_CREATE: MENU_ACCOUNTS,
ACCOUNTS_UPDATE: MENU_ACCOUNTS,
ACCOUNTS_DELETE: MENU_ACCOUNTS,
ACCOUNTS_START: MENU_ACCOUNTS,
ACCOUNTS_STOP: MENU_ACCOUNTS,
ACCOUNTS_COOKIE: MENU_ACCOUNTS,
ACCOUNTS_WRITE: MENU_ACCOUNTS,
MESSAGES_WRITE: MENU_MESSAGES,
RULES_WRITE: MENU_RULES,
LINK_CARDS_WRITE: MENU_RULES,
}
def _iter_required_actions(menu: str) -> tuple[str, ...]:
raw = MENU_REQUIRED_ACTIONS.get(menu)
if raw is None:
return ()
if isinstance(raw, str):
return (raw,)
return tuple(raw)
# Catalog / UI pairs (menu → first required action for checkbox hints).
MENU_ACTION_PAIRS: tuple[tuple[str, str], ...] = tuple(
(menu, actions[0])
for menu, actions in (
(m, _iter_required_actions(m)) for m in MENU_REQUIRED_ACTIONS
)
if actions
)
def expand_paired_permissions(codes: list[str] | tuple[str, ...] | None) -> list[str]:
"""Normalize, expand legacy bundles, and auto-complete menu/action pairs."""
selected = expand_legacy_bundles(set(normalize_permissions(codes)))
for menu in list(selected):
for action in _iter_required_actions(menu):
selected.add(action)
for action, menu in ACTION_PRIMARY_MENU.items():
if action in selected:
selected.add(menu)
# If every granular account write is present, keep the legacy bundle flag.
if all(code in selected for code in ACCOUNTS_WRITE_GRANULAR):
selected.add(ACCOUNTS_WRITE)
return [code for code in ALL_PERMISSIONS if code in selected]
def permission_implies(held: set[str], needed: str) -> bool:
"""Whether a held permission set satisfies ``needed`` (incl. legacy bundles)."""
if needed in held:
return True
for bundle, parts in LEGACY_BUNDLES.items():
if needed in parts and bundle in held:
return True
return False
def permission_catalog() -> dict[str, Any]:
menus = []
actions = []
data = []
meta_by_code: dict[str, dict[str, str]] = {}
for code in ALL_PERMISSIONS:
meta = _PERMISSION_META[code]
item = {"code": code, "label": meta["label"]}
meta_by_code[code] = meta
group = meta["group"]
if group == "menu":
menus.append(item)
elif group == "data":
data.append(item)
else:
actions.append(item)
# Reverse index: menu → child action codes (preserve ALL_PERMISSIONS order).
children_by_menu: dict[str, list[dict[str, str]]] = {m["code"]: [] for m in menus}
for code in ALL_PERMISSIONS:
if meta_by_code[code]["group"] != "action":
continue
parent = ACTION_PRIMARY_MENU.get(code)
if parent and parent in children_by_menu:
children_by_menu[parent].append(
{"code": code, "label": meta_by_code[code]["label"]}
)
tree: list[dict[str, Any]] = []
for menu in menus:
node: dict[str, Any] = {
"code": menu["code"],
"label": menu["label"],
"kind": "menu",
"children": children_by_menu.get(menu["code"], []),
}
tree.append(node)
if data:
tree.append(
{
"code": "__group.data__",
"label": "数据权限",
"kind": "group",
"children": [
{"code": item["code"], "label": item["label"]} for item in data
],
}
)
return {
"menus": menus,
"actions": actions,
"data": data,
"tree": tree,
"pairs": [
{"menu": menu, "action": action} for menu, action in MENU_ACTION_PAIRS
],
}
+306
View File
@@ -0,0 +1,306 @@
"""Persist and cache roles; seed built-ins on startup."""
from __future__ import annotations
import json
import logging
import re
from datetime import datetime
from fastapi import HTTPException, status
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from models.models import Role, User
from .permissions import ALL_PERMISSIONS, expand_paired_permissions, normalize_permissions
from .roles import (
ROLE_ADMIN,
RoleRecord,
default_role_seeds,
ensure_role,
get_cached_role,
is_admin,
list_cached_roles,
role_label,
sanitize_role_permissions,
set_role_cache,
)
logger = logging.getLogger("auth.roles")
_ROLE_CODE_RE = re.compile(r"^[a-z][a-z0-9_]{1,49}$")
def _encode_permissions(codes: list[str]) -> str:
return json.dumps(codes, ensure_ascii=False)
def _decode_permissions(raw: str | None) -> list[str]:
if not raw:
return []
try:
data = json.loads(raw)
except Exception:
return []
if not isinstance(data, list):
return []
return normalize_permissions([str(item) for item in data])
def role_to_record(row: Role) -> RoleRecord:
perms = (
list(ALL_PERMISSIONS)
if row.is_admin
else expand_paired_permissions(_decode_permissions(row.permissions))
)
return RoleRecord(
code=row.code,
label=row.label,
description=row.description or "",
is_system=bool(row.is_system),
is_admin=bool(row.is_admin),
permissions=perms,
)
async def refresh_role_cache(db: AsyncSession) -> list[RoleRecord]:
result = await db.execute(select(Role).order_by(Role.id.asc()))
rows = result.scalars().all()
records = [role_to_record(row) for row in rows]
if not records:
records = default_role_seeds()
set_role_cache(records)
return records
async def seed_builtin_roles(db: AsyncSession) -> None:
"""Insert missing built-in roles and keep system role permissions in sync."""
seeds = {seed.code: seed for seed in default_role_seeds()}
result = await db.execute(select(Role))
existing = {row.code: row for row in result.scalars().all()}
changed = False
for code, seed in seeds.items():
row = existing.get(code)
payload = _encode_permissions(seed.permissions)
if row is None:
db.add(
Role(
code=seed.code,
label=seed.label,
description=seed.description,
is_system=True,
is_admin=seed.is_admin,
permissions=payload,
)
)
changed = True
continue
# Keep system flags and built-in permission sets in sync with code.
if not row.is_system:
row.is_system = True
changed = True
if seed.is_admin:
if not row.is_admin or row.permissions != payload or row.label != seed.label:
row.is_admin = True
row.permissions = payload
row.label = seed.label
if seed.description and row.description != seed.description:
row.description = seed.description
changed = True
else:
# operator / viewer: resync catalog so new menu/action codes ship.
desired = _encode_permissions(expand_paired_permissions(seed.permissions))
if row.permissions != desired or row.label != seed.label:
row.permissions = desired
row.label = seed.label
row.is_admin = False
changed = True
# Repair custom roles that have unpaired menu/action selections.
for row in existing.values():
if row.code in seeds or row.is_admin:
continue
repaired = expand_paired_permissions(_decode_permissions(row.permissions))
encoded = _encode_permissions(repaired)
if row.permissions != encoded:
row.permissions = encoded
changed = True
if changed:
await db.commit()
await refresh_role_cache(db)
logger.info("Role cache loaded: %s", ", ".join(r.code for r in list_cached_roles()))
async def list_roles(db: AsyncSession) -> list[RoleRecord]:
await refresh_role_cache(db)
return list_cached_roles()
async def get_role_or_404(db: AsyncSession, code: str) -> Role:
result = await db.execute(select(Role).where(Role.code == code))
row = result.scalar_one_or_none()
if not row:
raise HTTPException(status_code=404, detail="角色不存在")
return row
async def count_users_with_role(db: AsyncSession, code: str) -> int:
result = await db.execute(
select(func.count()).select_from(User).where(User.role == code)
)
return int(result.scalar() or 0)
async def count_admin_users(db: AsyncSession) -> int:
result = await db.execute(
select(func.count()).select_from(User).where(User.role == ROLE_ADMIN)
)
return int(result.scalar() or 0)
def validate_role_code(code: str) -> str:
value = str(code or "").strip().lower()
if not _ROLE_CODE_RE.match(value):
raise HTTPException(
status_code=400,
detail="角色码需为小写字母开头,仅含小写字母/数字/下划线,长度 2-50",
)
return value
async def create_role(
db: AsyncSession,
*,
code: str,
label: str,
description: str | None,
permissions: list[str] | None,
) -> RoleRecord:
role_code = validate_role_code(code)
if get_cached_role(role_code) or (
await db.execute(select(Role).where(Role.code == role_code))
).scalar_one_or_none():
raise HTTPException(status_code=400, detail="角色码已存在")
name = (label or "").strip() or role_code
perms = sanitize_role_permissions(permissions, force_all=False)
row = Role(
code=role_code,
label=name,
description=(description or "").strip() or None,
is_system=False,
is_admin=False,
permissions=_encode_permissions(perms),
)
db.add(row)
await db.commit()
await db.refresh(row)
await refresh_role_cache(db)
return role_to_record(row)
async def update_role(
db: AsyncSession,
code: str,
*,
label: str | None = None,
description: str | None = None,
permissions: list[str] | None = None,
) -> RoleRecord:
row = await get_role_or_404(db, code)
if row.is_admin or row.code == ROLE_ADMIN:
# Admin role always keeps full permissions; label/description may update.
if label is not None:
row.label = (label or "").strip() or row.label
if description is not None:
row.description = (description or "").strip() or None
row.permissions = _encode_permissions(list(ALL_PERMISSIONS))
row.is_admin = True
row.is_system = True
row.updated_at = datetime.utcnow()
await db.commit()
await db.refresh(row)
await refresh_role_cache(db)
return role_to_record(row)
if label is not None:
row.label = (label or "").strip() or row.label
if description is not None:
row.description = (description or "").strip() or None
if permissions is not None:
row.permissions = _encode_permissions(
sanitize_role_permissions(permissions, force_all=False)
)
row.is_admin = False
row.updated_at = datetime.utcnow()
await db.commit()
await db.refresh(row)
await refresh_role_cache(db)
return role_to_record(row)
async def delete_role(db: AsyncSession, code: str) -> None:
row = await get_role_or_404(db, code)
if row.is_system or row.is_admin or row.code == ROLE_ADMIN:
raise HTTPException(status_code=400, detail="系统内置角色不可删除")
used = await count_users_with_role(db, code)
if used > 0:
raise HTTPException(
status_code=400,
detail=f"仍有 {used} 个用户使用该角色,请先调整用户角色后再删除",
)
await db.delete(row)
await db.commit()
await refresh_role_cache(db)
async def ensure_role_assignable(db: AsyncSession, role_code: str) -> str:
"""Validate role exists in DB (refresh cache if needed)."""
code = str(role_code or "").strip()
if not code:
raise HTTPException(status_code=400, detail="角色不能为空")
if get_cached_role(code) is None:
await refresh_role_cache(db)
try:
return ensure_role(code)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
async def guard_last_admin_change(
db: AsyncSession,
*,
user: User,
new_role: str | None = None,
deactivating: bool = False,
deleting: bool = False,
) -> None:
"""Prevent removing the last admin user."""
if not is_admin(user.role):
return
admin_count = await count_admin_users(db)
if admin_count > 1:
return
if deleting or deactivating:
raise HTTPException(
status_code=400,
detail="不能删除或禁用最后一个管理员账号",
)
if new_role is not None and not is_admin(new_role):
raise HTTPException(
status_code=400,
detail="不能将最后一个管理员改为非管理员角色",
)
def user_permission_payload(role_code: str) -> dict:
record = get_cached_role(role_code)
admin = bool(record.is_admin) if record else is_admin(role_code)
from .roles import permissions_for_role
return {
"role_label": role_label(role_code),
"is_admin": admin,
"permissions": permissions_for_role(role_code),
}
+164 -9
View File
@@ -1,5 +1,21 @@
"""Role code helpers and an in-memory role registry backed by the roles table."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Iterable
from .permissions import (
ALL_PERMISSIONS,
DATA_SCOPE_ALL,
OPERATOR_PERMISSIONS,
VIEWER_PERMISSIONS,
WRITE_PERMISSIONS,
expand_legacy_bundles,
permission_implies,
)
ROLE_ADMIN = "admin"
ROLE_OPERATOR = "operator"
ROLE_VIEWER = "viewer"
@@ -13,19 +29,158 @@ ROLE_LABELS = {
}
def is_admin(role: str) -> bool:
return role == ROLE_ADMIN
@dataclass
class RoleRecord:
code: str
label: str
description: str = ""
is_system: bool = False
is_admin: bool = False
permissions: list[str] = field(default_factory=list)
def can_write(role: str) -> bool:
return role in (ROLE_ADMIN, ROLE_OPERATOR)
_ROLE_CACHE: dict[str, RoleRecord] = {}
def can_manage_users(role: str) -> bool:
return role == ROLE_ADMIN
def default_role_seeds() -> list[RoleRecord]:
return [
RoleRecord(
code=ROLE_ADMIN,
label=ROLE_LABELS[ROLE_ADMIN],
description="拥有全部菜单、按钮与全局数据权限",
is_system=True,
is_admin=True,
permissions=list(ALL_PERMISSIONS),
),
RoleRecord(
code=ROLE_OPERATOR,
label=ROLE_LABELS[ROLE_OPERATOR],
description="管理自己的账号、规则与私信(仅本人数据)",
is_system=True,
is_admin=False,
permissions=list(OPERATOR_PERMISSIONS),
),
RoleRecord(
code=ROLE_VIEWER,
label=ROLE_LABELS[ROLE_VIEWER],
description="仅查看自己的业务数据,不可修改",
is_system=True,
is_admin=False,
permissions=list(VIEWER_PERMISSIONS),
),
]
def set_role_cache(roles: Iterable[RoleRecord]) -> None:
global _ROLE_CACHE
_ROLE_CACHE = {role.code: role for role in roles}
def get_cached_role(code: str | None) -> RoleRecord | None:
if not code:
return None
return _ROLE_CACHE.get(str(code))
def list_cached_roles() -> list[RoleRecord]:
return list(_ROLE_CACHE.values())
def role_label(code: str | None) -> str:
role = get_cached_role(code)
if role:
return role.label
return ROLE_LABELS.get(str(code or ""), str(code or ""))
def is_admin(role: str | None) -> bool:
"""True for the built-in admin role (global admin flag)."""
record = get_cached_role(role)
if record is not None:
return bool(record.is_admin)
return str(role or "") == ROLE_ADMIN
def has_global_scope(role: str | None) -> bool:
"""True when the role may see all users' data (admin or data.scope_all)."""
if is_admin(role):
return True
return has_permission(role, DATA_SCOPE_ALL)
def can_write(role: str | None) -> bool:
record = get_cached_role(role)
if record is not None:
if record.is_admin:
return True
held = expand_legacy_bundles(set(record.permissions))
return any(code in WRITE_PERMISSIONS for code in held)
return str(role or "") in (ROLE_ADMIN, ROLE_OPERATOR)
def can_manage_users(role: str | None) -> bool:
return has_permission(role, "users.manage")
def can_manage_roles(role: str | None) -> bool:
return has_permission(role, "roles.manage")
def has_permission(role: str | None, permission: str) -> bool:
code = str(permission or "").strip()
if not code:
return False
record = get_cached_role(role)
if record is None:
if str(role or "") == ROLE_ADMIN:
return True
if str(role or "") == ROLE_OPERATOR:
return permission_implies(expand_legacy_bundles(set(OPERATOR_PERMISSIONS)), code)
if str(role or "") == ROLE_VIEWER:
return permission_implies(set(VIEWER_PERMISSIONS), code)
return False
if record.is_admin:
return True
held = expand_legacy_bundles(set(record.permissions))
return permission_implies(held, code)
def permissions_for_role(role: str | None) -> list[str]:
record = get_cached_role(role)
if record is None:
if str(role or "") == ROLE_ADMIN:
return list(ALL_PERMISSIONS)
if str(role or "") == ROLE_OPERATOR:
from .permissions import expand_paired_permissions
return expand_paired_permissions(OPERATOR_PERMISSIONS)
if str(role or "") == ROLE_VIEWER:
return list(VIEWER_PERMISSIONS)
return []
if record.is_admin:
return list(ALL_PERMISSIONS)
return list(record.permissions)
def ensure_role(role: str) -> str:
if role not in ALL_ROLES:
raise ValueError(f"无效角色: {role}")
return role
"""Validate that a role code exists (cache or built-in fallback)."""
code = str(role or "").strip()
if not code:
raise ValueError("角色不能为空")
if get_cached_role(code) is not None:
return code
if code in ALL_ROLES:
return code
raise ValueError(f"无效角色: {code}")
def sanitize_role_permissions(
codes: list[str] | None,
*,
force_all: bool = False,
) -> list[str]:
if force_all:
return list(ALL_PERMISSIONS)
from .permissions import expand_paired_permissions
return expand_paired_permissions(codes)
+149 -16
View File
@@ -14,7 +14,7 @@ from .account_limits import (
normalize_max_accounts,
)
from .account_quota import default_stop_worker, sync_user_account_quota
from .dependencies import get_current_user, require_user_manager
from .dependencies import get_current_user, require_role_manager, require_user_manager
from .email_service import (
build_password_reset_link,
build_verification_link,
@@ -25,16 +25,31 @@ from .email_verification import create_verification_token, mask_email, verify_em
from .password_reset import create_password_reset_token, verify_password_reset_token
from .jwt_utils import create_access_token
from .passwords import hash_password, verify_password
from .roles import ALL_ROLES, ROLE_LABELS, ROLE_OPERATOR, ensure_role, is_admin
from .permissions import permission_catalog
from .role_service import (
count_users_with_role,
create_role,
delete_role,
ensure_role_assignable,
guard_last_admin_change,
list_roles as list_role_records,
update_role,
user_permission_payload,
)
from .roles import ROLE_OPERATOR, is_admin
from .scopes import ensure_user_manageable, users_for_manager
from .schemas import (
LoginRequest,
MessageResponse,
ForgotPasswordRequest,
ForgotPasswordResponse,
PermissionCatalogResponse,
RegisterRequest,
RegisterResponse,
ResendVerificationRequest,
ResetPasswordRequest,
RoleCreate,
RoleUpdate,
RolesResponse,
RoleInfo,
TokenResponse,
@@ -52,6 +67,10 @@ router = APIRouter(prefix="/api/auth", tags=["auth"])
async def _build_user_response(db: AsyncSession, user: User, with_count: bool = False) -> UserResponse:
payload = UserResponse.model_validate(user)
perm = user_permission_payload(user.role)
payload.role_label = perm["role_label"]
payload.is_admin = perm["is_admin"]
payload.permissions = perm["permissions"]
if with_count:
breakdown = await count_user_account_breakdown(db, user.id)
payload.account_count = breakdown["total"]
@@ -371,21 +390,40 @@ async def get_me(user: User = Depends(get_current_user), db: AsyncSession = Depe
@router.get("/roles", response_model=RolesResponse)
async def list_roles(_: User = Depends(get_current_user)):
async def list_auth_roles(
db: AsyncSession = Depends(get_db),
_: User = Depends(get_current_user),
):
"""Lightweight role list for dropdowns (any logged-in user).
Intentionally omits permission arrays to avoid leaking the full ACL map.
"""
records = await list_role_records(db)
return RolesResponse(
roles=[RoleInfo(value=r, label=ROLE_LABELS.get(r, r)) for r in ALL_ROLES]
roles=[
RoleInfo(
value=item.code,
label=item.label,
description=None,
is_system=item.is_system,
is_admin=item.is_admin,
permissions=[],
)
for item in records
]
)
users_router = APIRouter(prefix="/api/users", tags=["users"])
roles_router = APIRouter(prefix="/api/roles", tags=["roles"])
@users_router.get("", response_model=list[UserResponse])
async def list_users(
db: AsyncSession = Depends(get_db),
_: User = Depends(require_user_manager),
current: User = Depends(require_user_manager),
):
result = await db.execute(select(User).order_by(User.id.asc()))
result = await db.execute(users_for_manager(current).order_by(User.id.asc()))
users = result.scalars().all()
responses = []
for user in users:
@@ -397,16 +435,15 @@ async def list_users(
async def create_user(
body: UserCreate,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_user_manager),
current: User = Depends(require_user_manager),
):
settings = await load_settings(db)
exists = await db.execute(select(User).where(User.username == body.username))
if exists.scalar_one_or_none():
raise HTTPException(status_code=400, detail="用户名已存在")
try:
role = ensure_role(body.role)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
role = await ensure_role_assignable(db, body.role)
if is_admin(role) and not is_admin(current.role):
raise HTTPException(status_code=403, detail="只有管理员可以分配管理员角色")
email = await _ensure_email_available(db, str(body.email) if body.email else None)
if settings.email_binding_required and not is_admin(role) and not email:
raise HTTPException(status_code=400, detail="系统已开启「登录必须绑定邮箱」,请填写邮箱")
@@ -424,6 +461,7 @@ async def create_user(
email_verified=email_verified,
email_verified_at=datetime.utcnow() if email and email_verified else None,
max_accounts=normalize_max_accounts(body.max_accounts, role),
created_by=current.id,
)
db.add(user)
await db.commit()
@@ -442,17 +480,25 @@ async def update_user(
user = result.scalar_one_or_none()
if not user:
raise HTTPException(status_code=404, detail="用户不存在")
ensure_user_manageable(current, user)
settings = await load_settings(db)
if user.id == current.id and body.is_active is False:
raise HTTPException(status_code=400, detail="不能禁用当前登录账号")
updates = body.model_dump(exclude_unset=True)
if body.is_active is False:
await guard_last_admin_change(db, user=user, deactivating=True)
if body.role is not None:
new_role = await ensure_role_assignable(db, body.role)
if is_admin(new_role) and not is_admin(current.role):
raise HTTPException(status_code=403, detail="只有管理员可以分配管理员角色")
await guard_last_admin_change(db, user=user, new_role=new_role)
if body.display_name is not None:
user.display_name = body.display_name
if body.role is not None:
prev_role = user.role
try:
user.role = ensure_role(body.role)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
user.role = await ensure_role_assignable(db, body.role)
if is_admin(user.role):
user.max_accounts = UNLIMITED_ACCOUNTS
await sync_user_account_quota(db, user, stop_worker=default_stop_worker)
@@ -465,7 +511,6 @@ async def update_user(
if body.password:
user.password_hash = hash_password(body.password)
updates = body.model_dump(exclude_unset=True)
if "max_accounts" in updates and not is_admin(user.role):
user.max_accounts = normalize_max_accounts(updates["max_accounts"], user.role)
await sync_user_account_quota(db, user, stop_worker=default_stop_worker)
@@ -506,6 +551,94 @@ async def delete_user(
user = result.scalar_one_or_none()
if not user:
raise HTTPException(status_code=404, detail="用户不存在")
ensure_user_manageable(current, user)
await guard_last_admin_change(db, user=user, deleting=True)
await db.delete(user)
await db.commit()
return {"message": "用户已删除"}
@roles_router.get("", response_model=RolesResponse)
async def admin_list_roles(
db: AsyncSession = Depends(get_db),
_: User = Depends(require_role_manager),
):
records = await list_role_records(db)
roles = []
for item in records:
roles.append(
RoleInfo(
value=item.code,
label=item.label,
description=item.description or None,
is_system=item.is_system,
is_admin=item.is_admin,
permissions=list(item.permissions),
user_count=await count_users_with_role(db, item.code),
)
)
return RolesResponse(roles=roles)
@roles_router.get("/catalog", response_model=PermissionCatalogResponse)
async def get_permission_catalog(_: User = Depends(require_role_manager)):
return PermissionCatalogResponse(**permission_catalog())
@roles_router.post("", response_model=RoleInfo)
async def create_custom_role(
body: RoleCreate,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_role_manager),
):
record = await create_role(
db,
code=body.code,
label=body.label,
description=body.description,
permissions=body.permissions,
)
return RoleInfo(
value=record.code,
label=record.label,
description=record.description or None,
is_system=record.is_system,
is_admin=record.is_admin,
permissions=list(record.permissions),
user_count=0,
)
@roles_router.put("/{code}", response_model=RoleInfo)
async def update_custom_role(
code: str,
body: RoleUpdate,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_role_manager),
):
record = await update_role(
db,
code,
label=body.label,
description=body.description,
permissions=body.permissions,
)
return RoleInfo(
value=record.code,
label=record.label,
description=record.description or None,
is_system=record.is_system,
is_admin=record.is_admin,
permissions=list(record.permissions),
user_count=await count_users_with_role(db, record.code),
)
@roles_router.delete("/{code}")
async def delete_custom_role(
code: str,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_role_manager),
):
await delete_role(db, code)
return {"message": "角色已删除"}
+29
View File
@@ -67,6 +67,9 @@ class UserResponse(BaseModel):
email: Optional[str] = None
display_name: Optional[str] = None
role: str
role_label: str = ""
is_admin: bool = False
permissions: list[str] = Field(default_factory=list)
is_active: bool
email_verified: bool = False
max_accounts: int = 3
@@ -102,7 +105,33 @@ class UserUpdate(BaseModel):
class RoleInfo(BaseModel):
value: str
label: str
description: Optional[str] = None
is_system: bool = False
is_admin: bool = False
permissions: list[str] = Field(default_factory=list)
user_count: Optional[int] = None
class RolesResponse(BaseModel):
roles: list[RoleInfo]
class RoleCreate(BaseModel):
code: str = Field(min_length=2, max_length=50)
label: str = Field(min_length=1, max_length=100)
description: Optional[str] = Field(default=None, max_length=255)
permissions: list[str] = Field(default_factory=list)
class RoleUpdate(BaseModel):
label: Optional[str] = Field(default=None, min_length=1, max_length=100)
description: Optional[str] = Field(default=None, max_length=255)
permissions: Optional[list[str]] = None
class PermissionCatalogResponse(BaseModel):
menus: list[dict[str, Any]]
actions: list[dict[str, Any]]
data: list[dict[str, Any]] = Field(default_factory=list)
tree: list[dict[str, Any]] = Field(default_factory=list)
pairs: list[dict[str, Any]] = Field(default_factory=list)
+59 -17
View File
@@ -5,7 +5,12 @@ from sqlalchemy import or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from models.models import Account, AutoReplyRule, MessageLog, ReceivedMessageLog, SystemLog, User
from .roles import is_admin
from .permissions import (
ACCOUNTS_UPDATE,
ACCOUNTS_WRITE_GRANULAR,
RULES_WRITE,
)
from .roles import has_global_scope, has_permission, is_admin
async def get_owned_account(
@@ -14,29 +19,42 @@ async def get_owned_account(
account_id: int,
*,
write: bool = False,
write_permission: str | None = None,
) -> Account:
result = await db.execute(select(Account).where(Account.id == account_id))
account = result.scalar_one_or_none()
if not account:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="账号不存在")
if is_admin(user.role):
if has_global_scope(user.role):
if write:
needed = write_permission or ACCOUNTS_UPDATE
if not has_permission(user.role, needed):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"缺少权限:{needed}",
)
return account
if account.owner_id != user.id:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="无权访问该账号")
if write and user.role == "viewer":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="只读用户无法修改")
if write:
needed = write_permission or ACCOUNTS_UPDATE
if not has_permission(user.role, needed):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=f"缺少权限:{needed}",
)
return account
def accounts_for_user(user: User):
stmt = select(Account)
if not is_admin(user.role):
if not has_global_scope(user.role):
stmt = stmt.where(Account.owner_id == user.id)
return stmt
async def owned_account_ids(db: AsyncSession, user: User) -> Optional[set[int]]:
if is_admin(user.role):
if has_global_scope(user.role):
return None
result = await db.execute(select(Account.id).where(Account.owner_id == user.id))
return {row[0] for row in result.all()}
@@ -46,7 +64,7 @@ def logs_for_user(user: User, account_id: Optional[int] = None):
stmt = select(MessageLog)
if account_id is not None:
stmt = stmt.where(MessageLog.account_id == account_id)
if not is_admin(user.role):
if not has_global_scope(user.role):
owned = select(Account.id).where(Account.owner_id == user.id)
stmt = stmt.where(MessageLog.account_id.in_(owned))
return stmt
@@ -56,7 +74,7 @@ def received_logs_for_user(user: User, account_id: Optional[int] = None):
stmt = select(ReceivedMessageLog)
if account_id is not None:
stmt = stmt.where(ReceivedMessageLog.account_id == account_id)
if not is_admin(user.role):
if not has_global_scope(user.role):
owned = select(Account.id).where(Account.owner_id == user.id)
stmt = stmt.where(ReceivedMessageLog.account_id.in_(owned))
return stmt
@@ -66,31 +84,33 @@ def rules_for_user(user: User, account_id: Optional[int] = None):
stmt = select(AutoReplyRule)
if account_id is not None:
stmt = stmt.where(AutoReplyRule.account_id == account_id)
if is_admin(user.role):
if has_global_scope(user.role):
return stmt
owned = select(Account.id).where(Account.owner_id == user.id)
return stmt.where(AutoReplyRule.account_id.in_(owned))
async def get_accessible_rule(db: AsyncSession, user: User, rule_id: int, *, write: bool = False) -> AutoReplyRule:
async def get_accessible_rule(
db: AsyncSession, user: User, rule_id: int, *, write: bool = False
) -> AutoReplyRule:
result = await db.execute(select(AutoReplyRule).where(AutoReplyRule.id == rule_id))
rule = result.scalar_one_or_none()
if not rule:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="规则不存在")
if is_admin(user.role):
if write and user.role == "viewer":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="只读用户无法修改")
if has_global_scope(user.role):
if write and not has_permission(user.role, RULES_WRITE):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="缺少权限:rules.write")
return rule
if rule.account_id is None:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="无权访问全局规则")
account = await get_owned_account(db, user, rule.account_id, write=write)
account = await get_owned_account(db, user, rule.account_id, write=False)
if rule.owner_id and rule.owner_id != user.id and account.owner_id != user.id:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="无权访问该规则")
if write and user.role == "viewer":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="只读用户无法修改")
if write and not has_permission(user.role, RULES_WRITE):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="缺少权限:rules.write")
return rule
@@ -98,7 +118,7 @@ def system_logs_for_user(user: User, account_id: Optional[int] = None):
stmt = select(SystemLog)
if account_id is not None:
stmt = stmt.where(SystemLog.account_id == account_id)
if not is_admin(user.role):
if not has_global_scope(user.role):
owned = select(Account.id).where(Account.owner_id == user.id)
stmt = stmt.where(
or_(
@@ -107,3 +127,25 @@ def system_logs_for_user(user: User, account_id: Optional[int] = None):
)
)
return stmt
def users_for_manager(manager: User):
"""Admins see all users; others only see themselves and users they created."""
stmt = select(User)
if is_admin(manager.role):
return stmt
return stmt.where(or_(User.created_by == manager.id, User.id == manager.id))
def ensure_user_manageable(manager: User, target: User) -> None:
"""Raise 403 unless manager may edit/delete target user."""
if is_admin(manager.role):
return
if target.id == manager.id:
return
if target.created_by == manager.id:
return
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="只能管理自己创建的用户",
)
+12 -11
View File
@@ -14,7 +14,8 @@ from models.db_config import (
)
from models.db_transfer import inspect_sqlite_source, migrate_sqlite_to_target
from models.models import User
from .dependencies import require_admin
from .dependencies import require_permission
from .permissions import PAYMENTS_MANAGE, SETTINGS_DATABASE, SETTINGS_MANAGE
from .email_service import send_test_email
from .system_settings import (
PASSWORD_PLACEHOLDER,
@@ -208,7 +209,7 @@ async def get_public_settings(db: AsyncSession = Depends(get_db)):
@router.get("", response_model=SystemSettingsResponse)
async def get_system_settings(
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_MANAGE)),
):
data = await load_settings(db)
return SystemSettingsResponse(**settings_to_admin_response(data))
@@ -218,7 +219,7 @@ async def get_system_settings(
async def update_system_settings(
body: SystemSettingsUpdate,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_MANAGE)),
):
updates = body.model_dump(exclude_unset=True)
if "app_url" in updates and updates["app_url"]:
@@ -230,7 +231,7 @@ async def update_system_settings(
@router.get("/payment", response_model=PaymentSettingsResponse)
async def get_payment_settings(
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(PAYMENTS_MANAGE)),
):
data = await load_settings(db)
return PaymentSettingsResponse(**settings_to_payment_response(data))
@@ -240,7 +241,7 @@ async def get_payment_settings(
async def update_payment_settings(
body: PaymentSettingsUpdate,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(PAYMENTS_MANAGE)),
):
updates = body.model_dump(exclude_unset=True)
data = await save_settings(db, updates)
@@ -251,7 +252,7 @@ async def update_payment_settings(
async def test_smtp_email(
body: TestEmailRequest,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_MANAGE)),
):
data = await load_settings(db)
overrides = body.model_dump(exclude_unset=True, exclude={"to_email"})
@@ -276,14 +277,14 @@ async def test_smtp_email(
@router.get("/database", response_model=DatabaseSettingsResponse)
async def get_database_settings(_: User = Depends(require_admin)):
async def get_database_settings(_: User = Depends(require_permission(SETTINGS_DATABASE))):
return DatabaseSettingsResponse(**database_config_to_response())
@router.put("/database", response_model=MessageResponse)
async def update_database_settings(
body: DatabaseSettingsUpdate,
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_DATABASE)),
):
payload = body.model_dump(exclude_unset=True)
if payload.get("db_password") in (None, "", DB_PASSWORD_PLACEHOLDER):
@@ -302,7 +303,7 @@ async def update_database_settings(
@router.post("/database/test", response_model=MessageResponse)
async def test_database_settings(
body: DatabaseTestRequest,
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_DATABASE)),
):
payload = body.model_dump(exclude_unset=True)
if payload.get("db_password") in (None, "", DB_PASSWORD_PLACEHOLDER):
@@ -317,7 +318,7 @@ async def test_database_settings(
@router.get("/database/migrate/preview", response_model=DatabaseMigratePreviewResponse)
async def preview_database_migration(
source_db_path: str | None = None,
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_DATABASE)),
):
return DatabaseMigratePreviewResponse(**await inspect_sqlite_source(source_db_path))
@@ -325,7 +326,7 @@ async def preview_database_migration(
@router.post("/database/migrate", response_model=DatabaseMigrateResponse)
async def migrate_database_data(
body: DatabaseMigrateRequest,
_: User = Depends(require_admin),
_: User = Depends(require_permission(SETTINGS_DATABASE)),
):
payload = body.model_dump(exclude_unset=True)
clear_target = bool(payload.pop("clear_target", False))
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+855
View File
@@ -0,0 +1,855 @@
{
"frames": [
{
"index": 0,
"url": "https://www.douyin.com/",
"qrcodes": [
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "AXNt5Hoz",
"outer": "<img class=\"AXNt5Hoz\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "Wzqh8kMJ",
"outer": "<img class=\"Wzqh8kMJ\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "MaDupF4a",
"outer": "<img class=\"MaDupF4a\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "RSP3dVtx",
"outer": "<img class=\"RSP3dVtx\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "M3dFOzE4",
"outer": "<img class=\"M3dFOzE4\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "sB_GUV4n",
"outer": "<img class=\"sB_GUV4n\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "BD9BarA8",
"outer": "<img class=\"BD9BarA8\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "jMPyhzfG",
"outer": "<img class=\"jMPyhzfG\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "efPPcdLl",
"outer": "<img class=\"efPPcdLl\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "RnpNMA46",
"outer": "<img class=\"RnpNMA46\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "FqRV7w1P",
"outer": "<img class=\"FqRV7w1P\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "_Whzlv1b",
"outer": "<img class=\"_Whzlv1b\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "j1TwxzPC",
"outer": "<img class=\"j1TwxzPC\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "zYgniQaG",
"outer": "<img class=\"zYgniQaG\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "uJKU1tdN",
"outer": "<img class=\"uJKU1tdN\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "HkY6seUs",
"outer": "<img class=\"HkY6seUs\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "khDBMSjy",
"outer": "<img class=\"khDBMSjy\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "GkION2OQ",
"outer": "<img class=\"GkION2OQ\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "q2uupcgz",
"outer": "<img class=\"q2uupcgz\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 16,
"y": 58,
"width": 128,
"height": 40
},
"src_prefix": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "n6fjbOcQ",
"outer": "<img class=\"n6fjbOcQ\" src=\"https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png\">",
"screenshot": "frame0_IMG_16_58.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 539.328125,
"y": 108,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_ea6e0dc453683550de835cc2ded929b0~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_ea6e0dc453683550de835cc2ded929b0~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_539_108.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 894.65625,
"y": 108,
"width": 339.34375,
"height": 190.875
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_dfc1793c444e4e9731b08c24d409cfff~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_dfc1793c444e4e9731b08c24d409cfff~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_894_108.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 184,
"y": 412.875,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_032fb5c8e6ddd4703e115f1139fe01cf~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_032fb5c8e6ddd4703e115f1139fe01cf~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_184_412.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 539.328125,
"y": 412.875,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p9-pc-sign.douyinpic.com/tos-cn-i-dy/ef60f35dda7a4b1396df4b5b5abfb632~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?from",
"alt": "一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p9-pc-sign.douyinpic.com/tos-cn-i-dy/ef60f35dda7a4b1396df4b5b5abfb632~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?from=1189464143&amp;lk3s=46e5c84f&amp;x-expires=1788685200&amp;x-signature=YYrBL5uYHi0RBKP8Do8J0iMGHms%3D\" alt=\"一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选\" class=",
"screenshot": "frame0_IMG_539_412.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 894.65625,
"y": 412.875,
"width": 339.34375,
"height": 190.875
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_d4d6aace7531ed9cbd364c4313a21352~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_d4d6aace7531ed9cbd364c4313a21352~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_894_412.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 184,
"y": 717.75,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_49bbd3ffa13175f85795123107c70169~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元 #剧情 #反转 #扮猪吃虎名场面",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_49bbd3ffa13175f85795123107c70169~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_184_717.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 539.328125,
"y": 708.75,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/70d71f59a2bab8ab8a238f9276777e7a~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?fr",
"alt": "深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/70d71f59a2bab8ab8a238f9276777e7a~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?from=1189464143&amp;lk3s=46e5c84f&amp;x-expires=1788685200&amp;x-signature=vsIbNW%2BtgYYmFILlWMn38utpuzg%3D\" alt=\"深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史\" clas",
"screenshot": "frame0_IMG_539_708.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 894.65625,
"y": 708.75,
"width": 339.34375,
"height": 190.875
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_61db88f71bdba90e436a783bae92863f~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家 #洲人洲事",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_61db88f71bdba90e436a783bae92863f~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_894_708.png"
},
{
"selector": "img",
"tag": "IMG",
"visible": true,
"box": {
"x": 184,
"y": 1013.625,
"width": 339.328125,
"height": 190.859375
},
"src_prefix": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_17d53951c4fb7c73dbfa5acddfde35a3~tplv-dy-resize-walign-adapt-aq:540:q7",
"alt": "本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划",
"class": "XTdkxrLI discover-video-card-img",
"outer": "<img src=\"https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_17d53951c4fb7c73dbfa5acddfde35a3~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&amp;from=1189464143&amp;lk3s=46e5c84f&amp;s=PackSourceEnum_DOUYIN_WEB_NEW_PAGE&amp;sc=cover&amp;se=false&amp;x-expires=1789030800&amp;x-signatu",
"screenshot": "frame0_IMG_184_1013.png"
}
],
"panels": []
},
{
"index": 1,
"url": "https://lf-rc1.yhgfb-cn-static.com/obj/rc-verifycenter/rmc-nocaptcha/1.0.0.50/index.html",
"qrcodes": [],
"panels": []
}
],
"candidates": [
{
"tag": "IMG",
"index": 0,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "AXNt5Hoz",
"parentText": ""
},
{
"tag": "IMG",
"index": 1,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "Wzqh8kMJ",
"parentText": ""
},
{
"tag": "IMG",
"index": 2,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "MaDupF4a",
"parentText": ""
},
{
"tag": "IMG",
"index": 3,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "RSP3dVtx",
"parentText": ""
},
{
"tag": "IMG",
"index": 4,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "M3dFOzE4",
"parentText": ""
},
{
"tag": "IMG",
"index": 5,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "sB_GUV4n",
"parentText": ""
},
{
"tag": "IMG",
"index": 6,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "BD9BarA8",
"parentText": ""
},
{
"tag": "IMG",
"index": 7,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app.beb4a83a0d39bb5d.png",
"alt": "",
"class": "jMPyhzfG",
"parentText": ""
},
{
"tag": "IMG",
"index": 8,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "efPPcdLl",
"parentText": ""
},
{
"tag": "IMG",
"index": 9,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/get_app_hover.c9e81f511d248ae7.png",
"alt": "",
"class": "RnpNMA46",
"parentText": ""
},
{
"tag": "IMG",
"index": 10,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "FqRV7w1P",
"parentText": ""
},
{
"tag": "IMG",
"index": 11,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "_Whzlv1b",
"parentText": ""
},
{
"tag": "IMG",
"index": 12,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "j1TwxzPC",
"parentText": ""
},
{
"tag": "IMG",
"index": 13,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "zYgniQaG",
"parentText": ""
},
{
"tag": "IMG",
"index": 14,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "uJKU1tdN",
"parentText": ""
},
{
"tag": "IMG",
"index": 15,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "HkY6seUs",
"parentText": ""
},
{
"tag": "IMG",
"index": 16,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "khDBMSjy",
"parentText": ""
},
{
"tag": "IMG",
"index": 17,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app.795ee19b3ececa6b.png",
"alt": "",
"class": "GkION2OQ",
"parentText": ""
},
{
"tag": "IMG",
"index": 18,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "q2uupcgz",
"parentText": ""
},
{
"tag": "IMG",
"index": 19,
"width": 128,
"height": 40,
"x": 16,
"y": 58,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/fold_get_app_hover.911c3dd97d692c11.png",
"alt": "",
"class": "n6fjbOcQ",
"parentText": ""
},
{
"tag": "IMG",
"index": 20,
"width": 128,
"height": 123,
"x": 16,
"y": 701,
"src": "https://lf-douyin-pc-web.douyinstatic.com/obj/douyin-pc-web/ies/douyin_web/media/jxBtnBgV4.4405b8dd83623e92.png",
"alt": "",
"class": "ACBHzWNP",
"parentText": "手机随时看更方便\n下载 APP"
},
{
"tag": "IMG",
"index": 27,
"width": 339.328125,
"height": 190.859375,
"x": 184,
"y": -343,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_32cb3c82390459d1d91bd4f30c5d8ce7~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "【清稚竹马】我还想说,我想你了!#ai漫剧 #原创动画 #漫剧 #校园",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 28,
"width": 339.328125,
"height": 190.859375,
"x": 539.328125,
"y": -343,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_ea6e0dc453683550de835cc2ded929b0~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "法国搞笑三人组新作,结尾太好笑了 法国喜剧#电影长尾豹马修 #喜剧电影解说",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 29,
"width": 339.34375,
"height": 190.875,
"x": 894.65625,
"y": -343,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_dfc1793c444e4e9731b08c24d409cfff~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "你我怎么两清……#戴上耳机 #甲乙丙丁 #李佳薇 #音乐分享",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 30,
"width": 339.328125,
"height": 190.859375,
"x": 184,
"y": -38.125,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_032fb5c8e6ddd4703e115f1139fe01cf~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "被外卖大哥不小心蹭了车,但没想到他的手机铃声竟然是我的歌…但也正因如此我才有幸走进了一个父与子的故事里#人间观察计划#外卖小哥 #看见100种生活#日常分享 #雪下的时候",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 31,
"width": 339.328125,
"height": 190.859375,
"x": 539.328125,
"y": -38.125,
"src": "https://p9-pc-sign.douyinpic.com/tos-cn-i-dy/ef60f35dda7a4b1396df4b5b5abfb632~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?from=1189464143&lk3s=46e5c84f&x-expires=1788685200&x-signature=YYrBL5uYHi0RBKP8Do8J0",
"alt": "一口气听完当年火遍全网的说唱,谁的DNA动了#中文说唱 #马思唯 #kkluv #创作者扶持计划 #抖音精选",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 32,
"width": 339.34375,
"height": 190.875,
"x": 894.65625,
"y": -38.125,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_d4d6aace7531ed9cbd364c4313a21352~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "当你穿进老钱班33#老钱班 #侯绿萝#olly懂你漂亮做自己 #olly女性复合维生素",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 33,
"width": 339.328125,
"height": 190.859375,
"x": 184,
"y": 266.75,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_49bbd3ffa13175f85795123107c70169~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "轮回神话5 女儿试炼误入绝境,获S级血统轰动全宇宙!探秘禁忌陵宫,竟发现横扫万界的创世神正是自家咸鱼老爸!#原创动画 #二次元 #剧情 #反转 #扮猪吃虎名场面",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 34,
"width": 339.328125,
"height": 190.859375,
"x": 539.328125,
"y": 266.75,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/70d71f59a2bab8ab8a238f9276777e7a~tplv-dy-vqe2-sr-opt1:640:480:q80.webp?from=1189464143&lk3s=46e5c84f&x-expires=1788685200&x-signature=vsIbNW%2BtgYYmFILlW",
"alt": "深度解析《大明王朝1566》 明成祖朱棣定下的锦衣卫选拔标准,一般人还真达不到#大明王朝1566 #历史",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 35,
"width": 339.34375,
"height": 190.875,
"x": 894.65625,
"y": 266.75,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_61db88f71bdba90e436a783bae92863f~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "当大哥不接暗号,鼠鼠带着九格强行认大哥会发生什么呢? #三角洲行动 #三角洲得吃就行挑战 #鼠鼠我呀得吃了 #三角洲最仁义玩家 #洲人洲事",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
},
{
"tag": "IMG",
"index": 36,
"width": 339.328125,
"height": 190.859375,
"x": 184,
"y": 571.625,
"src": "https://p3-pc-sign.douyinpic.com/image-cut-tos/dp_17d53951c4fb7c73dbfa5acddfde35a3~tplv-dy-resize-walign-adapt-aq:540:q75.jpeg?biz_tag=aweme_video&from=1189464143&lk3s=46e5c84f&s=PackSourceEnum_DOUYIN",
"alt": "本想应付体验大学生活的表弟,不料竟意外发现表弟的万能用处 #搞笑 #动漫 #轻漫计划 #充能计划",
"class": "XTdkxrLI discover-video-card-img",
"parentText": ""
}
]
}
+239
View File
@@ -0,0 +1,239 @@
"""抖音登录二维码元素结构诊断脚本"""
import asyncio
import base64
import json
import os
import sys
from playwright.async_api import async_playwright
ROOT = os.path.dirname(os.path.abspath(__file__))
PROJECT_ROOT = os.path.dirname(ROOT)
BROWSERS_PATH = os.path.join(PROJECT_ROOT, "playwright-browsers")
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", BROWSERS_PATH)
OUT_DIR = os.path.join(ROOT, "debug_qr")
os.makedirs(OUT_DIR, exist_ok=True)
async def inspect():
async with async_playwright() as p:
print("launching browser")
browser = await p.chromium.launch(headless=True)
context = await browser.new_context(
viewport={"width": 1280, "height": 900},
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
)
page = await context.new_page()
print("goto douyin.com")
await page.goto("https://www.douyin.com", wait_until="load")
await asyncio.sleep(6)
print("page url:", page.url)
print("page title:", await page.title())
html = await page.content()
with open(os.path.join(OUT_DIR, "page_initial.html"), "w", encoding="utf-8") as f:
f.write(html)
print("saved page_initial.html")
# 点击登录按钮,触发登录弹窗
login_clicked = False
for sel in ["text=登录", "text=登录/注册", "text=立即登录", "button:has-text('登录')", "[class*='login']", "[class*='Login']"]:
try:
el = await page.wait_for_selector(sel, timeout=3000)
if el:
await el.click()
print("clicked via selector:", sel)
login_clicked = True
break
except Exception as e:
print(f"selector {sel} failed: {e}")
if not login_clicked:
for attempt in range(3):
try:
clicked = await page.evaluate("""() => {
const nodes = [...document.querySelectorAll('button, span, div, a, p')];
for (const el of nodes) {
const t = (el.innerText || '').trim();
if ((t.includes('登录') || t.toLowerCase().includes('login')) && el.offsetParent) {
el.click();
return t;
}
}
return '';
}""")
print("clicked via js:", clicked)
if clicked:
break
except Exception as e:
print(f"js click attempt {attempt} err: {e}")
await asyncio.sleep(1)
await asyncio.sleep(4)
html = await page.content()
with open(os.path.join(OUT_DIR, "page_after_login_click.html"), "w", encoding="utf-8") as f:
f.write(html)
print("saved page_after_login_click.html")
try:
await page.screenshot(path=os.path.join(OUT_DIR, "00_viewport.png"), full_page=False, timeout=10000)
print("saved 00_viewport.png")
except Exception as e:
print("viewport screenshot failed:", e)
# 尝试切换「扫码登录」
for frame in page.frames:
try:
switched = await frame.evaluate("""() => {
const nodes = [...document.querySelectorAll('span, div, a, button, p')];
let best = null;
for (const el of nodes) {
const t = (el.innerText || '').trim();
if ((t === '扫码登录' || t === '扫码') && el.offsetParent) {
if (!best || el.children.length < best.children.length) best = el;
}
}
if (best) { best.click(); return 'switched'; }
return '';
}""")
print(f"frame {frame.url[:60]} switched={switched}")
except Exception as e:
print(f"frame switch err: {e}")
await asyncio.sleep(2)
report = {"frames": [], "candidates": []}
# 遍历所有 frame,查找二维码相关元素
for idx, frame in enumerate(page.frames):
frame_report = {"index": idx, "url": frame.url, "qrcodes": [], "panels": []}
selectors = [
"[class*='qrcode'] img",
"[class*='QrCode'] img",
"[class*='qr-code'] img",
"img[class*='qrcode']",
"img[src*='qrcode']",
"img[alt*='二维码']",
"img[alt*='qr']",
"[class*='qrcode'] canvas",
"canvas[class*='qrcode']",
"[class*='scan'] img",
"[class*='scan'] canvas",
"img",
"canvas",
]
for sel in selectors:
try:
els = await frame.query_selector_all(sel)
for el in els:
try:
visible = await el.is_visible()
box = await el.bounding_box()
tag = await el.evaluate("e => e.tagName")
src = await el.get_attribute("src") or ""
alt = await el.get_attribute("alt") or ""
cls = await el.get_attribute("class") or ""
outer = await el.evaluate("e => e.outerHTML.slice(0, 300)")
info = {
"selector": sel,
"tag": tag,
"visible": visible,
"box": box,
"src_prefix": src[:120] if src else "",
"alt": alt,
"class": cls,
"outer": outer,
}
if (tag.lower() in ("img", "canvas") and box and box.get("width", 0) > 40 and visible):
frame_report["qrcodes"].append(info)
# 截图该元素
safe_name = f"frame{idx}_{tag}_{int(box['x'])}_{int(box['y'])}.png"
try:
await el.screenshot(path=os.path.join(OUT_DIR, safe_name))
info["screenshot"] = safe_name
except Exception as e:
info["screenshot_err"] = str(e)
except Exception as e:
print(f" el inspect err: {e}")
except Exception as e:
print(f"frame {idx} selector {sel} err: {e}")
# 登录面板/容器
panel_selectors = [
"[class*='qrcode-container']",
"[class*='qrcodeContainer']",
"[class*='qrcode']",
"[class*='QrCode']",
"[class*='login-scan']",
"[class*='scan-code']",
"#login-pannel",
"[class*='login_panel']",
"[class*='login-panel']",
"[class*='account_login']",
]
for sel in panel_selectors:
try:
els = await frame.query_selector_all(sel)
for el in els:
visible = await el.is_visible()
box = await el.bounding_box()
cls = await el.get_attribute("class") or ""
if visible and box and box.get("width", 0) > 80:
frame_report["panels"].append({
"selector": sel,
"class": cls,
"box": box,
})
safe_name = f"frame{idx}_panel_{int(box['x'])}_{int(box['y'])}.png"
try:
await el.screenshot(path=os.path.join(OUT_DIR, safe_name))
frame_report["panels"][-1]["screenshot"] = safe_name
except Exception as e:
frame_report["panels"][-1]["screenshot_err"] = str(e)
except Exception as e:
pass
report["frames"].append(frame_report)
# 尝试用 JS 暴力查找所有 img/canvas 中可能为二维码的
all_candidates = await page.evaluate("""() => {
const out = [];
document.querySelectorAll('img, canvas').forEach((el, i) => {
const rect = el.getBoundingClientRect();
if (rect.width > 30 && rect.height > 30 && rect.width < 600 && rect.height < 600) {
const style = window.getComputedStyle(el);
out.push({
tag: el.tagName,
index: i,
width: rect.width,
height: rect.height,
x: rect.x,
y: rect.y,
src: el.tagName === 'IMG' ? (el.src || '').slice(0, 200) : '',
alt: el.alt || '',
class: el.className || '',
parentText: (el.parentElement ? el.parentElement.innerText : '').slice(0, 80),
});
}
});
return out;
}""")
report["candidates"] = all_candidates
with open(os.path.join(OUT_DIR, "report.json"), "w", encoding="utf-8") as f:
json.dump(report, f, ensure_ascii=False, indent=2)
print("report saved to", os.path.join(OUT_DIR, "report.json"))
print("found qrcode-like elements:", sum(len(f["qrcodes"]) for f in report["frames"]))
print("found panels:", sum(len(f["panels"]) for f in report["frames"]))
await browser.close()
if __name__ == "__main__":
try:
asyncio.run(inspect())
except Exception as e:
print("FATAL:", e, file=sys.stderr)
import traceback
traceback.print_exc()
raise
+6 -5
View File
@@ -19,7 +19,8 @@ from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from auth.dependencies import require_admin
from auth.dependencies import require_permission
from auth.permissions import DESKTOP_MANAGE
from auth.system_settings import get_cached_settings
from desktop_release import (
INSTALLER_DIR,
@@ -120,7 +121,7 @@ async def desktop_download(db: AsyncSession = Depends(get_db)):
async def get_release(
request: Request,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(DESKTOP_MANAGE)),
):
data = await load_release(db)
return _to_response(data, request)
@@ -131,7 +132,7 @@ async def update_release(
body: DesktopReleaseUpdate,
request: Request,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(DESKTOP_MANAGE)),
):
updates = body.model_dump(exclude_unset=True)
if "version" in updates and updates["version"] is not None:
@@ -152,7 +153,7 @@ async def upload_installer(
request: Request,
file: UploadFile = File(...),
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(DESKTOP_MANAGE)),
):
filename = (file.filename or "").strip()
if not filename.lower().endswith(".exe"):
@@ -194,7 +195,7 @@ async def upload_installer(
async def delete_installer(
request: Request,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(DESKTOP_MANAGE)),
):
remove_installer()
data = await save_release(db, {"installer_name": "", "installer_size": 0})
Binary file not shown.
Binary file not shown.
BIN
View File
Binary file not shown.
+7 -5
View File
@@ -12,7 +12,9 @@ from pydantic import BaseModel, Field
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from auth.dependencies import get_current_user, require_write
from auth.dependencies import get_current_user, require_link_cards_write
from auth.roles import has_permission
from auth.permissions import LINK_CARDS_WRITE
from link_cards import (
absolute_media_url,
build_keywords,
@@ -68,8 +70,8 @@ async def _get_owned_card(
card = (await db.execute(stmt)).scalar_one_or_none()
if not card or card.owner_id != user.id:
raise HTTPException(status_code=404, detail="卡片不存在")
if write and user.role == "viewer":
raise HTTPException(status_code=403, detail="无写入权限")
if write and not has_permission(user.role, LINK_CARDS_WRITE):
raise HTTPException(status_code=403, detail="缺少权限:link_cards.write")
return card
@@ -93,7 +95,7 @@ def _card_response(card: LinkCardPage, request: Request) -> LinkCardResponse:
async def upload_link_card_image(
request: Request,
file: UploadFile = File(...),
user: User = Depends(require_write),
user: User = Depends(require_link_cards_write),
):
if not file.content_type or not file.content_type.startswith("image/"):
raise HTTPException(status_code=400, detail="仅支持上传图片文件")
@@ -137,7 +139,7 @@ async def upsert_link_card(
body: LinkCardUpsert,
request: Request,
db: AsyncSession = Depends(get_db),
user: User = Depends(require_write),
user: User = Depends(require_link_cards_write),
):
title = body.title.strip()
content = (body.content or "").strip()
+942 -126
View File
File diff suppressed because it is too large Load Diff
+61 -3
View File
@@ -9,8 +9,10 @@ from typing import Any
from urllib.parse import quote_plus
from dotenv import load_dotenv
from sqlalchemy import text
from sqlalchemy import event, text
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
from sqlalchemy.engine import make_url
from sqlalchemy.pool import AsyncAdaptedQueuePool, StaticPool
BACKEND_DIR = Path(__file__).resolve().parent.parent
PROJECT_ROOT = BACKEND_DIR.parent
@@ -37,6 +39,10 @@ def _env_int(name: str, default: int) -> int:
return default
def _bounded_env_int(name: str, default: int, minimum: int, maximum: int) -> int:
return max(minimum, min(maximum, _env_int(name, default)))
def normalize_db_type(value: str | None) -> str:
raw = (value or "sqlite").strip().lower()
if raw in ("postgres", "pgsql"):
@@ -136,7 +142,35 @@ def build_database_url(config: DatabaseConfig | None = None) -> str:
def engine_kwargs_for_url(url: str) -> dict[str, Any]:
kwargs: dict[str, Any] = {"echo": False}
if not url.startswith("sqlite"):
if url.startswith("sqlite"):
# SQLAlchemy 2.0.30 defaults file-backed aiosqlite to NullPool. At
# hundreds of hosted accounts that creates and tears down an aiosqlite
# worker thread for every short query. Reuse a small bounded pool;
# SQLite still serializes writers, so a large pool only adds lock
# contention and does not improve throughput.
busy_timeout_ms = _bounded_env_int(
"KEFU_SQLITE_BUSY_TIMEOUT_MS", 30_000, 1_000, 120_000
)
kwargs["connect_args"] = {"timeout": busy_timeout_ms / 1000.0}
sqlite_database = make_url(url).database
is_memory_database = (
not sqlite_database
or sqlite_database == ":memory:"
or "mode=memory" in url.lower()
)
if is_memory_database:
# Every connection to an in-memory SQLite URL otherwise receives a
# different database. StaticPool preserves the single shared
# connection expected by tests and utility callers.
kwargs["poolclass"] = StaticPool
else:
kwargs["poolclass"] = AsyncAdaptedQueuePool
kwargs["pool_size"] = _bounded_env_int(
"KEFU_SQLITE_POOL_SIZE", 5, 1, 10
)
kwargs["max_overflow"] = 0
kwargs["pool_timeout"] = max(5.0, busy_timeout_ms / 1000.0)
else:
kwargs["pool_pre_ping"] = True
kwargs["pool_recycle"] = 3600
# 默认连接池仅 pool_size=5 + max_overflow=10。多账号托管 + 前端并发请求时
@@ -155,9 +189,33 @@ def engine_kwargs_for_url(url: str) -> dict[str, Any]:
return kwargs
def _configure_sqlite_connection(dbapi_connection, _connection_record) -> None:
"""Apply process-wide SQLite settings to every pooled connection.
WAL lets readers continue while the single writer commits. NORMAL avoids
a full disk sync for every small log/status transaction while retaining
WAL crash consistency. busy_timeout turns transient writer contention
into bounded waiting instead of immediate ``database is locked`` errors.
"""
busy_timeout_ms = _bounded_env_int(
"KEFU_SQLITE_BUSY_TIMEOUT_MS", 30_000, 1_000, 120_000
)
cursor = dbapi_connection.cursor()
try:
cursor.execute(f"PRAGMA busy_timeout={busy_timeout_ms}")
cursor.execute("PRAGMA journal_mode=WAL")
cursor.execute("PRAGMA synchronous=NORMAL")
cursor.execute("PRAGMA foreign_keys=ON")
finally:
cursor.close()
def create_database_engine(config: DatabaseConfig | None = None) -> AsyncEngine:
url = build_database_url(config)
return create_async_engine(url, **engine_kwargs_for_url(url))
engine = create_async_engine(url, **engine_kwargs_for_url(url))
if url.startswith("sqlite"):
event.listen(engine.sync_engine, "connect", _configure_sqlite_connection)
return engine
def mask_database_url(url: str) -> str:
+156 -2
View File
@@ -33,12 +33,62 @@ def add_column_if_missing(conn, table: str, column: str, ddl_by_dialect: dict[st
conn.execute(text(ddl))
def add_index_if_missing(
conn,
table: str,
index_name: str,
columns: tuple[str, ...],
) -> None:
"""Create one portable index without relying on dialect-specific IF NOT EXISTS."""
try:
insp = inspect(conn)
if not insp.has_table(table):
return
existing = {item.get("name") for item in insp.get_indexes(table)}
except Exception:
return
if index_name in existing:
return
safe_columns = ", ".join(columns)
conn.execute(text(f"CREATE INDEX {index_name} ON {table} ({safe_columns})"))
def widen_mysql_text_columns(conn, table: str, columns: tuple[str, ...]) -> None:
"""Upgrade large account payloads from TEXT to LONGTEXT on MySQL.
SQLite and PostgreSQL TEXT values are not limited to 64 KiB, while MySQL
TEXT is. Browser storage_state and full-page QR/captcha screenshots can
legitimately exceed that size.
"""
if _dialect(conn) != "mysql":
return
allowed = {"cookie_data", "im_session_data", "qr_code_base64"}
try:
reflected = {
item["name"]: str(item.get("type") or "").upper()
for item in inspect(conn).get_columns(table)
}
except Exception:
return
for column in columns:
if column not in allowed or column not in reflected:
continue
if reflected[column] == "LONGTEXT":
continue
conn.execute(
text(f"ALTER TABLE {table} MODIFY COLUMN {column} LONGTEXT NULL")
)
def migrate_accounts_table(conn) -> None:
add_column_if_missing(
conn,
"accounts",
"cookie_data",
{"default": "ALTER TABLE accounts ADD COLUMN cookie_data TEXT"},
{
"default": "ALTER TABLE accounts ADD COLUMN cookie_data TEXT",
"mysql": "ALTER TABLE accounts ADD COLUMN cookie_data LONGTEXT",
},
)
add_column_if_missing(
conn,
@@ -53,7 +103,10 @@ def migrate_accounts_table(conn) -> None:
conn,
"accounts",
"im_session_data",
{"default": "ALTER TABLE accounts ADD COLUMN im_session_data TEXT"},
{
"default": "ALTER TABLE accounts ADD COLUMN im_session_data TEXT",
"mysql": "ALTER TABLE accounts ADD COLUMN im_session_data LONGTEXT",
},
)
add_column_if_missing(
conn,
@@ -94,6 +147,18 @@ def migrate_accounts_table(conn) -> None:
"user_agent",
{"default": "ALTER TABLE accounts ADD COLUMN user_agent TEXT"},
)
add_column_if_missing(
conn,
"accounts",
"egress_public_ip",
{"default": "ALTER TABLE accounts ADD COLUMN egress_public_ip VARCHAR(64)"},
)
add_column_if_missing(
conn,
"accounts",
"egress_auto_attempts",
{"default": "ALTER TABLE accounts ADD COLUMN egress_auto_attempts INTEGER DEFAULT 1"},
)
add_column_if_missing(
conn,
"accounts",
@@ -106,6 +171,11 @@ def migrate_accounts_table(conn) -> None:
"douyin_uid",
{"default": "ALTER TABLE accounts ADD COLUMN douyin_uid VARCHAR(64)"},
)
widen_mysql_text_columns(
conn,
"accounts",
("cookie_data", "im_session_data", "qr_code_base64"),
)
def migrate_account_videos_table(conn) -> None:
@@ -124,6 +194,36 @@ def migrate_message_logs_table(conn) -> None:
"sender_avatar",
{"default": "ALTER TABLE message_logs ADD COLUMN sender_avatar TEXT"},
)
add_index_if_missing(
conn,
"message_logs",
"ix_message_logs_account_created_at",
("account_id", "created_at"),
)
add_index_if_missing(
conn,
"message_logs",
"ix_message_logs_created_at",
("created_at",),
)
add_index_if_missing(
conn,
"message_logs",
"ix_message_logs_status_account_id",
("status", "account_id"),
)
add_index_if_missing(
conn,
"received_message_logs",
"ix_received_message_logs_account_created_at",
("account_id", "created_at"),
)
add_index_if_missing(
conn,
"system_logs",
"ix_system_logs_account_created_at",
("account_id", "created_at"),
)
def migrate_rules_table(conn) -> None:
@@ -146,6 +246,54 @@ def migrate_rules_table(conn) -> None:
)
def migrate_roles_table(conn) -> None:
"""Ensure roles table exists (create_all usually handles this; keep as safety net)."""
try:
insp = inspect(conn)
if insp.has_table("roles"):
return
except Exception:
return
dialect = _dialect(conn)
if dialect == "postgresql":
conn.execute(
text(
"""
CREATE TABLE roles (
id SERIAL PRIMARY KEY,
code VARCHAR(50) NOT NULL UNIQUE,
label VARCHAR(100) NOT NULL,
description VARCHAR(255),
is_system BOOLEAN DEFAULT FALSE,
is_admin BOOLEAN DEFAULT FALSE,
permissions TEXT NOT NULL DEFAULT '[]',
created_at TIMESTAMP,
updated_at TIMESTAMP
)
"""
)
)
else:
conn.execute(
text(
"""
CREATE TABLE IF NOT EXISTS roles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
code VARCHAR(50) NOT NULL UNIQUE,
label VARCHAR(100) NOT NULL,
description VARCHAR(255),
is_system BOOLEAN DEFAULT 0,
is_admin BOOLEAN DEFAULT 0,
permissions TEXT NOT NULL DEFAULT '[]',
created_at DATETIME,
updated_at DATETIME
)
"""
)
)
add_index_if_missing(conn, "roles", "ix_roles_code", ("code",))
def migrate_users_table(conn) -> None:
add_column_if_missing(
conn,
@@ -177,6 +325,12 @@ def migrate_users_table(conn) -> None:
"max_accounts",
{"default": "ALTER TABLE users ADD COLUMN max_accounts INTEGER DEFAULT 3"},
)
add_column_if_missing(
conn,
"users",
"created_by",
{"default": "ALTER TABLE users ADD COLUMN created_by INTEGER"},
)
cols = _table_columns(conn, "users")
if not cols:
return
+65 -7
View File
@@ -1,7 +1,31 @@
from datetime import datetime
from sqlalchemy import Column, Integer, String, Boolean, DateTime, ForeignKey, Text, UniqueConstraint
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, String, Boolean, DateTime, ForeignKey, Index, Text, UniqueConstraint
from sqlalchemy.dialects.mysql import LONGTEXT
from sqlalchemy.orm import relationship, validates
from .database import Base
from utils.log_limits import bound_error_log_content, bound_message_log_content
class Role(Base):
"""Assignable role with a fixed permission-code list.
``users.role`` stores ``Role.code``. Built-in roles are seeded on startup;
custom roles are owned by admins via the users.manage / roles.manage permissions.
"""
__tablename__ = "roles"
id = Column(Integer, primary_key=True, index=True)
code = Column(String(50), unique=True, index=True, nullable=False)
label = Column(String(100), nullable=False)
description = Column(String(255), nullable=True)
is_system = Column(Boolean, default=False)
# Global data scope + unlimited account quota. Only the built-in admin
# role may be true; custom roles are always own-scoped.
is_admin = Column(Boolean, default=False)
permissions = Column(Text, nullable=False, default="[]") # JSON string list
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
class User(Base):
@@ -12,11 +36,18 @@ class User(Base):
email = Column(String(255), unique=True, index=True, nullable=True)
password_hash = Column(String(255), nullable=False)
display_name = Column(String(100), nullable=True)
role = Column(String(20), default="operator", index=True) # admin, operator, viewer
role = Column(String(50), default="operator", index=True) # roles.code
is_active = Column(Boolean, default=True)
email_verified = Column(Boolean, default=False)
email_verified_at = Column(DateTime, nullable=True)
max_accounts = Column(Integer, default=3)
# User who created this account via「用户管理」; null for self-register / seeded.
created_by = Column(
Integer,
ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
index=True,
)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
@@ -78,15 +109,17 @@ class Account(Base):
phone = Column(String(20), nullable=True) # 绑定的手机号(可选)
status = Column(String(50), default="offline") # offline, logging_in, online, error
cookie_path = Column(String(255), nullable=True) # 存储 cookie/session 的路径
cookie_data = Column(Text, nullable=True) # Playwright storage_state JSON
cookie_data = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # Playwright storage_state JSON
cookie_updated_at = Column(DateTime, nullable=True) # Cookie 最近更新时间
im_session_data = Column(Text, nullable=True) # IM 直连会话 (WS URL / device_id 等)
im_session_data = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # IM 直连会话 (WS URL / device_id 等)
reply_delay_seconds = Column(Integer, default=0) # 账号回复排队间隔;0/NULL=继承系统默认
reply_cooldown_seconds = Column(Integer, nullable=True) # 自动回复冷却秒数;NULL=继承全局设置
follow_welcome_enabled = Column(Boolean, default=False) # 新粉丝关注后自动发送欢迎语
follow_welcome_content = Column(Text, nullable=True) # 关注欢迎语内容(空=不发)
user_agent = Column(Text, nullable=True) # 伪装设备头(User-Agent),空=默认
qr_code_base64 = Column(Text, nullable=True) # 当前登录二维码的 base64 字符串
egress_public_ip = Column(String(64), nullable=True) # 指定公网出口;空=自动选择
egress_auto_attempts = Column(Integer, nullable=False, default=1) # 发送失败时最多串行尝试的出口数
qr_code_base64 = Column(Text().with_variant(LONGTEXT(), "mysql"), nullable=True) # 当前登录二维码的 base64 字符串
error_message = Column(Text, nullable=True) # 错误信息
quota_disabled = Column(Boolean, default=False, index=True) # 额度不足被停用
created_at = Column(DateTime, default=datetime.utcnow)
@@ -204,10 +237,23 @@ class MessageLog(Base):
reply_content = Column(Text, nullable=True) # 回复的消息
status = Column(String(50), default="received") # received, replied, ignored, failed
error_message = Column(Text, nullable=True)
created_at = Column(DateTime, default=datetime.utcnow)
created_at = Column(DateTime, default=datetime.utcnow, index=True)
account = relationship("Account", back_populates="logs")
__table_args__ = (
Index("ix_message_logs_account_created_at", "account_id", "created_at"),
Index("ix_message_logs_status_account_id", "status", "account_id"),
)
@validates("message_content", "reply_content")
def _bound_message_content(self, _key, value):
return bound_message_log_content(value) if value is not None else None
@validates("error_message")
def _bound_error_content(self, _key, value):
return bound_error_log_content(value) if value is not None else None
class ReceivedMessageLog(Base):
"""接收消息原始日志:仅记录收到的消息,内容原样保存。"""
@@ -225,6 +271,14 @@ class ReceivedMessageLog(Base):
raw_content = Column(Text, nullable=False)
created_at = Column(DateTime, default=datetime.utcnow, index=True)
__table_args__ = (
Index(
"ix_received_message_logs_account_created_at",
"account_id",
"created_at",
),
)
class FollowWelcomeLog(Base):
"""关注欢迎语去重表:每个账号对每个新粉丝只发送一次欢迎语(重启后仍生效)。"""
@@ -255,6 +309,10 @@ class SystemLog(Base):
detail = Column(Text, nullable=True) # 详细原因
created_at = Column(DateTime, default=datetime.utcnow, index=True)
__table_args__ = (
Index("ix_system_logs_account_created_at", "account_id", "created_at"),
)
class PaymentOrder(Base):
"""账号额度购买订单。"""
+21 -3
View File
@@ -2,7 +2,9 @@ from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import PlainTextResponse
from sqlalchemy.ext.asyncio import AsyncSession
from auth.dependencies import get_current_user, require_admin
from auth.dependencies import get_current_user, require_permission
from auth.permissions import ORDERS_CREATE, ORDERS_READ, PAYMENTS_MANAGE
from auth.roles import has_permission
from auth.system_settings import load_settings
from models.database import get_db
from models.models import User
@@ -20,6 +22,18 @@ from .schemas import (
router = APIRouter(prefix="/api/payments", tags=["payments"])
def _require_orders_access(user: User) -> None:
if has_permission(user.role, ORDERS_READ) or has_permission(user.role, PAYMENTS_MANAGE):
return
raise HTTPException(status_code=403, detail="缺少权限:orders.read")
def _require_orders_create(user: User) -> None:
if has_permission(user.role, ORDERS_CREATE) or has_permission(user.role, PAYMENTS_MANAGE):
return
raise HTTPException(status_code=403, detail="缺少权限:orders.create")
@router.get("/config", response_model=PaymentConfigResponse)
async def get_payment_config(
db: AsyncSession = Depends(get_db),
@@ -35,6 +49,7 @@ async def create_payment_order(
db: AsyncSession = Depends(get_db),
user: User = Depends(get_current_user),
):
_require_orders_create(user)
order, demo_mode = await service.create_order(db, user, body.slots, body.channel)
return PaymentOrderResponse(**service.order_to_dict(order, demo_mode=demo_mode))
@@ -48,6 +63,7 @@ async def list_payment_orders(
db: AsyncSession = Depends(get_db),
user: User = Depends(get_current_user),
):
_require_orders_access(user)
if status and status not in service.ORDER_STATUSES:
raise HTTPException(status_code=400, detail="无效的订单状态")
data = await service.list_orders(
@@ -67,6 +83,7 @@ async def get_payment_order(
db: AsyncSession = Depends(get_db),
user: User = Depends(get_current_user),
):
_require_orders_access(user)
settings = await load_settings(db)
order = await service.get_user_order(db, user, order_no)
demo_mode = settings.payment_demo_mode and not settings.payment_channel_available(order.channel)
@@ -79,6 +96,7 @@ async def simulate_payment_order(
db: AsyncSession = Depends(get_db),
user: User = Depends(get_current_user),
):
_require_orders_create(user)
settings = await load_settings(db)
order = await service.simulate_pay(db, user, order_no)
return PaymentOrderResponse(**service.order_to_dict(order, demo_mode=settings.payment_demo_mode))
@@ -89,7 +107,7 @@ async def admin_update_payment_order_status(
order_no: str,
body: AdminUpdateOrderStatusRequest,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(PAYMENTS_MANAGE)),
):
order = await service.admin_update_order_status(db, order_no, body.status)
return PaymentOrderListItem(**order)
@@ -99,7 +117,7 @@ async def admin_update_payment_order_status(
async def admin_delete_payment_order(
order_no: str,
db: AsyncSession = Depends(get_db),
_: User = Depends(require_admin),
_: User = Depends(require_permission(PAYMENTS_MANAGE)),
):
await service.admin_delete_order(db, order_no)
return MessageResponse(message="订单已删除")
+8 -2
View File
@@ -12,7 +12,8 @@ from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from auth.account_quota import default_stop_worker, sync_user_account_quota
from auth.roles import is_admin
from auth.roles import has_global_scope, has_permission, is_admin
from auth.permissions import ORDERS_READ, PAYMENTS_MANAGE
from auth.system_settings import SystemSettingsData, load_settings
from models.models import PaymentOrder, User
from . import alipay, wechat
@@ -347,7 +348,12 @@ async def list_orders(
page_size = max(1, min(100, page_size))
filters = []
if not is_admin(current_user.role):
# Global order list for built-in admin, payments.manage, or data.scope_all.
if not (
is_admin(current_user.role)
or has_permission(current_user.role, PAYMENTS_MANAGE)
or has_global_scope(current_user.role)
):
filters.append(PaymentOrder.user_id == current_user.id)
if status:
filters.append(PaymentOrder.status == status)
+80 -24
View File
@@ -520,7 +520,12 @@ def _build_auth(cookie_data: str, user_agent: Optional[str] = None) -> tuple[Dou
session = DouyinImSession.from_storage_state(storage or {})
ua = resolve_user_agent(user_agent or session.user_agent or DEFAULT_USER_AGENT)
auth = DouyinAuth()
auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str)
auth.perepare_auth(
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=ua,
)
auth.user_agent = ua
auth.web_id = session.web_id or session.device_id or None
return auth, ua
@@ -556,6 +561,8 @@ def fetch_douyin_profile_detail_sync(
"total_favorited": None,
"favoriting_count": None,
"fetched": False,
# 抖音明确回「用户未登录」时置位:Cookie 还在,但服务端已判定登录失效。
"logged_out": False,
"message": "",
}
try:
@@ -565,16 +572,17 @@ def fetch_douyin_profile_detail_sync(
result["message"] = "Cookie 无效,无法解析登录凭证"
return result
uid = auth.get_uid()
if uid:
result["uid"] = str(uid)
headers = {
"User-Agent": ua,
"Referer": "https://www.douyin.com/",
"Accept": "application/json, text/plain, */*",
}
endpoints: list[tuple[str, dict[str, str]]] = [
# (url, params, 是否账号资料源)
# query/user 是「会话/设备」查询接口,不是资料接口:它返回的 id 是浏览器
# 设备注册号,user_uid 也与账号资料 UID 可以是两个不同的号(实测
# user_uid=938334054809296 而资料 UID=2609567359568155)。以前它被当成资料
# 源解析,_pick_str 会把 user_uid 当作 uid,第一个请求就把身份定死了。
endpoints: list[tuple[str, dict[str, str], bool]] = [
(
"https://www.douyin.com/aweme/v1/web/query/user/",
{
@@ -587,6 +595,7 @@ def fetch_douyin_profile_detail_sync(
"webid": generate_webid(auth, "https://www.douyin.com/"),
"msToken": auth.msToken or generate_msToken(),
},
False,
),
(
"https://creator.douyin.com/aweme/v1/creator/user/info/",
@@ -594,6 +603,7 @@ def fetch_douyin_profile_detail_sync(
"device_platform": "webapp",
"aid": "6383",
},
True,
),
(
"https://www.douyin.com/aweme/v1/web/user/profile/self/",
@@ -602,12 +612,14 @@ def fetch_douyin_profile_detail_sync(
"aid": "6383",
"channel": "channel_pc_web",
},
True,
),
]
proxies = _requests_proxies()
valid_profile_response = False
for url, base_params in endpoints:
query_user_uid = ""
for url, base_params, is_profile_source in endpoints:
try:
params = dict(base_params)
query = splice_url(params)
@@ -628,16 +640,32 @@ def fetch_douyin_profile_detail_sync(
status_code = data.get("status_code")
if status_code is not None:
try:
if int(status_code) != 0:
continue
status_code = int(status_code)
except (TypeError, ValueError):
continue
if status_code != 0:
# status_code=8「用户未登录」是抖音的权威答复:Cookie 还在,
# 但服务端已判定登录失效。以前只是 continue,于是同步“成功”
# 却什么都没更新,UI 仍显示「Cookie 有效」,账号资料、
# sec_user_id、私信收件箱全是空的,排查不到原因。
if status_code == 8 or "未登录" in str(
data.get("status_msg") or ""
):
result["logged_out"] = True
continue
# 会话/设备接口的 user_uid 只是最弱兜底:等所有资料接口都没给出
# UID 时才用,否则第一个请求就把身份定死,权威资料 UID 永远写不进来。
if data.get("user_uid") and not query_user_uid:
query_user_uid = str(data["user_uid"])
if not is_profile_source:
continue
basic = _extract_profile_from_payload(data)
stats = _extract_detail_from_payload(data)
payload_has_profile = bool(
data.get("user_uid")
or basic.get("uid")
basic.get("uid")
or basic.get("nickname")
or basic.get("avatar_url")
or stats.get("uid")
@@ -650,9 +678,6 @@ def fetch_douyin_profile_detail_sync(
continue
valid_profile_response = True
if data.get("user_uid") and not result["uid"]:
result["uid"] = str(data["user_uid"])
if basic.get("uid") and not result["uid"]:
result["uid"] = basic["uid"]
if basic.get("nickname") and not result["nickname"]:
@@ -672,11 +697,30 @@ def fetch_douyin_profile_detail_sync(
except Exception as exc:
logger.debug(f"profile detail fetch failed for {url}: {exc}")
# UID 优先级:资料接口 > query/user 的 user_uid > cookie/接口兜底。
# 以前 auth.get_uid() 在请求前就把 result["uid"] 占住,且后面所有赋值都带
# `not result["uid"]` 守卫,导致权威资料 UID 永远写不进来:账号卡片显示成
# 「用户<user_uid>」,按这个 UID 反查 sec_user_id 也必然查不到。
if not result["uid"] and query_user_uid:
result["uid"] = query_user_uid
if not result["uid"]:
fallback_uid = auth.get_uid()
if fallback_uid:
result["uid"] = str(fallback_uid)
if not result["fetched"] and valid_profile_response:
result["fetched"] = True
if not result["fetched"]:
result["message"] = result["message"] or "未能从抖音获取账号资料,请确认 Cookie 有效"
if result["logged_out"]:
result["message"] = (
"抖音返回「用户未登录」:Cookie 仍在但服务端登录态已失效,"
"请停止托管后重新扫码登录该账号。"
)
else:
result["message"] = (
result["message"] or "未能从抖音获取账号资料,请确认 Cookie 有效"
)
result["profile_response_valid"] = valid_profile_response
return result
@@ -955,7 +999,10 @@ async def sync_account_profile_to_db(
if detail.get("fetched"):
try:
await apply_douyin_profile(db, account, cookie_data)
# 复用刚拿到的 detail,不要再发一次请求:两次抓取会各自走一遍
# uid 兜底逻辑,任何一次抖动都会让「账号卡片」和「详细资料」写进
# 不同的 UID/昵称,出现同步成功但卡片没更新的现象。
await apply_profile_to_account(db, account, detail)
except Exception as exc:
logger.warning(f"apply douyin profile failed: {exc}")
@@ -1060,16 +1107,15 @@ async def _pick_unique_username(
return f"账号_{account.id}"
async def apply_douyin_profile(
async def apply_profile_to_account(
db: AsyncSession,
account: Account,
cookie_data: str,
) -> dict[str, str]:
"""抓取并写入账号资料,返回抓取结果"""
profile = await fetch_douyin_profile(cookie_data, account.user_agent)
uid = (profile.get("uid") or "").strip()
nickname = (profile.get("nickname") or "").strip()
avatar = (profile.get("avatar_url") or "").strip()
profile: dict[str, Any],
) -> dict[str, Any]:
"""把已抓取的资料写入 accounts 行(昵称/UID/头像)"""
uid = str(profile.get("uid") or "").strip()
nickname = str(profile.get("nickname") or "").strip()
avatar = str(profile.get("avatar_url") or "").strip()
if uid:
account.douyin_uid = uid
@@ -1079,3 +1125,13 @@ async def apply_douyin_profile(
account.username = await _pick_unique_username(db, account, nickname, uid)
return profile
async def apply_douyin_profile(
db: AsyncSession,
account: Account,
cookie_data: str,
) -> dict[str, str]:
"""抓取并写入账号资料,返回抓取结果。"""
profile = await fetch_douyin_profile(cookie_data, account.user_agent)
return await apply_profile_to_account(db, account, profile)
+88 -7
View File
@@ -18,11 +18,44 @@ StartHandler = Callable[[int], Awaitable[dict[str, Any]]]
JobToken = tuple[str, int]
class _StartPreparationTimeout(Exception):
"""Internal marker for the queue's own per-account deadline."""
DEFAULT_CONCURRENCY = 6
MAX_CONCURRENCY = 32
def _configured_concurrency() -> int:
"""Admission width for account preparation.
Every account spends most of its startup waiting: for the shared network
lane, for a WebSocket handshake, for signing work in a thread. Admitting
only two at a time therefore left the network lane idle and made a fleet of
several hundred accounts take tens of minutes. Actual outbound traffic is
still capped by the traffic controller, so a wider admission window fills
the existing lane instead of adding load.
"""
try:
return max(1, min(8, int(os.getenv("KEFU_BATCH_START_CONCURRENCY", "2"))))
return max(
1,
min(
MAX_CONCURRENCY,
int(os.getenv("KEFU_BATCH_START_CONCURRENCY", str(DEFAULT_CONCURRENCY))),
),
)
except (TypeError, ValueError):
return 2
return DEFAULT_CONCURRENCY
def _configured_timeout_seconds() -> float:
try:
value = float(os.getenv("KEFU_BATCH_START_TIMEOUT_SECONDS", "90"))
except (TypeError, ValueError):
return 90.0
if value <= 0:
return 0.0
return max(5.0, min(600.0, value))
def _utc_now() -> str:
@@ -55,28 +88,40 @@ class BatchStartQueue:
handler: StartHandler,
concurrency: int | None = None,
max_batches: int = 100,
timeout_seconds: float | None = None,
) -> None:
self._handler = handler
self.concurrency = max(1, int(concurrency or _configured_concurrency()))
self.max_batches = max(10, int(max_batches or 100))
self.timeout_seconds = (
_configured_timeout_seconds()
if timeout_seconds is None
else max(0.0, float(timeout_seconds or 0.0))
)
self._queue: asyncio.Queue[JobToken] = asyncio.Queue()
self._pending_jobs: dict[int, JobToken] = {}
self._active_tasks: dict[int, tuple[JobToken, asyncio.Task]] = {}
self._batches: dict[str, _BatchRecord] = {}
self._workers: list[asyncio.Task] = []
self._worker_sequence = 0
self._lock = asyncio.Lock()
self._stopping = False
async def _ensure_workers(self) -> None:
async with self._lock:
self._workers = [task for task in self._workers if not task.done()]
if self._workers or self._stopping:
if self._stopping:
return
for index in range(self.concurrency):
# Top up to the configured width instead of only starting from
# zero. A worker that died on an unexpected error used to shrink
# the queue permanently, so later batches crawled through a single
# remaining worker with no way to recover short of a restart.
while len(self._workers) < self.concurrency:
self._worker_sequence += 1
self._workers.append(
asyncio.create_task(
self._worker(index + 1),
name=f"account-batch-start-{index + 1}",
self._worker(self._worker_sequence),
name=f"account-batch-start-{self._worker_sequence}",
)
)
@@ -157,7 +202,21 @@ class BatchStartQueue:
)
self._active_tasks[account_id] = (job_token, handler_task)
result = await handler_task
if self.timeout_seconds > 0:
try:
result = await asyncio.wait_for(
handler_task,
timeout=self.timeout_seconds,
)
except asyncio.TimeoutError as exc:
# wait_for cancels its task only when this queue's
# deadline expires. Preserve a TimeoutError raised by
# the handler itself as its real account failure.
if handler_task.cancelled():
raise _StartPreparationTimeout from exc
raise
else:
result = await handler_task
async with self._lock:
record = self._batches.get(batch_id)
if record:
@@ -171,6 +230,28 @@ class BatchStartQueue:
elapsed_seconds=round(time.monotonic() - started_at, 3),
)
record.updated_at = _utc_now()
except _StartPreparationTimeout:
elapsed = round(time.monotonic() - started_at, 3)
logger.warning(
"Batch start timed out account=%s worker=%s after %.1fs",
account_id,
worker_number,
self.timeout_seconds,
)
async with self._lock:
record = self._batches.get(batch_id)
if record:
item = record.items[account_id]
if item.get("status") != "cancelled":
item.update(
status="failed",
message=(
f"启动准备超过 {self.timeout_seconds:g} 秒,"
"已跳过并继续处理后续账号"
),
elapsed_seconds=elapsed,
)
record.updated_at = _utc_now()
except asyncio.CancelledError:
async with self._lock:
record = self._batches.get(batch_id)
+90 -19
View File
@@ -5,13 +5,38 @@ import logging
from typing import Optional
from rpa_engine.douyin_im.auth import DouyinAuth
from rpa_engine.douyin_im.frontier import ensure_frontier_ws
from rpa_engine.douyin_im.http_client import DouyinImHttpClient
from rpa_engine.douyin_im.session import DouyinImSession
from utils.cookie_store import analyze_cookie
logger = logging.getLogger("credential")
CREDENTIAL_EGRESS_PUBLIC_IP_KEY = "credential_egress_public_ip"
def credential_egress_mismatch(
cookie_data: Optional[str],
selected_public_ip: str = "",
) -> bool:
"""Compare historical browser egress metadata for diagnostics only.
This is not an authentication check: a different or missing local marker
cannot prove that cookies are invalid. Callers must keep the credentials
and use normal validation instead of forcing a reset or browser login.
"""
if not cookie_data:
return False
try:
storage = json.loads(cookie_data)
except (TypeError, ValueError):
return False
if not isinstance(storage, dict):
return False
selected = str(selected_public_ip or "").strip()
if CREDENTIAL_EGRESS_PUBLIC_IP_KEY not in storage:
return bool(selected)
stored = str(storage.get(CREDENTIAL_EGRESS_PUBLIC_IP_KEY) or "").strip()
return stored != selected
def _should_reset_credentials(assessment: dict) -> bool:
"""凭证全面失效时需清空 Cookie/IM 数据并重新登录。"""
@@ -54,7 +79,19 @@ def build_im_session_from_storage(
session.keys_str = saved.keys_str
if saved.web_protect_str and not session.web_protect_str:
session.web_protect_str = saved.web_protect_str
if saved.my_uid and not session.my_uid:
# A UID verified from the account profile must win over collector
# guesses such as web_runtime_security_uid. Persisting this flag
# keeps API/manual-send builders on the same identity as hosting.
if saved.uid_verified and saved.my_uid:
session.my_uid = saved.my_uid
# device_id 必须与 my_uid 指向同一账号:protobuf/frontier 的
# device_id 优先取 session.device_id,凭证里残留的旧设备号
# (如 www 域 web_runtime_security_uid)会导致 device_id != my_uid
# -> 安全网关 decision=KICK。用已核验 UID 同步 device_id。
if str(session.device_id or "") != str(saved.my_uid):
session.device_id = str(saved.my_uid)
session.uid_verified = True
elif saved.my_uid and not session.my_uid:
session.my_uid = saved.my_uid
if saved.device_id and not session.device_id:
session.device_id = saved.device_id
@@ -131,41 +168,57 @@ async def build_cookie_credential_detail(
async def validate_im_session(
session: DouyinImSession,
_bypass_global_limit: bool = False,
*,
startup_priority: bool = False,
) -> tuple[bool, str]:
if not _bypass_global_limit:
from rpa_engine.douyin_im.traffic_control import get_traffic_controller
controller = get_traffic_controller()
async with controller.background_slot(0, "credential validation"):
return await validate_im_session(session, _bypass_global_limit=True)
# Startup validation must not sit behind hundreds of recurring
# conversation polls. It still shares the same global concurrency
# cap, so this changes ordering without increasing bandwidth usage.
async with controller.background_slot(
0,
"credential validation",
startup=startup_priority,
):
return await validate_im_session(
session,
_bypass_global_limit=True,
startup_priority=startup_priority,
)
if not session.can_direct_im():
if not has_im_session_token(session):
return False, "缺少 sessionid,无法直连 IM"
return False, "Cookie 不满足 IM 直连条件"
await asyncio.to_thread(ensure_frontier_ws, session)
# Frontier discovery belongs to the worker startup lifecycle. Running it
# here populated only this temporary assessment session, so a bulk start
# immediately repeated the same signing / query work for every account.
try:
auth = DouyinAuth.from_im_session(session)
# 优先用已持久化的 my_uid,避免每次都发起网络 query_my_uiduid_tt 是加密串,
# int() 解析必然失败而回退到网络请求;该请求偶发失败会误判为“未就绪”)。
uid = session.my_uid or auth.get_uid()
uid = session.my_uid
if not uid:
# get_uid() may fall back to a synchronous HTTP request with a
# multi-second timeout. Keep that work off FastAPI's event loop
# so a manual credential recheck cannot freeze account editing or
# unrelated API requests.
uid = await asyncio.to_thread(auth.get_uid)
if not uid:
return False, "服务端未认可当前 Cookie(无法获取用户 UID)"
if not auth.is_sign_ready():
return False, "缺少 IM 签名密钥(web_protect/keys),请用浏览器登录补全"
session.my_uid = int(uid)
async with DouyinImHttpClient(session) as http:
await http.get_unread_count()
# 若已缓存到会话票据,优先校验其是否仍新鲜(最理想)。
if session.conv_meta:
ok, reason = await http.verify_messaging_capability(auth, session.my_uid)
if ok:
return True, reason
# 没有缓存会话票据是首次登录的正常情况:会话 ticket 会在发送时即时
# 创建/获取(resolve_conversation_meta),因此只要 Cookie + sessionid +
# 签名密钥(web_protect/keys) + UID 齐全,就视为可 IM 直连托管,不必再开浏览器。
return True, "IM 凭证就绪(Cookie 与签名密钥齐全,可直连托管)"
# unread_count and ticket probes were previously issued here, but
# neither result changed the final decision: unread failures become
# zero and a stale/missing ticket is resolved lazily at send time.
# Keeping those probes doubled large-batch startup traffic without
# adding an authoritative validation signal.
return True, "IM 凭证就绪(Cookie 与签名密钥齐全,可直连托管)"
except Exception as e:
logger.warning(f"IM session validation failed: {e}")
return False, f"IM 运行时验证失败: {e}"
@@ -174,6 +227,9 @@ async def validate_im_session(
async def assess_account_credential(
cookie_data: Optional[str],
im_session_data: Optional[str] = None,
*,
startup_priority: bool = False,
egress_public_ip: str = "",
) -> dict:
cookie_info = analyze_cookie(cookie_data)
result = {
@@ -199,6 +255,18 @@ async def assess_account_credential(
return result
session = build_im_session_from_storage(storage, im_session_data)
selected_public_ip = str(egress_public_ip or "").strip()
if selected_public_ip:
try:
from rpa_engine.egress_channels import resolve_fixed_channel
route = await resolve_fixed_channel(selected_public_ip)
session.egress_public_ip = selected_public_ip
session.egress_source_ip = str(route.source_ip or "")
except Exception as exc:
result["message"] = f"指定公网通道 {selected_public_ip} 当前不可用:{exc}"
result["login_mode"] = "browser"
return result
result["has_sessionid"] = has_im_session_token(session)
if not cookie_info.get("cookie_valid"):
@@ -212,7 +280,10 @@ async def assess_account_credential(
result["should_reset"] = _should_reset_credentials(result)
return result
im_ok, im_reason = await validate_im_session(session)
im_ok, im_reason = await validate_im_session(
session,
startup_priority=startup_priority,
)
result["im_ready"] = im_ok
if im_ok:
result["can_skip_browser"] = True
+31 -6
View File
@@ -1,7 +1,7 @@
import base64
import json
import logging
import requests
from rpa_engine.egress_channels import source_bound_requests_session
from .dy_util import (
trans_cookies,
generate_msToken,
@@ -63,8 +63,16 @@ class DouyinAuth:
self.uid = None
self.msToken = None
self.web_id = None
self.source_ip = ""
self.user_agent = None
def perepare_auth(self, cookieStr: str, web_protect_: str = "", keys_: str = ""):
def perepare_auth(
self,
cookieStr: str,
web_protect_: str = "",
keys_: str = "",
user_agent: str = "",
):
self.cookie = trans_cookies(cookieStr)
self.cookie_str = cookieStr
self.msToken = self.cookie["msToken"] if "msToken" in self.cookie else generate_msToken()
@@ -88,6 +96,11 @@ class DouyinAuth:
except Exception as e:
logger.debug(f"keys parse failed: {e}")
if user_agent:
# 让签名上下文记住调用方 UAquery_my_uid / generate_webid 等后续
# 请求会复用它,避免退回硬编码 DEFAULT_USER_AGENT 造成 UA 不一致。
self.user_agent = user_agent
def is_sign_ready(self) -> bool:
return bool(
self.private_key
@@ -108,12 +121,15 @@ class DouyinAuth:
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=session.user_agent or DEFAULT_USER_AGENT,
)
auth.web_id = session.web_id or session.device_id or None
auth.user_agent = session.user_agent or DEFAULT_USER_AGENT
# device_id 是设备注册号(query/user 的 id),不是账号 UID
# my_uid 只作为最后兜底,由 resolve_proto_device_id 内部处理。
auth.device_id = resolve_proto_device_id(
session.device_id, session.web_id, session.my_uid
)
auth.source_ip = str(getattr(session, "egress_source_ip", "") or "")
# web_protect 缺 client_cert 时,才用 frontier 抓包证书兜底(不覆盖 ts_sign)
if not auth.client_cert and getattr(session, "sdk_cert", ""):
auth.client_cert = normalize_client_cert(session.sdk_cert)
@@ -138,9 +154,10 @@ class DouyinAuth:
return self.uid
def query_my_uid(self) -> int:
ua = self.user_agent or DEFAULT_USER_AGENT
url = 'https://www.douyin.com/aweme/v1/web/query/user/'
headers = {
"User-Agent": DEFAULT_USER_AGENT,
"User-Agent": ua,
"Referer": "https://www.douyin.com/",
"Accept": "application/json, text/plain, */*",
}
@@ -155,9 +172,17 @@ class DouyinAuth:
"msToken": self.msToken
}
query = splice_url(params)
abogus = generate_a_bogus(query, user_agent=DEFAULT_USER_AGENT)
abogus = generate_a_bogus(query, user_agent=ua)
params['a_bogus'] = abogus
resp = requests.get(url, params=params, headers=headers, cookies=self.cookie, verify=False, timeout=10)
with source_bound_requests_session(self.source_ip) as client:
resp = client.get(
url,
params=params,
headers=headers,
cookies=self.cookie,
verify=False,
timeout=10,
)
resp_json = resp.json()
return int(resp_json['user_uid'])
+23
View File
@@ -45,3 +45,26 @@ def normalize_conversation_id(conversation_id: str, my_uid: int) -> str:
if peer_uid and my_uid:
return build_conversation_id(my_uid, peer_uid)
return (conversation_id or "").strip()
def conversation_belongs_to(conversation_id: str, my_uid: int) -> bool:
"""判断单聊会话是否属于 my_uid 本人。
托管多个账号时一条属于别的账号的会话例如 frontier 长连接按设备号寻址
造成的跨账号推送一旦流进本账号的处理链路resolve_peer_uid 会把末段当成
对方normalize_conversation_id 再拼成 0:1:{本账号}:{别人的好友}于是
本账号就把消息发给了另一个账号的好友这里给出唯一的归属判据
无法判定时一律返回 True保守放行 my_uid群聊 UID 等形态本来就
不带参与方信息只有两个参与方都已知且都不是本账号时才判定为不属于本账号
"""
try:
uid = int(my_uid or 0)
except (TypeError, ValueError):
return True
if not uid:
return True
parts = parse_conversation_parts(conversation_id)
if not parts:
return True
return uid in parts
+9 -2
View File
@@ -123,17 +123,24 @@ def generate_fake_webid(random_length=19):
return random_str
def generate_webid(auth=None, url=""):
def generate_webid(auth=None, url="", user_agent=""):
# 优先用已采集到的 web_id(避免每次发送都发起一次阻塞的 HTTP 请求,导致事件循环卡顿)
cached = getattr(auth, "web_id", None) if auth is not None else None
if cached:
return str(cached)
if url == "":
url = "https://www.douyin.com/discover?modal_id=7376449060384935209"
# UA 优先级:显式参数 > auth.user_agentfrom_im_session / perepare_auth 已带)> 全局默认。
# 必须与 a_bogus 签名、其余请求头使用同一个 UA,否则服务端重算失配 -> 7911。
ua = (
user_agent
or (getattr(auth, "user_agent", "") if auth is not None else "")
or DEFAULT_USER_AGENT
)
try:
from .auth import DouyinAuth
headers = {
"User-Agent": DEFAULT_USER_AGENT,
"User-Agent": ua,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"upgrade-insecure-requests": "1"
@@ -83,7 +83,12 @@ def fetch_recent_followers_sync(
try:
auth = DouyinAuth()
auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str)
auth.perepare_auth(
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=session.user_agent or DEFAULT_USER_AGENT,
)
except Exception as exc:
logger.warning("fetch followers: build auth failed: %s", exc)
return []
+27 -2
View File
@@ -8,7 +8,7 @@ from urllib.parse import unquote
import requests
from .auth import DouyinAuth
from .dy_util import generate_a_bogus, generate_msToken, generate_webid, splice_url
from .dy_util import DEFAULT_USER_AGENT, generate_a_bogus, generate_msToken, generate_webid, splice_url
from .session import DouyinImSession, is_frontier_ws_url
logger = logging.getLogger("douyin_im.frontier")
@@ -42,6 +42,7 @@ def fetch_device_id(session: DouyinImSession) -> str:
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=session.user_agent or DEFAULT_USER_AGENT,
)
url = "https://www.douyin.com/aweme/v1/web/query/user"
headers = {
@@ -101,11 +102,16 @@ def resolve_frontier_device_id(session: DouyinImSession) -> str:
return ""
def _ws_device_id(url: str) -> str:
def ws_device_id(url: str) -> str:
"""frontier 推送的寻址键:设备号(不是账号 UID)。"""
m = re.search(r"[?&]device_id=([^&\s]+)", url or "")
return unquote(m.group(1)) if m else ""
# 兼容内部旧引用
_ws_device_id = ws_device_id
def _ws_device_matches_session(session: DouyinImSession, url: str) -> bool:
ws_dev = _ws_device_id(url)
if not ws_dev or not ws_dev.isdigit():
@@ -134,12 +140,31 @@ def ensure_frontier_ws(session: DouyinImSession) -> Optional[str]:
logger.info("Using captured real frontier WS URL (with sdk_cert)")
return url
# Only fpid=9 is Douyin private messaging. Other Frontier products (for
# example fpid=971 opened by the generic /chat page shell) also handshake
# successfully but never carry this account's IM push stream.
for url in session.ws_urls:
if (
is_frontier_ws_url(url)
and "zijieapi.com" in url
and "access_key=" in url
and re.search(r"[?&]fpid=9(?:&|$)", url)
and _ws_device_matches_session(session, url)
):
session.ws_urls = [url]
logger.info("Using captured browser frontier WebSocket URL")
return url
for url in session.ws_urls:
if is_frontier_ws_url(url) and _ws_token_looks_encoded(url):
session.ws_urls = [url]
logger.info("Using captured frontier WS URL")
return url
# frontier 按 device_id 寻址推送,它不是账号 UID:抖音 query/user 返回的
# id 才是本浏览器的设备注册号(my_uid 走 DouyinAuth,两者不能互换)。
# 用 my_uid 拼出来的地址握手同样成功,但订阅的是另一个地址,
# 于是长连接一直是「连上但收不到任何私信」。
device_id = resolve_frontier_device_id(session)
if not device_id:
session.ws_urls = []
File diff suppressed because it is too large Load Diff
+88 -67
View File
@@ -31,6 +31,8 @@ import zlib
from typing import Any
from urllib.parse import urlencode
from rpa_engine.egress_channels import source_bound_requests_session
logger = logging.getLogger("douyin_im.image_upload")
_LOCAL_URL_RE = re.compile(
@@ -269,10 +271,8 @@ def _decode_sts(sts_token: str) -> tuple[str, str]:
return "", ""
def _fetch_im_upload_sts(session) -> tuple[str, str, str, str]:
def _fetch_im_upload_sts(session, source_ip: str = "") -> tuple[str, str, str, str]:
"""返回 (access_key_id, secret_access_key, sts_token, space_name)。"""
import requests
from .auth import DouyinAuth
from .dy_util import (
DEFAULT_USER_AGENT,
@@ -283,8 +283,13 @@ def _fetch_im_upload_sts(session) -> tuple[str, str, str, str]:
)
auth = DouyinAuth()
auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str)
ua = session.user_agent or DEFAULT_USER_AGENT
auth.perepare_auth(
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=ua,
)
params = {
"device_platform": "webapp",
@@ -328,15 +333,16 @@ def _fetch_im_upload_sts(session) -> tuple[str, str, str, str]:
"Referer": "https://www.douyin.com/",
"Accept": "application/json, text/plain, */*",
}
resp = requests.get(
IM_UPLOAD_CONFIG_URL,
params=params,
headers=headers,
cookies=auth.cookie,
timeout=20,
verify=False,
proxies=_requests_proxies(),
)
with source_bound_requests_session(source_ip) as client:
resp = client.get(
IM_UPLOAD_CONFIG_URL,
params=params,
headers=headers,
cookies=auth.cookie,
timeout=20,
verify=False,
proxies=None if source_ip else _requests_proxies(),
)
data = _safe_json(resp)
if data.get("error"):
raise RuntimeError(f"获取 IM 上传配置失败:{data['error']}")
@@ -453,10 +459,8 @@ def _extract_apply_inner(data: dict[str, Any]) -> tuple[str, str, str, str]:
def _vod_apply_upload_inner(
ak: str, sk: str, token: str, space: str, file_size: int
ak: str, sk: str, token: str, space: str, file_size: int, source_ip: str = ""
) -> tuple[str, str, str, str]:
import requests
from .dy_util import DEFAULT_USER_AGENT
now = datetime.datetime.utcnow()
@@ -483,20 +487,21 @@ def _vod_apply_upload_inner(
secret_access_key=sk,
service=VOD_SERVICE,
)
resp = requests.get(
f"{VOD_HOST}?{qs}",
headers={
"accept": "*/*",
"authorization": authorization,
"user-agent": DEFAULT_USER_AGENT,
"x-amz-date": amz_date,
"x-amz-security-token": token,
"Referer": "https://www.douyin.com/",
},
timeout=30,
verify=False,
proxies=_requests_proxies(),
)
with source_bound_requests_session(source_ip) as client:
resp = client.get(
f"{VOD_HOST}?{qs}",
headers={
"accept": "*/*",
"authorization": authorization,
"user-agent": DEFAULT_USER_AGENT,
"x-amz-date": amz_date,
"x-amz-security-token": token,
"Referer": "https://www.douyin.com/",
},
timeout=30,
verify=False,
proxies=None if source_ip else _requests_proxies(),
)
data = _safe_json(resp)
if data.get("error"):
raise RuntimeError(f"申请上传地址失败:{data['error']}")
@@ -512,10 +517,14 @@ def _vod_apply_upload_inner(
# ---------------------------------------------------------------------------
def _vod_upload_binary(
host: str, store_uri: str, jwt_auth: str, user_id: str, raw: bytes, session=None
host: str,
store_uri: str,
jwt_auth: str,
user_id: str,
raw: bytes,
session=None,
source_ip: str = "",
) -> None:
import requests
from .dy_util import DEFAULT_USER_AGENT
crc32 = format(zlib.crc32(raw) & 0xFFFFFFFF, "08x")
@@ -530,14 +539,15 @@ def _vod_upload_binary(
}
if user_id:
headers["X-Storage-U"] = str(user_id)
resp = requests.post(
url,
headers=headers,
data=raw,
timeout=60,
verify=False,
proxies=_requests_proxies(),
)
with source_bound_requests_session(source_ip) as client:
resp = client.post(
url,
headers=headers,
data=raw,
timeout=60,
verify=False,
proxies=None if source_ip else _requests_proxies(),
)
data = _safe_json(resp)
if data.get("error"):
raise RuntimeError(f"上传图片数据失败:{data['error']}")
@@ -550,10 +560,8 @@ def _vod_upload_binary(
# ---------------------------------------------------------------------------
def _vod_commit_upload_inner(
ak: str, sk: str, token: str, space: str, session_key: str
ak: str, sk: str, token: str, space: str, session_key: str, source_ip: str = ""
) -> dict[str, Any]:
import requests
from .dy_util import DEFAULT_USER_AGENT
now = datetime.datetime.utcnow()
@@ -580,23 +588,24 @@ def _vod_commit_upload_inner(
signed_headers=signed_headers,
service=VOD_SERVICE,
)
resp = requests.post(
f"{VOD_HOST}?{qs}",
data=body,
headers={
"accept": "*/*",
"authorization": authorization,
"content-type": "application/json",
"user-agent": DEFAULT_USER_AGENT,
"x-amz-content-sha256": payload_hash,
"x-amz-date": amz_date,
"x-amz-security-token": token,
"Referer": "https://www.douyin.com/",
},
timeout=30,
verify=False,
proxies=_requests_proxies(),
)
with source_bound_requests_session(source_ip) as client:
resp = client.post(
f"{VOD_HOST}?{qs}",
data=body,
headers={
"accept": "*/*",
"authorization": authorization,
"content-type": "application/json",
"user-agent": DEFAULT_USER_AGENT,
"x-amz-content-sha256": payload_hash,
"x-amz-date": amz_date,
"x-amz-security-token": token,
"Referer": "https://www.douyin.com/",
},
timeout=30,
verify=False,
proxies=None if source_ip else _requests_proxies(),
)
data = _safe_json(resp)
if data.get("error"):
raise RuntimeError(f"确认上传失败:{data['error']}")
@@ -613,6 +622,7 @@ def upload_im_image(
*,
filename: str = "image.jpg",
content_type: str = "image/jpeg",
source_ip: str = "",
) -> dict[str, Any]:
"""上传图片到抖音 IM 私信图床(VOD/zhenzhen 空间)。
@@ -622,16 +632,16 @@ def upload_im_image(
if not raw:
return {"error": "图片为空"}
try:
ak, sk, token, space = _fetch_im_upload_sts(session)
ak, sk, token, space = _fetch_im_upload_sts(session, source_ip)
host, store_uri, jwt_auth, session_key = _vod_apply_upload_inner(
ak, sk, token, space, len(raw)
ak, sk, token, space, len(raw), source_ip
)
if not host or not store_uri or not jwt_auth:
return {"error": "申请上传地址失败:缺少 UploadHost/StoreUri/Auth"}
user_id = str(getattr(session, "my_uid", "") or "")
_vod_upload_binary(host, store_uri, jwt_auth, user_id, raw, session)
_vod_commit_upload_inner(ak, sk, token, space, session_key)
_vod_upload_binary(host, store_uri, jwt_auth, user_id, raw, session, source_ip)
_vod_commit_upload_inner(ak, sk, token, space, session_key, source_ip)
uri = store_uri.lstrip("/")
out: dict[str, Any] = {"uri": uri, "md5": hashlib.md5(raw).hexdigest()}
@@ -650,7 +660,12 @@ def upload_im_image(
return {"error": str(exc)}
def prepare_image_reply_spec(spec: dict[str, Any], session, upload_dir: str) -> tuple[dict[str, Any], str]:
def prepare_image_reply_spec(
spec: dict[str, Any],
session,
upload_dir: str,
source_ip: str = "",
) -> tuple[dict[str, Any], str]:
"""若图片仍是本地地址,则上传到抖音 CDN 并补全 uri。返回 (spec, error)。"""
if spec.get("type") != "image":
return spec, ""
@@ -705,7 +720,13 @@ def prepare_image_reply_spec(spec: dict[str, Any], session, upload_dir: str) ->
return spec, "图片地址必须是抖音 CDN 或本地上传后的地址,外部 URL 无法用于 IM 发送"
return spec, "缺少可上传的图片数据"
uploaded = upload_im_image(session, raw, filename=filename, content_type=content_type)
uploaded = upload_im_image(
session,
raw,
filename=filename,
content_type=content_type,
source_ip=source_ip,
)
if uploaded.get("error"):
return spec, uploaded["error"]
if not uploaded.get("uri"):
+40
View File
@@ -160,6 +160,7 @@ def analyze_send_response(raw: bytes) -> dict:
"raw_check_code": None,
"delivered_with_notice": False,
"status_reason": "",
"decision": "",
"message": "",
"error_desc": "",
"server_message_id": None,
@@ -169,6 +170,45 @@ def analyze_send_response(raw: bytes) -> dict:
if not raw:
info["summary"] = "空响应"
return info
# 风控/登录网关有时不返回 protobuf,而是直接返回 JSON,例如:
# {"decision":"KICK"}。若继续按 protobuf 解码,JSON 的首字节“{”会被
# 误读为 wire type 3,只留下 unsupported wire type 3 这种次生错误。
stripped = raw.lstrip()
if stripped.startswith(b"{"):
try:
payload = json.loads(stripped.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError):
payload = None
if isinstance(payload, dict):
decision = str(
payload.get("decision") or payload.get("decision_type") or ""
).strip()
info["decision"] = decision
info["status_code"] = payload.get("status_code")
info["raw_check_code"] = payload.get("raw_check_code")
info["message"] = str(payload.get("message") or "")
info["error_desc"] = str(
payload.get("error_desc") or payload.get("error") or ""
)
info["status_reason"] = str(
payload.get("tips") or payload.get("reason") or ""
)
summary_parts = ["JSON响应"]
if decision:
summary_parts.append(f"decision={decision}")
if info["status_code"] is not None:
summary_parts.append(f"status_code={info['status_code']}")
if info["raw_check_code"] is not None:
summary_parts.append(f"raw_check_code={info['raw_check_code']}")
if info["message"]:
summary_parts.append(f"message={info['message']}")
if info["error_desc"]:
summary_parts.append(f"error_desc={info['error_desc']}")
info["summary"] = " ".join(summary_parts)
# /message/send 的正常成功响应是 protobuf;独立 JSON 是网关级响应,
# 不能据此确认消息已经写入会话。
return info
try:
fields = decode_fields(raw)
except Exception as e:
+29 -14
View File
@@ -7,9 +7,8 @@ import logging
import time
from typing import Any, Optional
import requests
from rpa_engine.device_profiles import resolve_user_agent
from rpa_engine.egress_channels import resolve_fixed_channel, source_bound_requests_session
from .auth import DouyinAuth
from .conv_util import resolve_peer_uid
from .dy_util import (
@@ -74,9 +73,14 @@ def _requests_proxies() -> dict | None:
def _build_auth(session: DouyinImSession) -> tuple[DouyinAuth, str]:
auth = DouyinAuth()
auth.perepare_auth(session.cookie_header(), session.web_protect_str, session.keys_str)
ua = resolve_user_agent(session.user_agent or DEFAULT_USER_AGENT)
auth = DouyinAuth()
auth.perepare_auth(
session.cookie_header(),
session.web_protect_str,
session.keys_str,
user_agent=ua,
)
return auth, ua
@@ -99,6 +103,7 @@ def fetch_peer_profile_sync(
session: DouyinImSession,
peer_uid: int | str,
account_id: int = 0,
source_ip: str = "",
) -> dict[str, str]:
uid = str(peer_uid or "").strip()
if not uid.isdigit():
@@ -156,21 +161,22 @@ def fetch_peer_profile_sync(
"https://www.douyin.com/aweme/v1/web/im/user/info/",
]
proxies = _requests_proxies()
proxies = None if source_ip else _requests_proxies()
for url in endpoints:
try:
params = dict(base_params)
query = splice_url(params)
params["a_bogus"] = generate_a_bogus(query, user_agent=ua)
resp = requests.get(
url,
params=params,
headers=headers,
cookies=auth.cookie,
verify=False,
timeout=12,
proxies=proxies,
)
with source_bound_requests_session(source_ip) as client:
resp = client.get(
url,
params=params,
headers=headers,
cookies=auth.cookie,
verify=False,
timeout=12,
proxies=proxies,
)
data = resp.json()
extracted = _extract_profile_from_payload(data)
if extracted.get("uid") and not result["uid"]:
@@ -199,12 +205,21 @@ async def fetch_peer_profile(
from .traffic_control import get_traffic_controller
controller = get_traffic_controller()
source_ip = str(getattr(session, "egress_source_ip", "") or "").strip()
selected_public_ip = str(getattr(session, "egress_public_ip", "") or "").strip()
if selected_public_ip and not source_ip:
try:
route = await resolve_fixed_channel(selected_public_ip)
source_ip = str(route.source_ip or "")
except Exception as exc:
logger.debug("peer profile egress resolution failed: %s", exc)
async with controller.background_slot(account_id, "peer profile"):
return await asyncio.to_thread(
fetch_peer_profile_sync,
session,
peer_uid,
account_id,
source_ip,
)
@@ -59,6 +59,24 @@ class ProtoBuilder:
request.sdk_cert = normalize_client_cert(auth.client_cert or "")
return request
@staticmethod
def build_read_request(auth, cmd):
"""读接口(收件箱/会话消息)的 Request 信封。
Request.token 必须是 x_tt_token cookiebuild_normal_request 填的是
auth.ticketbd-ticket-guard 票据长度合法服务端照样回
status_code=0 "OK"但把调用方当成匿名用户body 恒为空收件箱
里没有消息完全无法区分实测同一请求只换 token
auth.ticket -> 73 字节 0 x_tt_token -> 113KB 47
发送接口另有签名沿用 build_normal_request不在此处改动
"""
request = ProtoBuilder.build_normal_request(auth, cmd)
cookies = getattr(auth, "cookie", None) or {}
token = str(cookies.get("x_tt_token") or "").strip()
if token:
request.token = token
return request
@staticmethod
def build_create_conversation_request(auth, toId, myId):
request = ProtoBuilder.build_normal_request(auth, 609)
+133 -7
View File
@@ -1,7 +1,11 @@
import gzip
import json
import logging
import logging.handlers
import os
import queue
import re
import threading
from typing import Any, Optional
from .message_content import (
@@ -21,7 +25,6 @@ from .message_content import (
logger = logging.getLogger("douyin_im.protocol")
import os
from datetime import datetime
@@ -42,6 +45,73 @@ def _is_control_payload(content_json: Any, msg_type: int = 0) -> bool:
return False
_WS_DEBUG_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "ws_media_debug.log")
_WS_DEBUG_WRITER_LOCK = threading.Lock()
_WS_DEBUG_LOGGER: Optional[logging.Logger] = None
def _bounded_env_int(name: str, default: int, minimum: int, maximum: int) -> int:
try:
value = int(os.getenv(name, str(default)) or default)
except (TypeError, ValueError):
value = default
return max(minimum, min(maximum, value))
class _DroppingQueueHandler(logging.handlers.QueueHandler):
"""Never let optional diagnostics block the IM event loop."""
def enqueue(self, record) -> None:
try:
self.queue.put_nowait(record)
except queue.Full:
# Debug output is intentionally lossy under pressure. Receiving
# and replying to messages must always take precedence.
return
def _get_ws_debug_logger() -> logging.Logger:
global _WS_DEBUG_LOGGER
if _WS_DEBUG_LOGGER is not None:
return _WS_DEBUG_LOGGER
with _WS_DEBUG_WRITER_LOCK:
if _WS_DEBUG_LOGGER is not None:
return _WS_DEBUG_LOGGER
max_bytes = _bounded_env_int(
"KEFU_WS_DEBUG_MAX_BYTES", 10 * 1024 * 1024, 1024 * 1024, 100 * 1024 * 1024
)
backup_count = _bounded_env_int(
"KEFU_WS_DEBUG_BACKUP_COUNT", 2, 1, 10
)
queue_size = _bounded_env_int(
"KEFU_WS_DEBUG_QUEUE_SIZE", 1000, 100, 10000
)
records: queue.Queue = queue.Queue(maxsize=queue_size)
rotating = logging.handlers.RotatingFileHandler(
_WS_DEBUG_PATH,
maxBytes=max_bytes,
backupCount=backup_count,
encoding="utf-8",
delay=True,
)
rotating.setFormatter(logging.Formatter("%(message)s"))
listener = logging.handlers.QueueListener(
records,
rotating,
respect_handler_level=True,
)
listener.start()
debug_logger = logging.getLogger("douyin_im.ws_raw_debug")
debug_logger.handlers.clear()
debug_logger.addHandler(_DroppingQueueHandler(records))
debug_logger.setLevel(logging.INFO)
debug_logger.propagate = False
# Keep strong references for the lifetime of the logger/listener.
debug_logger._kefu_queue_listener = listener # type: ignore[attr-defined]
debug_logger._kefu_rotating_handler = rotating # type: ignore[attr-defined]
_WS_DEBUG_LOGGER = debug_logger
return debug_logger
def _should_emit_ws_message(
@@ -84,8 +154,13 @@ def _dump_ws_message(msg_type: int, conversation_id: str, content_str: str, msg:
f"{datetime.now().isoformat()} type={msg_type} "
f"conv={conversation_id} content={content_str}{extra}\n"
)
with open(_WS_DEBUG_PATH, "a", encoding="utf-8") as fh:
fh.write(line)
record_limit = _bounded_env_int(
"KEFU_WS_DEBUG_RECORD_MAX_CHARS", 16384, 1024, 262144
)
if len(line) > record_limit:
marker = "...[单条调试记录过长,已截断]\n"
line = line[: max(0, record_limit - len(marker))] + marker
_get_ws_debug_logger().info(line.rstrip("\n"))
except Exception:
pass
@@ -129,9 +204,46 @@ def extract_json_objects(raw: bytes | str) -> list[dict]:
return results
def _looks_like_push_frame(frame) -> bool:
"""判断这段字节确实是 frontier 的 PushFrame 信封。
真实帧一定带 service/method frontier 自己的 headers/traceid随手一段
二进制偶尔也能被 protobuf 宽松解析成 PushFrame那种不算
"""
return bool(
frame.service
or frame.method
or frame.payloadType
or frame.payloadEncoding
or frame.logIdNew
or len(frame.headersList)
)
def _decode_push_frame_payload(frame) -> bytes:
"""取出 PushFrame 内层负载,按 payloadEncoding 解压。
frontier 会用 gzip 压缩 payload直接把压缩字节喂给 Response.ParseFromString
只会抛异常并被吞掉整条私信就此丢失
"""
body = bytes(frame.payload or b"")
if not body:
return b""
encoding = str(frame.payloadEncoding or "").lower()
if encoding in ("gzip", "gz"):
try:
return gzip.decompress(body)
except Exception as exc:
logger.warning("Failed to gunzip frontier frame payload: %s", exc)
return body
return body
def parse_ws_payload(raw: bytes | str) -> list[dict]:
"""解析 WebSocket 二进制帧,返回标准化消息 dict 列表"""
messages = []
frame_payload = b""
is_push_frame = False
# 尝试 Protobuf 解包
if isinstance(raw, bytes):
@@ -139,9 +251,14 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]:
from .static import Live_pb2, Response_pb2
frame = Live_pb2.PushFrame()
frame.ParseFromString(raw)
if frame.payloadType == 'pb':
is_push_frame = _looks_like_push_frame(frame)
frame_payload = _decode_push_frame_payload(frame)
# payloadType 不再作为判据:现网 frontier 帧会带 'pb'、'text/json'
# 或空值,之前只认 'pb' 会把其余帧整帧丢弃。真正的判据是解出来
# 有没有 new_message_notify;解不出就照旧走下面的 JSON/文本兜底。
if frame_payload:
response = Response_pb2.Response()
response.ParseFromString(frame.payload)
response.ParseFromString(frame_payload)
body = response.body
if body.HasField("new_message_notify"):
notify = body.new_message_notify
@@ -153,8 +270,6 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]:
content_str = msg.content
server_message_id = str(getattr(msg, "server_message_id", "") or "")
_dump_ws_message(msg_type, conversation_id, content_str, msg)
text_content = ""
media_msg: dict = {}
content_json: dict = {}
@@ -175,6 +290,12 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]:
)
return messages
# Raw diagnostics are optional and intentionally run
# only after control/status frames have been filtered.
# The writer itself is queued and rotating, so it can
# never block message parsing or grow without bound.
_dump_ws_message(msg_type, conversation_id, content_str, msg)
if _should_emit_ws_message(conversation_id, msg_type):
sender_uid = str(msg.sender)
if media_msg and (
@@ -227,6 +348,11 @@ def parse_ws_payload(raw: bytes | str) -> list[dict]:
if isinstance(raw, str):
payloads = [raw.encode("utf-8", errors="ignore")]
elif is_push_frame:
# 已确认是 frontier PushFrame:只解析它的内层负载。整帧字节里还有
# seqId / traceid / payloadType 等元数据,拿去做纯文本兜底会把每条
# 「连接建立」等控制帧误当成一条用户私信记录并触发一次自动回复。
payloads = [frame_payload] if frame_payload else []
else:
payloads = [raw]
# 尝试 gzip 解压(frontier 常见)
+16 -6
View File
@@ -87,20 +87,29 @@ class AccountReplyQueue:
details: Optional[dict[str, Any]] = None,
merge_key: str = "",
merge_keys: Optional[Iterable[str]] = None,
immediate_if_idle: bool = False,
) -> int:
"""Append one reply job and return its current 1-based queue position."""
"""Append one reply job and return its current 1-based queue position.
When ``immediate_if_idle`` is enabled, the first job in a completely
idle account queue reserves a zero-second slot. Jobs arriving behind
it still reserve the configured interval, so the normal per-account
pacing resumes from the second job onward.
"""
interval = max(0.0, float(delay_seconds or 0))
loop = asyncio.get_running_loop()
async with self._state_lock:
if not self._running or not self._task or self._task.done():
raise RuntimeError("reply queue is not running")
due_at = max(loop.time(), self._tail_due_at) + interval
queue_is_idle = self.pending_count == 0
slot_seconds = 0.0 if immediate_if_idle and queue_is_idle else interval
due_at = max(loop.time(), self._tail_due_at) + slot_seconds
self._tail_due_at = due_at
self._waiting.append(
_QueueItem(
job_id=uuid.uuid4().hex,
due_at=due_at,
slot_seconds=interval,
slot_seconds=slot_seconds,
callback=callback,
description=description,
queued_at=time.time(),
@@ -256,9 +265,10 @@ class AccountReplyQueue:
item = self._waiting.pop(selected_index)
shift_seconds = max(0.0, item.slot_seconds)
shifted_count = 0
for later in self._waiting[selected_index:]:
later.due_at -= shift_seconds
shifted_count += 1
if shift_seconds > 0:
for later in self._waiting[selected_index:]:
later.due_at -= shift_seconds
shifted_count += 1
item.due_at = asyncio.get_running_loop().time()
item.expedited = True
File diff suppressed because it is too large Load Diff
+71 -20
View File
@@ -14,9 +14,21 @@ def is_frontier_ws_url(url: str) -> bool:
真实抓包里 host 可能是 frontier-im.douyin.com也可能是
frontierNN-normal.zijieapi.com 这类内部别名二者都要认
"""
if not url or "token=" not in url:
if not url:
return False
return "frontier-im.douyin.com" in url or ("frontier" in url and "zijieapi.com" in url)
parsed = urlparse(url)
host = (parsed.hostname or "").lower()
query = parse_qs(parsed.query)
fpid = (query.get("fpid") or [""])[0]
legacy = host == "frontier-im.douyin.com" and "token" in query and fpid == "9"
browser_frontier = (
"frontier" in host
and host.endswith("zijieapi.com")
and "access_key" in query
and "device_id" in query
and fpid == "9"
)
return legacy or browser_frontier
@dataclass
@@ -34,12 +46,18 @@ class DouyinImSession:
keys_str: str = ""
web_protect_str: str = ""
my_uid: int = 0
# my_uid 是否已用 query/user 接口核验过(采集端推断的 my_uid 可能取错 tea_cache id
# my_uid 是否已由账号资料 UID 等可靠来源核验。query/user 的 user_uid
# 不是所有账号的 IM UID,不能据此覆盖已采集/已同步的 my_uid。
uid_verified: bool = False
conv_meta: dict = field(default_factory=dict)
# 方案 A:直接复用浏览器抓到的真实 frontier 连接凭证(绕开我们自己推导 token/access_key 不准的问题)
sdk_cert: str = "" # bd-ticket-guard 客户端证书(frontier sdk_cert / HTTP client-cert
frontier_ts_sign: str = "" # 抓包得到的新鲜 ts_sign(覆盖 web_protect 里可能已过期的)
# 账号级公网出口配置来自 accounts 表,不写回 im_session_data,避免网络配置
# 与登录凭证重复存储。egress_source_ip 是当前服务器探测出的本地绑定地址。
egress_public_ip: str = ""
egress_source_ip: str = ""
egress_auto_attempts: int = 1
@classmethod
def from_storage_state(cls, data: dict, extra: Optional[dict] = None) -> "DouyinImSession":
@@ -91,6 +109,14 @@ class DouyinImSession:
my_uid = _as_uid(extra.get("my_uid")) or _as_uid(data.get("my_uid"))
user_agent = str(extra.get("user_agent") or data.get("user_agent") or "").strip()
# 先整段扫描 localStorage,收集字段(避免遍历顺序导致取值不确定)。
# 关键背景:新版抖音 web 端 __tea_cache_tokens_6383 的 user_unique_id 实际存的是
# web_id(如 7678646545793812008),并非账号 UID;而 web_runtime_security_uid
# 才是账号真实 UID(如 2609567359568155)。混合登录态下若把 tea 的 user_unique_id
# 当 my_uid,会导致 device_id != my_uidIM 发送被安全网关 KICK。
ls_sec_uid = "" # web_runtime_security_uid(最可靠的账号 UID 来源)
ls_web_id = "" # 第一个 tea 条目的 web_id/user_unique_id
ls_tea_pairs = [] # [(user_unique_id, web_id), ...] 按出现顺序
if not device_id or not web_id or not keys_str or not web_protect_str or not my_uid:
for origin in data.get("origins", []):
for entry in origin.get("localStorage", []):
@@ -102,27 +128,42 @@ class DouyinImSession:
keys_str = value
if name == "security-sdk/s_sdk_sign_data_key/web_protect" and not web_protect_str:
web_protect_str = value
if "tea_cache_tokens" in name and not web_id:
if "tea_cache_tokens" in name:
try:
parsed = json.loads(value)
web_id = str(
parsed.get("web_id")
or parsed.get("user_unique_id")
or ""
)
except Exception:
pass
if name == "web_runtime_security_uid" and not device_id:
if str(value or "").isdigit():
device_id = value
if "tea_cache_tokens" in name and not my_uid:
try:
parsed = json.loads(value)
uid = parsed.get("user_unique_id")
if uid and str(uid).isdigit():
my_uid = int(uid)
if isinstance(parsed, dict):
wid = str(parsed.get("web_id") or "")
uid = str(parsed.get("user_unique_id") or "")
if not ls_web_id:
ls_web_id = wid or uid
ls_tea_pairs.append((uid, wid))
except Exception:
pass
if name == "web_runtime_security_uid":
v = str(value or "")
if v.isdigit() and not ls_sec_uid:
ls_sec_uid = v
# web_idextra 显式值 > localStorage tea
if not web_id:
web_id = ls_web_id
# my_uid 优先级:extra/顶层 > web_runtime_security_uid(真实账号 UID>
# tea 的 user_unique_id(仅当与自身 web_id 不同才可信,避免误取 web_id)
if not my_uid and ls_sec_uid:
my_uid = int(ls_sec_uid)
if not my_uid:
for uid, wid in ls_tea_pairs:
if uid.isdigit() and not (wid and uid == wid):
my_uid = int(uid)
break
# device_id 优先级:extra/cookies > web_runtime_security_uid(与账号 UID 绑定)
if not device_id:
if ls_sec_uid:
device_id = ls_sec_uid
elif my_uid:
device_id = str(my_uid)
if not my_uid:
for item in data.get("cookies", []):
@@ -139,6 +180,14 @@ class DouyinImSession:
elif not device_id and web_id:
device_id = web_id
# 最终一致性收敛:protobuf/frontier 的 device_id 优先取 session.device_id
# resolve_proto_device_id),若凭证里残留旧设备号(如 www 域
# web_runtime_security_uid),发送时 device_id != my_uid 会被安全网关
# 判为设备指纹异常 -> decision=KICK。my_uid 此时已是权威账号 UID,
# 不一致时以 my_uid 收敛 device_id。
if my_uid and device_id and str(device_id) != str(my_uid):
device_id = str(my_uid)
ws_urls = list(extra.get("ws_urls") or [])
# 方案 A:凭证采集工具可携带浏览器抓到的真实 frontier 连接(含 token/sdk_cert/ts_sign)。
@@ -188,6 +237,7 @@ class DouyinImSession:
"keys_str": self.keys_str,
"web_protect_str": self.web_protect_str,
"my_uid": self.my_uid,
"uid_verified": self.uid_verified,
"conv_meta": self.conv_meta,
"sdk_cert": self.sdk_cert,
"frontier_ts_sign": self.frontier_ts_sign,
@@ -207,6 +257,7 @@ class DouyinImSession:
keys_str=str(data.get("keys_str") or ""),
web_protect_str=str(data.get("web_protect_str") or ""),
my_uid=int(data.get("my_uid") or 0),
uid_verified=bool(data.get("uid_verified", False)),
conv_meta=dict(data.get("conv_meta") or {}),
sdk_cert=str(data.get("sdk_cert") or ""),
frontier_ts_sign=str(data.get("frontier_ts_sign") or ""),
@@ -393,8 +393,27 @@ class TrafficController:
1.0,
)
)
self._background = asyncio.Semaphore(
_env_int("KEFU_BACKGROUND_NETWORK_CONCURRENCY", 2)
background_capacity = _env_int("KEFU_BACKGROUND_NETWORK_CONCURRENCY", 4)
self._background = asyncio.Semaphore(background_capacity)
# Recurring polls can create hundreds of waiters when many accounts
# are online. Admit at most one normal waiter to the semaphore at a
# time so startup validation can join near the front instead of being
# buried behind the entire polling backlog. The shared semaphore is
# still the single bandwidth cap; startup work does not add extra
# network concurrency.
self._background_normal_admission = asyncio.Lock()
self._background_startup_clear = asyncio.Event()
self._background_startup_clear.set()
# Starting several hundred accounts keeps startup waiters queued for
# many minutes on end. Leave one slot of the shared lane for recurring
# work so hosted accounts keep receiving messages during a bulk start
# instead of going silent until the last account is up.
self._background_startup = asyncio.Semaphore(
max(1, background_capacity - 1)
)
self._background_normal_max_defer = _env_float(
"KEFU_BACKGROUND_NORMAL_MAX_DEFER_SECONDS",
5.0,
)
self._browser = asyncio.Semaphore(
_env_int("KEFU_BROWSER_START_CONCURRENCY", 1)
@@ -410,12 +429,20 @@ class TrafficController:
)
self.background_waiting = 0
self.background_active = 0
self.background_startup_waiting = 0
self.background_startup_active = 0
self.browser_waiting = 0
self.browser_active = 0
self.media_proxy_active = 0
@asynccontextmanager
async def background_slot(self, account_id: int = 0, description: str = "request"):
async def background_slot(
self,
account_id: int = 0,
description: str = "request",
*,
startup: bool = False,
):
current_task = asyncio.current_task()
owner_task, depth = self._background_owner.get()
if owner_task is current_task and depth > 0:
@@ -429,12 +456,52 @@ class TrafficController:
started = asyncio.get_running_loop().time()
self.background_waiting += 1
try:
await self._background.acquire()
if startup:
self.background_startup_waiting += 1
self._background_startup_clear.clear()
startup_reservation = False
try:
await self._background_startup.acquire()
startup_reservation = True
await self._background.acquire()
except BaseException:
if startup_reservation:
self._background_startup.release()
raise
finally:
self.background_startup_waiting -= 1
if self.background_startup_waiting == 0:
self._background_startup_clear.set()
else:
# Only one recurring/background request may wait directly on
# the shared semaphore. A later startup request therefore
# has at most one normal request ahead of it, not hundreds.
async with self._background_normal_admission:
# Yield to pending startup work, but only while a startup
# waiter could still claim a slot, and never for longer
# than the deferral budget. A batch of several hundred
# accounts otherwise keeps startup waiters pending for the
# whole run, which stalled every recurring poll behind it.
if (
self._background_normal_max_defer > 0
and not self._background_startup_clear.is_set()
and not self._background_startup.locked()
):
try:
await asyncio.wait_for(
self._background_startup_clear.wait(),
timeout=self._background_normal_max_defer,
)
except asyncio.TimeoutError:
pass
await self._background.acquire()
except BaseException:
self.background_waiting -= 1
raise
self.background_waiting -= 1
self.background_active += 1
if startup:
self.background_startup_active += 1
token = self._background_owner.set((current_task, 1))
waited = asyncio.get_running_loop().time() - started
if waited >= 1.0:
@@ -448,6 +515,9 @@ class TrafficController:
yield
finally:
self._background_owner.reset(token)
if startup:
self.background_startup_active -= 1
self._background_startup.release()
self.background_active -= 1
self._background.release()
@@ -492,6 +562,8 @@ class TrafficController:
"background": {
"active": self.background_active,
"waiting": self.background_waiting,
"startup_active": self.background_startup_active,
"startup_waiting": self.background_startup_waiting,
},
"browser": {
"active": self.browser_active,
+523 -138
View File
@@ -1,9 +1,11 @@
import asyncio
import gzip
import logging
import threading
import os
import weakref
from typing import Awaitable, Callable, Optional
from websocket import WebSocketApp
from websockets.legacy.client import WebSocketClientProtocol, connect as websocket_connect
from utils import system_logger
from .protocol import parse_ws_payload
@@ -14,8 +16,132 @@ logger = logging.getLogger("douyin_im.ws")
MessageHandler = Callable[[dict], Awaitable[None]]
def _safe_frame_metadata(payload: bytes) -> str:
"""Return non-content protobuf metadata for early connection diagnostics."""
try:
from .static import Live_pb2, Response_pb2
frame = Live_pb2.PushFrame()
frame.ParseFromString(payload)
body = bytes(frame.payload)
if str(frame.payloadEncoding or "").lower() == "gzip":
body = gzip.decompress(body)
response = Response_pb2.Response()
response.ParseFromString(body)
fields = [field.name for field, _ in response.body.ListFields()]
message = str(response.message or response.error_desc or "")[:80]
return (
f"service={frame.service} method={frame.method} "
f"encoding={frame.payloadEncoding or 'none'} "
f"type={frame.payloadType or 'none'} payload_bytes={len(body)} "
f"cmd={response.cmd} body={','.join(fields) or 'none'} "
f"status={message or 'ok'}"
)
except Exception as exc:
return f"metadata_unavailable={type(exc).__name__}"
# Both stages are finite. The transport queue gives the receive coroutine a
# small amount of breathing room, while the application queue decouples Pong /
# frame reads from potentially slow database and reply work. Once both fill,
# backpressure intentionally reaches TCP instead of allocating more tasks.
_TRANSPORT_MAX_QUEUE = 4
_APPLICATION_QUEUE_SIZE = 8
_INCOMING_MAX_SIZE = 2**20
_STABLE_CONNECTION_SECONDS = 60.0
_MAX_RECONNECT_BASE_SECONDS = 60.0
_CLOSE_GRACE_SECONDS = 2.0
_PING_TIMEOUT_SECONDS = 120.0
_HANDLER_CONCURRENCY_ENV = "KEFU_WS_HANDLER_CONCURRENCY"
_SYSTEM_LOG_THROTTLE_ENV = "KEFU_WS_SYSTEM_LOG_THROTTLE_SECONDS"
def _env_int_clamped(name: str, default: int, minimum: int, maximum: int) -> int:
try:
value = int(os.getenv(name, str(default)) or default)
except (TypeError, ValueError):
value = default
return max(minimum, min(maximum, value))
def _env_float_clamped(
name: str,
default: float,
minimum: float,
maximum: float,
) -> float:
try:
value = float(os.getenv(name, str(default)) or default)
except (TypeError, ValueError):
value = default
return max(minimum, min(maximum, value))
def _handler_concurrency_limit() -> int:
# SQLite serializes writes. Eight allows unrelated parsing / reads to
# progress without letting an accidental value such as 500 recreate the
# original event-loop and database stampede.
return _env_int_clamped(_HANDLER_CONCURRENCY_ENV, 8, 1, 32)
def _system_log_throttle_seconds() -> float:
return _env_float_clamped(_SYSTEM_LOG_THROTTLE_ENV, 300.0, 10.0, 3600.0)
class _LoopWsState:
"""Shared limits for all WS clients owned by one asyncio event loop."""
def __init__(self) -> None:
self.handler_slots = asyncio.Semaphore(_handler_concurrency_limit())
self.system_log_last_at: dict[tuple[int, str], float] = {}
# asyncio synchronization primitives belong to their creating event loop.
# Keeping one weakly-keyed state per loop gives production a process-wide
# limit while keeping isolated test loops and uncommon threaded loops safe.
_LOOP_STATES: "weakref.WeakKeyDictionary[asyncio.AbstractEventLoop, _LoopWsState]" = (
weakref.WeakKeyDictionary()
)
# frontier 按 device_id 寻址推送:两个托管账号共用同一个设备号时,两条长连接会
# 订阅到同一个地址并互相收到对方的私信。真正的拦截在 service 的会话归属校验里,
# 这里只负责把「为什么会串号」明确告诉用户。持弱引用,账号停管后自动失效。
_FRONTIER_DEVICE_OWNERS: "dict[str, weakref.ref[DouyinImWsClient]]" = {}
def _get_loop_state() -> _LoopWsState:
loop = asyncio.get_running_loop()
state = _LOOP_STATES.get(loop)
if state is None:
state = _LoopWsState()
_LOOP_STATES[loop] = state
return state
def _reconnect_delay(account_id: int | None, retry: int) -> float:
"""Return exponential backoff with stable, account-specific full jitter.
A connection that is accepted and immediately closed is still a failed
attempt. The old client reset its retry counter whenever ``run_forever``
returned normally, which kept those accounts reconnecting every 2-7s.
This delay reaches a 60-90s range after repeated short-lived connections.
"""
attempt = max(1, int(retry or 1))
base = min(_MAX_RECONNECT_BASE_SECONDS, float(2 ** min(attempt, 6)))
# Spread later retries across half of the base interval. Keep at least a
# five-second spread on early retries so a shared outage doesn't reconnect
# every account in the same instant.
spread = max(5.0, base / 2.0)
# Keep one account's fraction stable across attempts. This preserves the
# exponential ordering while different accounts remain spread apart.
seed = (int(account_id or 0) * 2654435761) & 0xFFFFFFFF
fraction = (seed % 10000) / 10000.0
return base + (spread * fraction)
class DouyinImWsClient:
"""直连 frontier-im WebSocketwebsocket-client,与 DouYin_Spider 一致)"""
"""Async frontier-im WebSocket client with bounded message backpressure."""
def __init__(
self,
@@ -27,12 +153,20 @@ class DouyinImWsClient:
self.on_message = on_message
self.account_id = account_id
self._running = False
self.connected = False
self._task: Optional[asyncio.Task] = None
self._loop: Optional[asyncio.AbstractEventLoop] = None
self._ws_app: Optional[WebSocketApp] = None
self._ws_lock = threading.Lock()
self._connection: Optional[WebSocketClientProtocol] = None
self._last_connection_lifetime = 0.0
self._message_queue: Optional[asyncio.Queue[dict]] = None
self._dispatcher_task: Optional[asyncio.Task] = None
self._received_frame_count = 0
self._heartbeat_ack_logged = False
self._frontier_device_id = ""
self._blocked_device_owner_id: Optional[int] = None
async def start(self):
if self._task and not self._task.done():
return
url = self.session.frontier_ws_url()
if not url:
logger.warning("No frontier WebSocket URL captured; WS listener disabled")
@@ -45,186 +179,437 @@ class DouyinImWsClient:
)
return
self._running = True
self._loop = asyncio.get_running_loop()
self._task = asyncio.create_task(self._run_loop(url))
self._ensure_dispatcher()
self._task = asyncio.create_task(
self._run_loop(url),
name=f"im-ws-{self.account_id or 'na'}",
)
async def stop(self):
self._running = False
with self._ws_lock:
if self._ws_app:
self.connected = False
connection = self._connection
if connection is not None:
try:
await asyncio.wait_for(
connection.close(code=1000, reason="client stopping"),
timeout=_CLOSE_GRACE_SECONDS,
)
except asyncio.TimeoutError:
# Shutdown iterates over every hosted account. One broken
# peer must not consume close_timeout repeatedly and turn a
# 500-account shutdown into a many-minute operation.
logger.debug("Timed out closing IM WebSocket; aborting transport")
try:
self._ws_app.close()
connection.fail_connection()
except Exception:
pass
self._ws_app = None
if self._task:
self._task.cancel()
except Exception:
logger.debug("Failed to close IM WebSocket cleanly", exc_info=True)
task = self._task
if task and task is not asyncio.current_task() and not task.done():
task.cancel()
try:
await self._task
await task
except asyncio.CancelledError:
pass
if self._task is task:
self._task = None
self._connection = None
self._release_frontier_device()
await self._stop_dispatcher()
async def _run_ws_thread(self, url: str):
"""在独立守护线程中跑 run_forever,直到连接断开/关闭。
def _record_connection_system_event(
self,
event_key: str,
message: str,
*,
detail: str,
level: str,
) -> bool:
"""Persist at most one repeated lifecycle event per account/window."""
不能用共享默认线程池run_in_executor(None)/asyncio.to_thread
WS 长连接会永久占用一个池线程账号数超过池大小默认 64
所有账号的签名/轮询任务被饿死表现为启动几十个账号后全部卡死超时
"""
loop = asyncio.get_running_loop()
done = asyncio.Event()
error: list[BaseException] = []
def _runner():
try:
self._connect_sync(url)
except BaseException as e:
error.append(e)
finally:
try:
loop.call_soon_threadsafe(done.set)
except RuntimeError:
pass # 事件循环已关闭
thread = threading.Thread(
target=_runner,
name=f"im-ws-{self.account_id or 'na'}",
daemon=True,
state = _get_loop_state()
key = (int(self.account_id or 0), event_key)
now = loop.time()
last_at = state.system_log_last_at.get(key)
if last_at is not None and now - last_at < _system_log_throttle_seconds():
return False
state.system_log_last_at[key] = now
system_logger.record(
message,
detail=detail,
level=level,
category="ws",
account_id=self.account_id,
)
thread.start()
try:
await done.wait()
except asyncio.CancelledError:
# stop() 会 close ws_app 使 run_forever 退出,线程随之结束
raise
if error:
raise error[0]
return True
async def _run_loop(self, url: str):
retry = 0
while self._running:
from .frontier import ensure_frontier_ws
from .traffic_control import get_traffic_controller
def _reset_connection_system_log_throttle(self) -> None:
loop = asyncio.get_running_loop()
state = _LOOP_STATES.get(loop)
if state is None:
return
account_key = int(self.account_id or 0)
state.system_log_last_at.pop((account_key, "connected"), None)
state.system_log_last_at.pop((account_key, "retry"), None)
state.system_log_last_at.pop((account_key, "device_taken"), None)
# ensure_frontier_ws 可能触发签名/HTTP(阻塞),放线程池避免卡事件循环
controller = get_traffic_controller()
async with controller.background_slot(self.account_id or 0, "websocket prepare"):
await asyncio.to_thread(ensure_frontier_ws, self.session)
connect_url = self.session.frontier_ws_url() or url
def _ensure_dispatcher(self) -> None:
if self._dispatcher_task and not self._dispatcher_task.done():
return
if self._message_queue is None:
self._message_queue = asyncio.Queue(maxsize=_APPLICATION_QUEUE_SIZE)
self._dispatcher_task = asyncio.create_task(
self._dispatch_loop(),
name=f"im-ws-dispatch-{self.account_id or 'na'}",
)
async def _stop_dispatcher(self) -> None:
task = self._dispatcher_task
self._dispatcher_task = None
if task and task is not asyncio.current_task() and not task.done():
task.cancel()
try:
logger.info(f"Connecting IM WebSocket: {connect_url[:100]}...")
await self._run_ws_thread(connect_url)
retry = 0
await task
except asyncio.CancelledError:
pass
queue = self._message_queue
self._message_queue = None
if queue is not None:
# Dropped messages must decrement the unfinished counter so tests,
# diagnostics, and a later restart can never hang on queue.join().
while True:
try:
queue.get_nowait()
except asyncio.QueueEmpty:
break
else:
queue.task_done()
async def _prepare_url(self, fallback_url: str) -> str:
from .frontier import ensure_frontier_ws
from .traffic_control import get_traffic_controller
# Frontier discovery can perform synchronous signing / HTTP work. It
# remains in the shared background lane and off the FastAPI event loop.
controller = get_traffic_controller()
async with controller.background_slot(
self.account_id or 0,
"websocket prepare",
):
await asyncio.to_thread(ensure_frontier_ws, self.session)
return self.session.frontier_ws_url() or fallback_url
async def _run_loop(self, initial_url: str):
retry = 0
first_attempt = True
while self._running:
self._last_connection_lifetime = 0.0
try:
# Startup validation already prepared the captured URL. Avoid
# repeating signing / frontier discovery for all 500 accounts
# on their first connect; refresh only after a disconnect.
if first_attempt and initial_url:
connect_url = initial_url
else:
connect_url = await self._prepare_url(initial_url)
first_attempt = False
if not connect_url:
raise RuntimeError("frontier WebSocket URL is unavailable")
if self._claim_frontier_device(connect_url):
logger.info("Connecting IM WebSocket: %s...", connect_url[:100])
await self._run_connection(connect_url)
else:
# 设备号已被另一个在跑的账号占用:绝不并连同一个推送地址,
# 本账号本轮退回 HTTP 轮询兜底(connected 保持 False
# service 会自动切到更快的会话对账节奏),并在退避后重试,
# 等占用方停管时自动接管。
self._report_frontier_device_taken(connect_url)
except asyncio.CancelledError:
break
except Exception as e:
logger.warning(f"IM WebSocket error: {e}")
system_logger.record(
"实时接收连接异常",
detail=f"建立 frontier WebSocket 失败:{e}",
level="error",
category="ws",
account_id=self.account_id,
)
except Exception as exc:
if self._running:
logger.warning("IM WebSocket error: %s", exc)
self._record_connection_system_event(
"retry",
"实时接收连接异常",
detail=f"建立 frontier WebSocket 失败:{exc}",
level="error",
)
# Only a genuinely stable connection earns a retry reset. A
# successful handshake followed by an immediate normal close must
# continue exponential backoff rather than reconnect forever at
# the first delay.
if self._last_connection_lifetime >= _STABLE_CONNECTION_SECONDS:
retry = 0
if not self._running:
break
retry += 1
# Stable per-account jitter prevents every hosted account from
# reconnecting in the same second after a shared network outage.
jitter = ((int(self.account_id or 0) * 2654435761) % 5000) / 1000.0
wait = min(30.0, 2.0 * retry) + jitter
logger.info(f"IM WebSocket reconnect in {wait:.1f}s...")
system_logger.record(
wait = _reconnect_delay(self.account_id, retry)
logger.info("IM WebSocket reconnect in %.1fs...", wait)
self._record_connection_system_event(
"retry",
f"实时接收断开,{wait:.1f}s 后重连",
detail="frontier WebSocket 连接已断开,正在自动重连。",
level="warning",
category="ws",
account_id=self.account_id,
)
await asyncio.sleep(wait)
try:
await asyncio.sleep(wait)
except asyncio.CancelledError:
break
def _connect_sync(self, url: str):
if not self._loop:
return
self._release_frontier_device()
def on_open(_ws):
logger.info("IM WebSocket connected")
system_logger.record(
"实时接收通道已连接",
detail="frontier WebSocket 已建立,可实时接收私信。",
level="success",
category="ws",
account_id=self.account_id,
)
def _frontier_device_owner(self, device_id: str) -> "Optional[DouyinImWsClient]":
"""当前仍活着的设备号占用方(run 循环任务还在跑才算数)。"""
reference = _FRONTIER_DEVICE_OWNERS.get(device_id)
owner = reference() if reference is not None else None
if owner is None or owner is self:
return None
task = owner._task
if not owner._running or task is None or task.done():
return None
return owner
def on_message(_ws, message):
asyncio.run_coroutine_threadsafe(self._dispatch(message), self._loop)
def _claim_frontier_device(self, url: str) -> bool:
"""独占本账号的 frontier 设备地址;已被别的账号占用时返回 False。
def on_error(_ws, error):
if self._running:
logger.warning(f"IM WebSocket error: {error}")
system_logger.record(
"实时接收通道报错",
detail=f"{error}",
level="error",
category="ws",
account_id=self.account_id,
)
frontier device_id 寻址推送两个账号共用同一个设备号时同时建连
会让两条连接互相收到对方的私信串号的根因且抖音也可能只保留最后
一条连接把先连上的那个账号踢成连着但收不到所以同一个设备地址
永远只允许一个账号建连另一个账号走 HTTP 轮询兜底
"""
from .frontier import ws_device_id
def on_close(_ws, code, msg):
logger.info(f"IM WebSocket closed: code={code}, msg={msg}")
if self._running:
system_logger.record(
"实时接收通道关闭",
detail=f"code={code}, msg={msg}",
level="warning",
category="ws",
account_id=self.account_id,
)
device_id = ws_device_id(url)
if not device_id:
# 判不出设备号(自建地址/异常格式)时不阻断连接,交给会话归属校验兜底。
return True
owner = self._frontier_device_owner(device_id)
if owner is not None and int(owner.account_id or 0) != int(self.account_id or 0):
self._blocked_device_owner_id = owner.account_id
return False
_FRONTIER_DEVICE_OWNERS[device_id] = weakref.ref(self)
self._frontier_device_id = device_id
self._blocked_device_owner_id = None
return True
headers = {
"User-Agent": self.session.user_agent,
"Pragma": "no-cache",
"Cache-Control": "no-cache",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"Sec-WebSocket-Protocol": "binary, base64, pbbp2",
"Sec-WebSocket-Extensions": "permessage-deflate; client_max_window_bits",
}
ws_app = WebSocketApp(
url,
header=headers,
cookie=self.session.cookie_header(),
on_open=on_open,
on_message=on_message,
on_error=on_error,
on_close=on_close,
def _report_frontier_device_taken(self, url: str) -> None:
from .frontier import ws_device_id
device_id = ws_device_id(url)
owner_id = self._blocked_device_owner_id
logger.error(
"Account %s cannot open frontier device_id %s: already held by "
"account %s; falling back to HTTP polling this round",
self.account_id,
device_id,
owner_id,
)
with self._ws_lock:
self._ws_app = ws_app
self._record_connection_system_event(
"device_taken",
"实时接收已让出:与另一个账号共用长连接设备号",
detail=(
f"本账号与账号 {owner_id} 的 frontier 设备号相同(device_id={device_id})。"
"同一个设备地址只允许一个账号建立长连接,否则两个账号会互相收到对方的"
"私信。本账号本轮不建连,改由 HTTP 会话轮询接收(有几十秒级延迟),"
"并在对方停止托管后自动接管。"
"根治办法:为每个账号在独立的浏览器配置/设备上重新采集凭证。"
),
level="error",
)
def _release_frontier_device(self) -> None:
device_id = self._frontier_device_id
self._frontier_device_id = ""
if not device_id:
return
reference = _FRONTIER_DEVICE_OWNERS.get(device_id)
if reference is not None and reference() is self:
_FRONTIER_DEVICE_OWNERS.pop(device_id, None)
def _connection_headers(self) -> list[tuple[str, str]]:
headers = [
("Pragma", "no-cache"),
("Cache-Control", "no-cache"),
("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8"),
]
cookie = self.session.cookie_header()
if cookie:
headers.append(("Cookie", cookie))
return headers
@staticmethod
def _uses_browser_frontier(url: str) -> bool:
return "zijieapi.com" in url and "access_key=" in url
async def _run_browser_heartbeat(self, websocket) -> None:
"""Mirror Frontier's browser SDK application-level ``hi`` heartbeat."""
while self._running:
await websocket.send("hi")
await asyncio.sleep(30)
async def _run_connection(self, url: str) -> None:
"""Open one connection and dispatch messages sequentially.
``max_queue`` bounds the library's receive buffer and ``_dispatch``
feeds one lifecycle-owned, bounded application queue. This receive
loop therefore remains responsive to control frames during ordinary
database stalls without creating one task per incoming frame.
"""
loop = asyncio.get_running_loop()
connected_at: float | None = None
connection: Optional[WebSocketClientProtocol] = None
heartbeat_task: Optional[asyncio.Task] = None
browser_frontier = self._uses_browser_frontier(url)
source_ip = str(getattr(self.session, "egress_source_ip", "") or "").strip()
connect_kwargs = {"local_addr": (source_ip, 0)} if source_ip else {}
try:
ws_app.run_forever(origin="https://www.douyin.com", ping_interval=20, ping_timeout=10)
async with websocket_connect(
url,
origin="https://www.douyin.com",
subprotocols=["binary", "base64", "pbbp2"],
extra_headers=self._connection_headers(),
user_agent_header=self.session.user_agent,
compression="deflate",
open_timeout=10,
# The current Douyin browser Frontier SDK uses a text ``hi``
# heartbeat instead of RFC WebSocket ping frames.
ping_interval=None if browser_frontier else 20,
# A handler may legitimately wait up to SQLite's 30s busy
# timeout. Leave enough headroom for queued work so a healthy
# socket isn't mistaken for a dead peer during that stall.
ping_timeout=_PING_TIMEOUT_SECONDS,
close_timeout=3,
# Frontier frames contain metadata and media URLs rather than
# media bytes. A finite frame limit plus a finite queue makes
# receive memory genuinely bounded across hundreds of peers.
max_size=_INCOMING_MAX_SIZE,
max_queue=_TRANSPORT_MAX_QUEUE,
**connect_kwargs,
) as websocket:
connection = websocket
self._connection = websocket
connected_at = loop.time()
self.connected = True
logger.info(
"IM WebSocket connected: subprotocol=%s",
getattr(websocket, "subprotocol", None) or "none",
)
self._record_connection_system_event(
"connected",
"实时接收通道已连接",
detail="frontier WebSocket 已建立,可实时接收私信。",
level="success",
)
if browser_frontier:
heartbeat_task = asyncio.create_task(
self._run_browser_heartbeat(websocket),
name=f"im-ws-heartbeat-{self.account_id or 'na'}",
)
try:
async for raw in websocket:
if not self._running:
break
await self._dispatch(raw)
finally:
if heartbeat_task and not heartbeat_task.done():
heartbeat_task.cancel()
try:
await heartbeat_task
except asyncio.CancelledError:
pass
finally:
with self._ws_lock:
if self._ws_app is ws_app:
self._ws_app = None
if connected_at is not None:
self._last_connection_lifetime = max(0.0, loop.time() - connected_at)
if self._last_connection_lifetime >= _STABLE_CONNECTION_SECONDS:
# A genuinely healthy session starts a new lifecycle. Its
# next outage should be visible immediately rather than
# hidden by an old retry window.
self._reset_connection_system_log_throttle()
self.connected = False
if self._connection is connection:
self._connection = None
if connection is not None:
code = connection.close_code
reason = connection.close_reason
logger.info("IM WebSocket closed: code=%s, msg=%s", code, reason)
if self._running:
self._record_connection_system_event(
"retry",
"实时接收通道关闭",
detail=f"code={code}, msg={reason}",
level="warning",
)
async def _dispatch(self, raw):
if raw == "hi":
if not self._heartbeat_ack_logged:
logger.info("IM WebSocket application heartbeat acknowledged")
self._heartbeat_ack_logged = True
return
self._ensure_dispatcher()
queue = self._message_queue
if queue is None:
return
if isinstance(raw, str):
payload = raw.encode("utf-8", errors="ignore")
else:
payload = raw
items = parse_ws_payload(payload)
self._received_frame_count += 1
if self._received_frame_count <= 3:
metadata = _safe_frame_metadata(payload) if not items else "parsed-message"
logger.info(
"IM WebSocket frame received: seq=%d kind=%s bytes=%d parsed=%d %s",
self._received_frame_count,
"text" if isinstance(raw, str) else "binary",
len(payload),
len(items),
metadata,
)
for item in items:
if not self._running:
return
await queue.put(item)
async def _dispatch_loop(self) -> None:
queue = self._message_queue
if queue is None:
return
handler_slots = _get_loop_state().handler_slots
while True:
item = await queue.get()
try:
await self.on_message(item)
except Exception as e:
logger.debug(f"WS message handler error: {e}")
if self._running:
# Every account owns one dispatcher, preserving its FIFO.
# The shared semaphore prevents 500 dispatchers from
# entering SQLite / reply work at the same instant. A
# dispatcher waiting here is directly cancellable by
# stop(); no detached per-message task is created.
async with handler_slots:
if self._running:
await self.on_message(item)
except asyncio.CancelledError:
raise
except Exception as exc:
logger.debug("WS message handler error: %s", exc)
system_logger.record(
"实时消息处理失败",
detail=f"处理收到的私信时出错:{e}",
detail=f"处理收到的私信时出错:{exc}",
level="error",
category="recv",
account_id=self.account_id,
)
finally:
queue.task_done()
+339
View File
@@ -0,0 +1,339 @@
"""Discover and select server egress channels for account-bound IM traffic.
One public address may be reached through a private address on the host (for
example, an ECS secondary private IP mapped to an EIP). A channel therefore
keeps both values: ``source_ip`` is bound on the socket and ``public_ip`` is
what the remote service observes.
"""
from __future__ import annotations
import asyncio
import ipaddress
import json
import logging
import os
import socket
import subprocess
import threading
import time
from dataclasses import dataclass
from typing import Iterable
import httpx
import requests
from requests.adapters import HTTPAdapter
logger = logging.getLogger("rpa_engine.egress")
_DISCOVERY_TTL_SECONDS = 300.0
_PROBE_TIMEOUT_SECONDS = 6.0
_MAX_CHANNEL_ATTEMPTS = 8
_PROBE_URLS = (
"https://www.cloudflare.com/cdn-cgi/trace",
"https://api64.ipify.org?format=json",
)
@dataclass(frozen=True)
class LocalAddress:
source_ip: str | None
interface: str
is_default: bool = False
@dataclass(frozen=True)
class EgressChannel:
public_ip: str
source_ip: str | None
interface: str = ""
is_default: bool = False
@property
def id(self) -> str:
return self.public_ip
@dataclass(frozen=True)
class EgressSnapshot:
channels: tuple[EgressChannel, ...]
errors: tuple[str, ...]
detected_at: float
class EgressChannelUnavailable(RuntimeError):
pass
_cache_lock = threading.Lock()
_cached_snapshot: EgressSnapshot | None = None
def clamp_attempts(value: int | None) -> int:
try:
parsed = int(value or 1)
except (TypeError, ValueError):
parsed = 1
return max(1, min(_MAX_CHANNEL_ATTEMPTS, parsed))
def _usable_source_ip(value: str) -> bool:
try:
addr = ipaddress.ip_address(str(value or "").strip())
except ValueError:
return False
return bool(
addr.version == 4
and not addr.is_loopback
and not addr.is_link_local
and not addr.is_multicast
and not addr.is_unspecified
)
def _linux_local_addresses() -> list[LocalAddress]:
if os.name != "posix":
return []
try:
proc = subprocess.run(
["ip", "-j", "-4", "addr", "show", "scope", "global"],
capture_output=True,
text=True,
timeout=3,
check=False,
)
payload = json.loads(proc.stdout or "[]") if proc.returncode == 0 else []
except (OSError, subprocess.SubprocessError, json.JSONDecodeError):
return []
found: list[LocalAddress] = []
for item in payload if isinstance(payload, list) else []:
interface = str(item.get("ifname") or "")
for info in item.get("addr_info") or []:
source_ip = str(info.get("local") or "").strip()
if _usable_source_ip(source_ip):
found.append(LocalAddress(source_ip, interface))
return found
def _socket_local_addresses() -> list[LocalAddress]:
found: list[LocalAddress] = []
names = {socket.gethostname(), socket.getfqdn()}
for name in names:
try:
records = socket.getaddrinfo(name, None, socket.AF_INET, socket.SOCK_STREAM)
except OSError:
continue
for record in records:
source_ip = str(record[4][0] or "").strip()
if _usable_source_ip(source_ip):
found.append(LocalAddress(source_ip, name))
return found
def local_address_candidates() -> list[LocalAddress]:
"""Return the default route plus each bindable global/private IPv4."""
candidates = [LocalAddress(None, "default", True)]
seen: set[str] = set()
for item in [*_linux_local_addresses(), *_socket_local_addresses()]:
source_ip = str(item.source_ip or "")
if not source_ip or source_ip in seen:
continue
seen.add(source_ip)
candidates.append(item)
return candidates
def _extract_public_ip(response: httpx.Response) -> str:
text = response.text.strip()
content_type = response.headers.get("content-type", "").lower()
candidate = ""
if "json" in content_type or text.startswith("{"):
try:
candidate = str(response.json().get("ip") or "").strip()
except (ValueError, AttributeError):
candidate = ""
if not candidate:
for line in text.splitlines():
if line.startswith("ip="):
candidate = line.partition("=")[2].strip()
break
if not candidate and "\n" not in text and len(text) <= 64:
candidate = text
try:
addr = ipaddress.ip_address(candidate)
except ValueError:
return ""
return str(addr) if addr.version == 4 else ""
async def _probe_local_address(candidate: LocalAddress) -> tuple[EgressChannel | None, str]:
transport = httpx.AsyncHTTPTransport(
local_address=candidate.source_ip,
retries=0,
)
last_error = ""
try:
async with httpx.AsyncClient(
transport=transport,
timeout=httpx.Timeout(_PROBE_TIMEOUT_SECONDS),
follow_redirects=True,
trust_env=False,
) as client:
for url in _PROBE_URLS:
try:
response = await client.get(url, headers={"Accept": "text/plain, application/json"})
response.raise_for_status()
public_ip = _extract_public_ip(response)
if public_ip:
return (
EgressChannel(
public_ip=public_ip,
source_ip=candidate.source_ip,
interface=candidate.interface,
is_default=candidate.is_default,
),
"",
)
last_error = "探测响应中没有 IPv4"
except Exception as exc: # one endpoint may be unavailable
last_error = str(exc) or type(exc).__name__
finally:
await transport.aclose()
label = candidate.source_ip or "默认路由"
return None, f"{label}: {last_error or '无法访问公网探测服务'}"
def _dedupe_channels(channels: Iterable[EgressChannel]) -> tuple[EgressChannel, ...]:
by_public_ip: dict[str, EgressChannel] = {}
order: list[str] = []
for channel in channels:
existing = by_public_ip.get(channel.public_ip)
if existing is None:
by_public_ip[channel.public_ip] = channel
order.append(channel.public_ip)
continue
# Keep an explicit bindable source when possible, while preserving the
# fact that this is also the host's default public route.
if existing.source_ip is None and channel.source_ip:
by_public_ip[channel.public_ip] = EgressChannel(
public_ip=channel.public_ip,
source_ip=channel.source_ip,
interface=channel.interface,
is_default=existing.is_default or channel.is_default,
)
elif channel.is_default and not existing.is_default:
by_public_ip[channel.public_ip] = EgressChannel(
public_ip=existing.public_ip,
source_ip=existing.source_ip,
interface=existing.interface,
is_default=True,
)
return tuple(by_public_ip[key] for key in order)
async def discover_egress_channels(*, force: bool = False) -> EgressSnapshot:
global _cached_snapshot
now = time.time()
with _cache_lock:
cached = _cached_snapshot
if not force and cached and now - cached.detected_at < _DISCOVERY_TTL_SECONDS:
return cached
candidates = await asyncio.to_thread(local_address_candidates)
results = await asyncio.gather(*(_probe_local_address(item) for item in candidates))
channels = _dedupe_channels(item[0] for item in results if item[0] is not None)
errors = tuple(item[1] for item in results if item[1])
snapshot = EgressSnapshot(channels=channels, errors=errors, detected_at=time.time())
with _cache_lock:
_cached_snapshot = snapshot
return snapshot
async def resolve_fixed_channel(public_ip: str) -> EgressChannel:
selected = str(public_ip or "").strip()
if not selected:
return EgressChannel(public_ip="", source_ip=None, interface="default", is_default=True)
snapshot = await discover_egress_channels()
for channel in snapshot.channels:
if channel.public_ip == selected:
return channel
raise EgressChannelUnavailable(
f"指定公网通道 {selected} 当前不可用;请在账号编辑中重新检测并选择可用通道"
)
async def resolve_send_channels(
preferred_public_ip: str = "",
max_attempts: int = 1,
) -> list[EgressChannel]:
"""Order channels for one serial send operation.
The ordinary one-channel automatic mode deliberately avoids discovery so
a temporary outage of the probe service never blocks existing sends.
"""
preferred = str(preferred_public_ip or "").strip()
attempts = clamp_attempts(max_attempts)
if not preferred and attempts == 1:
return [EgressChannel(public_ip="", source_ip=None, interface="default", is_default=True)]
snapshot = await discover_egress_channels()
channels = list(snapshot.channels)
if not channels:
if preferred:
raise EgressChannelUnavailable(
f"指定公网通道 {preferred} 无法探测;请检查服务器网卡、路由或公网访问"
)
return [EgressChannel(public_ip="", source_ip=None, interface="default", is_default=True)]
ordered: list[EgressChannel] = []
if preferred:
selected = next((item for item in channels if item.public_ip == preferred), None)
if selected is None:
raise EgressChannelUnavailable(
f"指定公网通道 {preferred} 当前不可用;请在账号编辑中重新检测"
)
ordered.append(selected)
else:
default = next((item for item in channels if item.is_default), None)
if default is not None:
ordered.append(default)
ordered.extend(item for item in channels if item not in ordered)
return ordered[:attempts]
class _SourceAddressAdapter(HTTPAdapter):
"""Requests adapter that binds outgoing sockets to one local IPv4."""
def __init__(self, source_ip: str, *args, **kwargs):
self._source_address = (source_ip, 0)
super().__init__(*args, **kwargs)
def init_poolmanager(self, connections, maxsize, block=False, **pool_kwargs):
pool_kwargs["source_address"] = self._source_address
return super().init_poolmanager(connections, maxsize, block=block, **pool_kwargs)
def proxy_manager_for(self, proxy, **proxy_kwargs):
proxy_kwargs["source_address"] = self._source_address
return super().proxy_manager_for(proxy, **proxy_kwargs)
def source_bound_requests_session(source_ip: str | None = None) -> requests.Session:
client = requests.Session()
source = str(source_ip or "").strip()
if source:
client.trust_env = False
adapter = _SourceAddressAdapter(source)
client.mount("http://", adapter)
client.mount("https://", adapter)
return client
def reset_egress_cache_for_tests() -> None:
global _cached_snapshot
with _cache_lock:
_cached_snapshot = None
File diff suppressed because it is too large Load Diff
+16
View File
@@ -82,6 +82,22 @@ def resolve_headless(default: bool = False) -> bool:
return default
def ui_conversation_page_budget(default: int = 3) -> int:
"""用户点开会话列表时允许翻的收件箱页数(KEFU_UI_CONVERSATION_PAGES)。
抖音收件箱按游标分页一次请求只给一页实测每页约 100-500KB某账号翻
6 页拿到 35 个会话仍未翻完所以必须有预算只拿一页会把其中一页当成
完整会话列表不设上限又可能为一次点击拉下好几 MB默认 3 页只是折中
花多少流量换多完整的列表属于业务取舍用环境变量调整即可
无论调到多少翻不完时都会并入本地历史不会把残缺列表伪装成完整列表
"""
try:
value = int(os.getenv("KEFU_UI_CONVERSATION_PAGES", str(default)) or default)
except (TypeError, ValueError):
value = default
return max(1, min(20, value))
# 进程内仅启动一次的虚拟显示(Xvfb)句柄
_virtual_display = None
_virtual_display_failed = False
+233
View File
@@ -0,0 +1,233 @@
"""Loopback HTTP proxy whose outbound sockets bind to one local IPv4.
Playwright does not expose a ``local_address`` option. Accounts that select a
specific server egress channel therefore use this tiny process-local proxy so
their browser login/refresh traffic leaves through the same interface as IM
HTTP and WebSocket traffic. The listener is loopback-only and does not rotate
or retry public addresses.
"""
from __future__ import annotations
import asyncio
import ipaddress
import logging
import socket
import weakref
from urllib.parse import urlsplit
logger = logging.getLogger("rpa_engine.source_proxy")
_MAX_HEADER_BYTES = 64 * 1024
_HEADER_TIMEOUT_SECONDS = 20.0
class SourceBoundProxy:
"""Minimal HTTP/HTTPS CONNECT proxy bound to a fixed source address."""
def __init__(self, source_ip: str):
address = ipaddress.ip_address(str(source_ip or "").strip())
if address.version != 4 or address.is_unspecified or address.is_multicast:
raise ValueError(f"invalid IPv4 source address: {source_ip!r}")
self.source_ip = str(address)
self._server: asyncio.AbstractServer | None = None
@property
def server_url(self) -> str:
if self._server is None or not self._server.sockets:
raise RuntimeError("source-bound proxy has not started")
port = int(self._server.sockets[0].getsockname()[1])
return f"http://127.0.0.1:{port}"
async def start(self) -> "SourceBoundProxy":
if self._server is None:
self._server = await asyncio.start_server(
self._handle_client,
host="127.0.0.1",
port=0,
family=socket.AF_INET,
)
logger.info(
"source-bound browser proxy ready: %s -> source %s",
self.server_url,
self.source_ip,
)
return self
async def close(self) -> None:
server = self._server
self._server = None
if server is not None:
server.close()
await server.wait_closed()
async def _open_upstream(
self,
host: str,
port: int,
) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:
return await asyncio.open_connection(
host=host,
port=port,
family=socket.AF_INET,
local_addr=(self.source_ip, 0),
)
@staticmethod
async def _relay(
source: asyncio.StreamReader,
destination: asyncio.StreamWriter,
) -> None:
try:
while True:
chunk = await source.read(64 * 1024)
if not chunk:
break
destination.write(chunk)
await destination.drain()
except (ConnectionError, asyncio.CancelledError):
pass
finally:
try:
destination.write_eof()
except (AttributeError, OSError, RuntimeError):
pass
@classmethod
async def _bridge(
cls,
client_reader: asyncio.StreamReader,
client_writer: asyncio.StreamWriter,
upstream_reader: asyncio.StreamReader,
upstream_writer: asyncio.StreamWriter,
) -> None:
tasks = (
asyncio.create_task(cls._relay(client_reader, upstream_writer)),
asyncio.create_task(cls._relay(upstream_reader, client_writer)),
)
try:
await asyncio.gather(*tasks)
finally:
for task in tasks:
if not task.done():
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
@staticmethod
def _parse_authority(authority: str, default_port: int) -> tuple[str, int]:
parsed = urlsplit(f"//{authority}")
host = str(parsed.hostname or "").strip()
if not host:
raise ValueError("proxy request is missing a host")
return host, int(parsed.port or default_port)
async def _handle_client(
self,
client_reader: asyncio.StreamReader,
client_writer: asyncio.StreamWriter,
) -> None:
upstream_writer: asyncio.StreamWriter | None = None
try:
header = await asyncio.wait_for(
client_reader.readuntil(b"\r\n\r\n"),
timeout=_HEADER_TIMEOUT_SECONDS,
)
if len(header) > _MAX_HEADER_BYTES:
raise ValueError("proxy request headers are too large")
lines = header.decode("latin-1").split("\r\n")
request_line = lines[0].split(" ", 2)
if len(request_line) != 3:
raise ValueError("malformed proxy request line")
method, target, version = request_line
if method.upper() == "CONNECT":
host, port = self._parse_authority(target, 443)
upstream_reader, upstream_writer = await self._open_upstream(host, port)
client_writer.write(b"HTTP/1.1 200 Connection Established\r\n\r\n")
await client_writer.drain()
else:
parsed = urlsplit(target)
host_header = next(
(
line.partition(":")[2].strip()
for line in lines[1:]
if line.lower().startswith("host:")
),
"",
)
authority = parsed.netloc or host_header
host, port = self._parse_authority(
authority,
443 if parsed.scheme.lower() == "https" else 80,
)
upstream_reader, upstream_writer = await self._open_upstream(host, port)
origin_target = parsed.path or "/"
if parsed.query:
origin_target += f"?{parsed.query}"
forwarded = [f"{method} {origin_target} {version}"]
forwarded.extend(
line for line in lines[1:]
if line and not line.lower().startswith("proxy-connection:")
)
upstream_writer.write(("\r\n".join(forwarded) + "\r\n\r\n").encode("latin-1"))
await upstream_writer.drain()
await self._bridge(
client_reader,
client_writer,
upstream_reader,
upstream_writer,
)
except asyncio.IncompleteReadError:
pass
except asyncio.CancelledError:
# Event-loop shutdown may cancel an in-flight browser tunnel.
# Closing both writers below is sufficient; do not leak a noisy
# cancelled handler callback into the server log.
pass
except Exception as exc:
logger.warning("source-bound browser proxy request failed: %s", exc)
try:
client_writer.write(
b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\n\r\n"
)
await client_writer.drain()
except (ConnectionError, RuntimeError):
pass
finally:
for writer in (upstream_writer, client_writer):
if writer is None:
continue
try:
writer.close()
await writer.wait_closed()
except (ConnectionError, RuntimeError):
pass
class _LoopProxyState:
def __init__(self) -> None:
self.lock = asyncio.Lock()
self.proxies: dict[str, SourceBoundProxy] = {}
_loop_states: weakref.WeakKeyDictionary[
asyncio.AbstractEventLoop, _LoopProxyState
] = weakref.WeakKeyDictionary()
async def playwright_proxy_for_source(source_ip: str) -> dict[str, str]:
"""Return a Playwright proxy config fixed to ``source_ip``."""
loop = asyncio.get_running_loop()
state = _loop_states.get(loop)
if state is None:
state = _LoopProxyState()
_loop_states[loop] = state
normalized = str(ipaddress.ip_address(str(source_ip or "").strip()))
async with state.lock:
proxy = state.proxies.get(normalized)
if proxy is None:
proxy = await SourceBoundProxy(normalized).start()
state.proxies[normalized] = proxy
return {"server": proxy.server_url}
+340
View File
@@ -0,0 +1,340 @@
from __future__ import annotations
import os
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import sessionmaker
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ["KEFU_DB_TYPE"] = "sqlite"
os.environ["KEFU_DATABASE_URL"] = ""
os.environ["KEFU_DB_PATH"] = str(BACKEND_DIR / "kefu.db")
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import main
from models.database import Base
from models.models import Account, MessageLog
class _CountResult:
def __init__(self, count: int):
self.count = count
def scalar_one(self):
return self.count
class _RowsResult:
def __init__(self, rows):
self.rows = list(rows)
def scalars(self):
return self
def all(self):
return list(self.rows)
class AccountPaginationTests(unittest.IsolatedAsyncioTestCase):
async def test_legacy_cookie_sync_only_reads_accounts_missing_db_cookie(self):
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
session_factory = sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False,
)
try:
async with session_factory() as db:
db.add_all(
[
Account(
id=2301,
username="already-in-db",
cookie_data='{"cookies": [{"value": "large"}]}',
im_session_data="x" * 100_000,
),
Account(id=2302, username="legacy-file", cookie_data=None),
]
)
await db.commit()
def read_cookie(account_id: int):
self.assertEqual(account_id, 2302)
return '{"cookies": [{"value": "migrated"}]}'
with (
patch.object(main, "AsyncSessionLocal", session_factory),
patch.object(main, "read_cookie_file", side_effect=read_cookie) as read_file,
patch.object(main, "get_cookie_path", return_value="legacy-2302.json"),
):
await main._sync_legacy_cookie_files()
read_file.assert_called_once_with(2302)
async with session_factory() as db:
migrated = await db.get(Account, 2302)
self.assertEqual(
migrated.cookie_data,
'{"cookies": [{"value": "migrated"}]}',
)
self.assertEqual(migrated.cookie_path, "legacy-2302.json")
finally:
await engine.dispose()
async def test_account_edit_invalidates_running_follow_config_cache(self):
account = SimpleNamespace(id=2201)
db = SimpleNamespace(commit=AsyncMock(), refresh=AsyncMock())
worker = SimpleNamespace(invalidate_follow_welcome_config=MagicMock())
original_workers = main.manager.workers
main.manager.workers = {2201: worker}
try:
with (
patch.object(main, "get_owned_account", AsyncMock(return_value=account)),
patch.object(main, "_build_account_response", return_value={"id": 2201}),
):
response = await main.update_account(
account_id=2201,
body=main.AccountUpdate(follow_welcome_enabled=True),
db=db,
user=SimpleNamespace(id=7, role="operator"),
)
self.assertEqual(response, {"id": 2201})
worker.invalidate_follow_welcome_config.assert_called_once_with()
finally:
main.manager.workers = original_workers
async def test_account_channel_change_stops_running_worker(self):
account = SimpleNamespace(
id=2202,
egress_public_ip="116.62.23.103",
status="online",
)
db = SimpleNamespace(
commit=AsyncMock(),
refresh=AsyncMock(),
execute=AsyncMock(),
)
with (
patch.object(main, "get_owned_account", AsyncMock(return_value=account)),
patch.object(main.manager, "is_running", return_value=True),
patch.object(main.manager, "stop_worker", AsyncMock(return_value=True)) as stop,
patch.object(main, "_build_account_response", return_value={"id": 2202}),
):
response = await main.update_account(
account_id=2202,
body=main.AccountUpdate(egress_public_ip="47.96.154.74"),
db=db,
user=SimpleNamespace(id=7, role="operator"),
)
self.assertEqual(response, {"id": 2202})
self.assertEqual(account.egress_public_ip, "47.96.154.74")
stop.assert_awaited_once_with(2202)
self.assertEqual(db.commit.await_count, 2)
values = db.execute.await_args.args[0].compile().params
self.assertIn("已保留登录凭证", values["error_message"])
self.assertNotIn("cookie_data", values)
self.assertNotIn("im_session_data", values)
async def test_log_stats_uses_one_aggregate_and_respects_ownership(self):
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
session_factory = sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False,
)
try:
async with session_factory() as db:
db.add_all(
[
Account(id=2101, owner_id=7, status="offline"),
Account(id=2102, owner_id=8, status="offline"),
MessageLog(account_id=2101, status="received"),
MessageLog(account_id=2101, status="replied"),
MessageLog(account_id=2102, status="replied"),
]
)
await db.commit()
with patch.object(db, "execute", wraps=db.execute) as execute:
stats = await main.get_logs_stats(
account_id=None,
db=db,
user=SimpleNamespace(id=7, role="user"),
)
self.assertEqual(stats, {"total": 2, "replied": 1})
self.assertEqual(execute.await_count, 1)
finally:
await engine.dispose()
async def test_account_options_returns_lightweight_runtime_fields(self):
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
session_factory = sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False,
)
original_workers = main.manager.workers
main.manager.workers = {2001: SimpleNamespace(is_running=True)}
try:
async with session_factory() as db:
db.add_all(
[
Account(
id=2001,
owner_id=7,
username="owned",
status="offline",
cookie_data='{"cookies": []}',
im_session_data="x" * 100_000,
reply_cooldown_seconds=12,
),
Account(
id=2002,
owner_id=8,
username="other",
status="online",
cookie_data='{"cookies": []}',
),
]
)
await db.commit()
options = await main.get_account_options(
db=db,
user=SimpleNamespace(id=7, role="user"),
)
self.assertEqual(len(options), 1)
self.assertEqual(options[0].id, 2001)
self.assertEqual(options[0].status, "online")
self.assertTrue(options[0].has_cookie)
self.assertEqual(options[0].reply_cooldown_seconds, 12)
self.assertEqual(options[0].reply_cooldown_effective, 12)
self.assertFalse(hasattr(options[0], "im_session_data"))
finally:
main.manager.workers = original_workers
await engine.dispose()
async def test_paginated_list_counts_then_loads_only_current_page(self):
page_rows = [
SimpleNamespace(id=10, status="offline"),
SimpleNamespace(id=11, status="online"),
]
db = SimpleNamespace(
execute=AsyncMock(
side_effect=[
_CountResult(392),
_RowsResult(page_rows),
]
)
)
with (
patch.object(main.manager, "is_running", side_effect=[False, True]),
patch.object(
main,
"_build_account_response",
side_effect=lambda account: {"id": account.id, "status": account.status},
) as build_response,
):
response = await main.get_accounts(
page=20,
page_size=20,
q=None,
status=None,
db=db,
user=SimpleNamespace(id=1, role="admin"),
)
self.assertEqual(db.execute.await_count, 2)
self.assertEqual(response["total"], 392)
self.assertEqual(response["page"], 20)
self.assertEqual(response["page_size"], 20)
self.assertEqual([item["id"] for item in response["items"]], [10, 11])
self.assertEqual(build_response.call_count, 2)
count_sql = str(db.execute.await_args_list[0].args[0]).upper()
page_sql = str(db.execute.await_args_list[1].args[0]).upper()
self.assertIn("COUNT", count_sql)
self.assertNotIn(" LIMIT ", count_sql)
self.assertIn(" LIMIT ", page_sql)
self.assertIn(" OFFSET ", page_sql)
async def test_status_filter_uses_effective_runtime_worker_state(self):
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
session_factory = sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False,
)
original_workers = main.manager.workers
main.manager.workers = {
1001: SimpleNamespace(is_running=True),
1002: SimpleNamespace(is_running=False),
}
try:
async with session_factory() as db:
db.add_all(
[
Account(id=1001, status="offline"),
Account(id=1002, status="online"),
]
)
await db.commit()
with patch.object(
main,
"_build_account_response",
side_effect=lambda account: {
"id": account.id,
"status": account.status,
},
):
online = await main.get_accounts(
page=1,
page_size=20,
q=None,
status="online",
db=db,
user=SimpleNamespace(id=1, role="admin"),
)
await db.rollback()
db.expire_all()
offline = await main.get_accounts(
page=1,
page_size=20,
q=None,
status="offline",
db=db,
user=SimpleNamespace(id=1, role="admin"),
)
self.assertEqual(online["total"], 1)
self.assertEqual(online["items"], [{"id": 1001, "status": "online"}])
self.assertEqual(offline["total"], 1)
self.assertEqual(offline["items"], [{"id": 1002, "status": "offline"}])
finally:
main.manager.workers = original_workers
await engine.dispose()
if __name__ == "__main__":
unittest.main()
+353
View File
@@ -1,5 +1,7 @@
from __future__ import annotations
import asyncio
import json
import os
import sys
import unittest
@@ -32,6 +34,357 @@ def _fake_db(rows):
class BatchStartApiTests(unittest.IsolatedAsyncioTestCase):
async def test_shutdown_continues_when_batch_queue_cleanup_times_out(self):
original_workers = main.manager.workers
original_flush_task = main._system_log_flush_task
main.manager.workers = {}
main._system_log_flush_task = None
stop_started = asyncio.Event()
async def blocked_batch_stop():
stop_started.set()
await asyncio.Event().wait()
try:
with (
patch.dict(
os.environ,
{"KEFU_BATCH_STOP_TIMEOUT_SECONDS": "1"},
),
patch.object(
main.batch_start_queue,
"stop",
AsyncMock(side_effect=blocked_batch_stop),
),
patch(
"rpa_engine.douyin_im.traffic_control.shutdown_traffic_controller",
AsyncMock(),
) as stop_traffic,
):
await asyncio.wait_for(main.shutdown(), timeout=2.0)
self.assertTrue(stop_started.is_set())
stop_traffic.assert_awaited_once_with()
finally:
main.manager.workers = original_workers
main._system_log_flush_task = original_flush_task
async def test_shutdown_stops_many_accounts_with_bounded_parallelism(self):
active = 0
maximum_active = 0
stopped: list[int] = []
original_workers = main.manager.workers
original_flush_task = main._system_log_flush_task
main.manager.workers = {
account_id: SimpleNamespace(is_running=True)
for account_id in range(601, 613)
}
main._system_log_flush_task = None
async def stop_worker(account_id: int):
nonlocal active, maximum_active
active += 1
maximum_active = max(maximum_active, active)
try:
await asyncio.sleep(0.005)
stopped.append(account_id)
main.manager.workers.pop(account_id, None)
return True
finally:
active -= 1
try:
with (
patch.dict(
os.environ,
{
"KEFU_SHUTDOWN_CONCURRENCY": "3",
"KEFU_SHUTDOWN_TIMEOUT_SECONDS": "5",
},
),
patch.object(main.batch_start_queue, "stop", AsyncMock()),
patch.object(main.manager, "stop_worker", AsyncMock(side_effect=stop_worker)),
patch(
"rpa_engine.douyin_im.traffic_control.shutdown_traffic_controller",
AsyncMock(),
),
):
await main.shutdown()
self.assertEqual(len(stopped), 12)
self.assertEqual(maximum_active, 3)
finally:
main.manager.workers = original_workers
main._system_log_flush_task = original_flush_task
async def test_worker_manager_waits_for_full_ready_and_reuses_validation(self):
worker = SimpleNamespace(
is_running=True,
start=AsyncMock(),
wait_until_ready=AsyncMock(),
)
manager = main.WorkerManager()
with patch.object(main, "DouyinWorker", return_value=worker) as worker_factory:
started = await manager.start_worker(
501,
login_mode="im_direct",
wait_until_ready=True,
credential_prevalidated=True,
)
self.assertTrue(started)
worker_factory.assert_called_once_with(
501,
login_mode="im_direct",
credential_prevalidated=True,
)
worker.start.assert_awaited_once_with()
worker.wait_until_ready.assert_awaited_once_with()
self.assertIs(manager.workers[501], worker)
async def test_cancelled_ready_wait_stops_and_removes_detached_worker(self):
wait_started = asyncio.Event()
waiting = asyncio.Event()
async def wait_forever():
wait_started.set()
await waiting.wait()
worker = SimpleNamespace(
is_running=True,
start=AsyncMock(),
wait_until_ready=AsyncMock(side_effect=wait_forever),
stop=AsyncMock(),
)
manager = main.WorkerManager()
with patch.object(main, "DouyinWorker", return_value=worker):
task = asyncio.create_task(
manager.start_worker(
502,
login_mode="im_direct",
wait_until_ready=True,
credential_prevalidated=True,
)
)
await asyncio.wait_for(wait_started.wait(), timeout=0.2)
task.cancel()
with self.assertRaises(asyncio.CancelledError):
await task
worker.stop.assert_awaited_once_with()
self.assertNotIn(502, manager.workers)
async def test_batch_start_waits_for_ready_and_skips_duplicate_validation(self):
account = SimpleNamespace(
id=503,
status="offline",
qr_code_base64=None,
error_message=None,
im_session_data="saved-session",
)
db = SimpleNamespace(commit=AsyncMock())
assessment = {
"login_mode": "im_direct",
"should_reset": False,
"can_skip_browser": True,
"message": "ready",
"cookie_valid": True,
"im_ready": True,
}
with (
patch.object(main.manager, "is_running", return_value=False),
patch.object(main.manager, "start_worker", AsyncMock(return_value=True)) as start,
patch.object(main, "_get_account_cookie_data", return_value="{}"),
patch.object(main, "assess_account_credential", AsyncMock(return_value=assessment)),
):
result = await main._start_account_rpa_impl(
account,
db,
wait_for_ready=True,
)
start.assert_awaited_once_with(
503,
login_mode="im_direct",
wait_until_ready=True,
credential_prevalidated=True,
)
self.assertEqual(result["status"], "running")
async def test_batch_start_holds_no_db_connection_while_it_waits(self):
"""A queued start must not pin one of the few pooled connections.
Credential validation and readiness waiting take seconds per account.
Holding a session open across them exhausted the pool during a bulk
start, so every unrelated request waited out ``pool_timeout``.
"""
account = SimpleNamespace(
id=505,
status="starting",
qr_code_base64=None,
error_message=None,
im_session_data="saved-session",
)
events: list[str] = []
assessment = {
"login_mode": "im_direct",
"should_reset": False,
"can_skip_browser": True,
"message": "ready",
"cookie_valid": True,
"im_ready": True,
}
async def commit():
events.append("release")
async def assess(*_args, **_kwargs):
events.append("assess")
return assessment
async def start_worker(*_args, **_kwargs):
events.append("start-worker")
return True
db = SimpleNamespace(commit=AsyncMock(side_effect=commit))
with (
patch.object(main.manager, "is_running", return_value=False),
patch.object(main.manager, "start_worker", AsyncMock(side_effect=start_worker)),
patch.object(main, "_get_account_cookie_data", return_value="{}"),
patch.object(main, "assess_account_credential", AsyncMock(side_effect=assess)),
):
await main._start_account_rpa_impl(account, db, wait_for_ready=True)
# "starting" was already persisted, so the only commits here exist to
# return the connection: one before validation, one before the wait.
self.assertEqual(events, ["release", "assess", "release", "start-worker"])
async def test_changed_egress_preserves_valid_credentials(self):
ready_assessment = {
"login_mode": "im_direct",
"should_reset": False,
"can_skip_browser": True,
"message": "ready",
"cookie_valid": True,
"im_ready": True,
}
scenarios = (
({"cookies": []}, "47.96.154.74"),
({"cookies": [], "credential_egress_public_ip": "116.62.23.103"}, "47.96.154.74"),
({"cookies": [], "credential_egress_public_ip": "47.96.154.74"}, ""),
)
modes = (("im_direct", False), (None, False), (None, True))
for storage, selected_ip in scenarios:
for requested_mode, wait_for_ready in modes:
with self.subTest(storage=storage, mode=requested_mode, batch=wait_for_ready):
cookie_data = json.dumps(storage)
account = SimpleNamespace(
id=506,
status="offline",
qr_code_base64=None,
error_message="old channel warning",
cookie_data=cookie_data,
im_session_data="saved-session",
egress_public_ip=selected_ip,
)
db = SimpleNamespace(commit=AsyncMock())
with (
patch.object(main.manager, "is_running", return_value=False),
patch.object(main.manager, "start_worker", AsyncMock(return_value=True)) as start,
patch.object(main, "_get_account_cookie_data", return_value=cookie_data),
patch.object(main, "_reset_account_credentials", AsyncMock()) as reset,
patch.object(main, "assess_account_credential", AsyncMock(return_value=ready_assessment)) as assess,
):
result = await main._start_account_rpa_impl(
account, db, requested_mode, wait_for_ready=wait_for_ready
)
reset.assert_not_awaited()
assess.assert_awaited_once_with(
cookie_data, "saved-session",
startup_priority=True, egress_public_ip=selected_ip,
)
start.assert_awaited_once_with(
506, login_mode="im_direct",
wait_until_ready=wait_for_ready, credential_prevalidated=True,
)
self.assertEqual(account.cookie_data, cookie_data)
self.assertEqual(account.im_session_data, "saved-session")
self.assertIsNone(account.error_message)
self.assertTrue(result["skip_qr"])
self.assertTrue(result["skip_browser"])
async def test_changed_egress_still_rejects_invalid_im_credentials(self):
account = SimpleNamespace(
id=506,
status="offline",
qr_code_base64=None,
error_message=None,
im_session_data="saved-session",
egress_public_ip="47.96.154.74",
)
db = SimpleNamespace(commit=AsyncMock())
invalid_assessment = {
"login_mode": "browser",
"should_reset": False,
"can_skip_browser": False,
"message": "缺少 IM 签名密钥(web_protect/keys),请用浏览器登录补全",
"cookie_valid": True,
"im_ready": False,
}
with (
patch.object(main.manager, "is_running", return_value=False),
patch.object(main.manager, "start_worker", AsyncMock()) as start,
patch.object(main, "_get_account_cookie_data", return_value='{"cookies": []}'),
patch.object(main, "_reset_account_credentials", AsyncMock()) as reset,
patch.object(main, "assess_account_credential", AsyncMock(return_value=invalid_assessment)),
):
with self.assertRaises(main.HTTPException) as error:
await main._start_account_rpa_impl(account, db, "im_direct")
self.assertEqual(error.exception.status_code, 400)
self.assertEqual(error.exception.detail, invalid_assessment["message"])
reset.assert_not_awaited()
start.assert_not_awaited()
async def test_batch_start_does_not_launch_interactive_browser_login(self):
account = SimpleNamespace(
id=504,
status="offline",
qr_code_base64=None,
error_message=None,
im_session_data=None,
)
db = SimpleNamespace(commit=AsyncMock())
assessment = {
"login_mode": "browser",
"should_reset": False,
"can_skip_browser": False,
"message": "login required",
"cookie_valid": False,
"im_ready": False,
}
with (
patch.object(main.manager, "is_running", return_value=False),
patch.object(main.manager, "start_worker", AsyncMock()) as start,
patch.object(main, "_get_account_cookie_data", return_value=None),
patch.object(main, "assess_account_credential", AsyncMock(return_value=assessment)),
):
with self.assertRaisesRegex(RuntimeError, "批量启动"):
await main._start_account_rpa_impl(
account,
db,
wait_for_ready=True,
)
start.assert_not_awaited()
async def test_start_all_uses_lightweight_select_and_submits_once(self):
db = _fake_db([(1, False), (2, True), (3, False), (4, False)])
user = SimpleNamespace(id=9, role="admin")
+96 -2
View File
@@ -17,8 +17,18 @@ from rpa_engine import batch_start as batch_start_module
class BatchStartQueueTests(unittest.IsolatedAsyncioTestCase):
def _make_queue(self, handler, *, concurrency: int = 2) -> BatchStartQueue:
queue = BatchStartQueue(handler, concurrency=concurrency)
def _make_queue(
self,
handler,
*,
concurrency: int = 2,
timeout_seconds: float | None = None,
) -> BatchStartQueue:
queue = BatchStartQueue(
handler,
concurrency=concurrency,
timeout_seconds=timeout_seconds,
)
self.addAsyncCleanup(queue.stop)
return queue
@@ -77,6 +87,42 @@ class BatchStartQueueTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(maximum_active, 2)
self.assertEqual(active, 0)
async def test_default_concurrency_admits_more_than_two_accounts(self):
with patch.dict(os.environ, {}, clear=False):
os.environ.pop("KEFU_BATCH_START_CONCURRENCY", None)
queue = BatchStartQueue(lambda _account_id: None)
self.assertEqual(queue.concurrency, batch_start_module.DEFAULT_CONCURRENCY)
self.assertGreaterEqual(queue.concurrency, 4)
async def test_dead_worker_is_replaced_so_width_never_shrinks(self):
"""One crashed worker must not permanently narrow the queue.
Width used to be restored only when every worker had exited, so a
single unexpected worker death left later batches crawling through the
survivors until the process restarted.
"""
async def handler(account_id: int) -> dict:
return {"message": f"started-{account_id}"}
queue = self._make_queue(handler, concurrency=3)
first = await queue.submit([91])
await self._wait_for_complete(queue, first["batch_id"])
self.assertEqual(len(queue._workers), 3)
casualty = queue._workers[0]
casualty.cancel()
await asyncio.gather(casualty, return_exceptions=True)
second = await queue.submit([92])
await self._wait_for_complete(queue, second["batch_id"])
self.assertEqual(len(queue._workers), 3)
self.assertNotIn(casualty, queue._workers)
self.assertTrue(all(not task.done() for task in queue._workers))
names = [task.get_name() for task in queue._workers]
self.assertEqual(len(set(names)), 3)
async def test_submit_returns_while_handler_is_blocked(self):
handler_started = asyncio.Event()
release_handler = asyncio.Event()
@@ -168,6 +214,54 @@ class BatchStartQueueTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(by_account[32]["status"], "submitted")
self.assertEqual(by_account[33]["status"], "submitted")
async def test_two_timeouts_release_both_workers_for_following_accounts(self):
never_release = asyncio.Event()
calls: list[int] = []
async def handler(account_id: int) -> dict:
calls.append(account_id)
if account_id in (71, 72):
await never_release.wait()
return {"message": f"started-{account_id}"}
queue = self._make_queue(
handler,
concurrency=2,
timeout_seconds=0.02,
)
with patch.object(batch_start_module.logger, "warning"):
submitted = await queue.submit([71, 72, 73, 74])
completed = await self._wait_for_complete(
queue,
submitted["batch_id"],
)
self.assertEqual(calls, [71, 72, 73, 74])
self.assertEqual(completed["failed_count"], 2)
self.assertEqual(completed["submitted_count"], 2)
by_account = {item["account_id"]: item for item in completed["items"]}
self.assertIn("已跳过并继续处理后续账号", by_account[71]["message"])
self.assertIn("已跳过并继续处理后续账号", by_account[72]["message"])
self.assertEqual(by_account[73]["status"], "submitted")
self.assertEqual(by_account[74]["status"], "submitted")
async def test_handler_timeout_error_keeps_its_original_detail(self):
async def handler(_account_id: int) -> dict:
raise asyncio.TimeoutError("upstream request timed out")
queue = self._make_queue(
handler,
concurrency=1,
timeout_seconds=10,
)
with patch.object(batch_start_module.logger, "exception"):
submitted = await queue.submit([75])
completed = await self._wait_for_complete(queue, submitted["batch_id"])
item = completed["items"][0]
self.assertEqual(item["status"], "failed")
self.assertEqual(item["message"], "upstream request timed out")
async def test_failed_account_can_be_submitted_again(self):
attempts = 0
+399 -36
View File
@@ -1,10 +1,13 @@
from __future__ import annotations
import asyncio
import os
import sys
import unittest
from contextlib import asynccontextmanager
from pathlib import Path
from unittest.mock import AsyncMock
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
BACKEND_DIR = Path(__file__).resolve().parents[1]
@@ -16,6 +19,8 @@ if str(BACKEND_DIR) not in sys.path:
from rpa_engine.douyin_im.http_client import DouyinImHttpClient
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.douyin_im.service import DouyinImService, _conversation_poll_timing
from rpa_engine.douyin_im import service as service_module
class ConversationPollBandwidthTests(unittest.IsolatedAsyncioTestCase):
@@ -25,56 +30,414 @@ class ConversationPollBandwidthTests(unittest.IsolatedAsyncioTestCase):
account_id=9,
)
async def test_terminal_token_error_does_not_probe_other_payloads(self):
async def test_inbox_is_fetched_with_exactly_one_protobuf_request(self):
"""imapi 只认 protobuf;轮询一次就只该发一个请求。"""
client = self._make_client()
client._request = AsyncMock(
return_value={
"status_code": 500,
"error_desc": "empty token",
"body": {},
}
)
client.fetch_inbox_messages = AsyncMock(return_value=[])
client._request = AsyncMock()
self.assertEqual(await client.get_conversations(), [])
client._request.assert_awaited_once()
self.assertEqual(client._request.await_args.args[0], "POST")
client.fetch_inbox_messages.assert_awaited_once()
# 不能再退回 JSON 的 /v1/conversation/list:那个请求恒被抖音拒绝。
client._request.assert_not_awaited()
async def test_successful_empty_response_stops_after_first_payload(self):
async def test_transport_failure_is_recorded_and_returns_empty(self):
client = self._make_client()
client._request = AsyncMock(
return_value={"status_code": 0, "body": {"conversation_list": []}}
)
client.fetch_inbox_messages = AsyncMock(side_effect=RuntimeError("boom"))
self.assertEqual(await client.get_conversations(), [])
client._request.assert_awaited_once()
with self.assertLogs("douyin_im.http", level="WARNING"):
self.assertEqual(await client.get_conversations(), [])
async def test_parameter_error_can_fall_through_to_compatible_payload(self):
self.assertIn("boom", client.last_error)
async def test_inbox_messages_group_into_one_row_per_conversation(self):
client = self._make_client()
client._request = AsyncMock(
side_effect=[
{"status_code": 400, "error_desc": "invalid parameter"},
{"status_code": 0, "body": {"conversation_list": []}},
client.fetch_inbox_messages = AsyncMock(
return_value=[
{
"conversation_id": "0:1:10001:20001",
"server_message_id": "700",
"conversation_short_id": "555",
"message_type": 7,
"sender": "20001",
"content": '{"text":""}',
},
{
"conversation_id": "0:1:10001:20001",
"server_message_id": "900",
"conversation_short_id": "555",
"message_type": 7,
"sender": "20001",
"content": '{"text":""}',
},
{
"conversation_id": "0:1:10001:20002",
"server_message_id": "800",
"message_type": 7,
"sender": "20002",
"content": '{"text":"另一个"}',
},
]
)
self.assertEqual(await client.get_conversations(), [])
self.assertEqual(client._request.await_count, 2)
self.assertTrue(
all(call.args[0] == "POST" for call in client._request.await_args_list)
)
rows = await client.get_conversations(enrich_profiles=False)
async def test_get_fallback_only_runs_after_transport_failure(self):
client = self._make_client()
client._request = AsyncMock(
side_effect=[None, {"status_code": 0, "body": {}}]
)
self.assertEqual(await client.get_conversations(), [])
by_id = {r["conversation_id"]: r for r in rows}
self.assertEqual(len(rows), 2)
# 同一会话只保留 server_message_id 最大的那条
self.assertEqual(by_id["0:1:10001:20001"]["server_message_id"], "900")
self.assertIn("", by_id["0:1:10001:20001"]["content"])
# peer_uid 由 conversation_id 推导,不能直接取 sender(可能是自己)
self.assertEqual(by_id["0:1:10001:20002"]["peer_uid"], "20002")
# 顺手缓存 short_id,发送时就不必再 create 一次会话
self.assertEqual(
[call.args[0] for call in client._request.await_args_list],
["POST", "GET"],
client.session.conv_meta["0:1:10001:20001"]["conversation_short_id"],
"555",
)
def test_websocket_reconciliation_is_slow_and_http_fallback_stays_fast(self):
with patch.dict(
os.environ,
{
"KEFU_WS_RECONCILE_INTERVAL_SECONDS": "120",
"KEFU_HTTP_POLL_INTERVAL_SECONDS": "15",
},
):
ws_interval, ws_stagger = _conversation_poll_timing(123, True)
http_interval, http_stagger = _conversation_poll_timing(123, False)
self.assertEqual(ws_interval, 120)
self.assertEqual(http_interval, 15)
self.assertGreaterEqual(ws_stagger, 0)
self.assertLess(ws_stagger, ws_interval)
self.assertGreaterEqual(http_stagger, 0)
self.assertLess(http_stagger, http_interval)
def test_poll_interval_expands_to_the_configured_population_budget(self):
with patch.dict(
os.environ,
{
"KEFU_WS_RECONCILE_INTERVAL_SECONDS": "120",
"KEFU_HTTP_POLL_INTERVAL_SECONDS": "15",
"KEFU_WS_POLL_BUDGET_RPS": "1",
"KEFU_HTTP_POLL_BUDGET_RPS": "1",
},
):
ws_interval, _ = _conversation_poll_timing(
123,
True,
population=500,
)
http_interval, _ = _conversation_poll_timing(
123,
False,
population=500,
)
self.assertEqual(ws_interval, 500)
self.assertEqual(http_interval, 500)
def test_initial_unread_concurrency_is_configurable_and_bounded(self):
with patch.dict(
os.environ,
{"KEFU_INITIAL_UNREAD_CONCURRENCY": "4"},
):
self.assertEqual(service_module._initial_unread_concurrency(), 4)
with patch.dict(
os.environ,
{"KEFU_INITIAL_UNREAD_CONCURRENCY": "999"},
):
self.assertEqual(service_module._initial_unread_concurrency(), 8)
async def test_service_poll_uses_one_conversation_request_without_unread_probe(self):
class _Controller:
@asynccontextmanager
async def background_slot(self, *_args, **_kwargs):
yield
class _HttpClient:
def __init__(self):
self.get_conversations = AsyncMock(return_value=[])
self.conversation_list_unsupported = False
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
return False
http = _HttpClient()
service = DouyinImService(
session=DouyinImSession(cookies={"sessionid": "test"}, my_uid=10001),
match_reply=AsyncMock(return_value=None),
log_fn=AsyncMock(),
account_id=9,
)
service._index_conversations = AsyncMock()
with (
patch.object(service_module, "get_traffic_controller", return_value=_Controller()),
patch.object(service_module, "DouyinImHttpClient", return_value=http),
):
await service._poll_conversations()
http.get_conversations.assert_awaited_once_with(enrich_profiles=False)
service._index_conversations.assert_awaited_once_with(
[],
enrich_profiles=False,
)
async def test_poll_only_handles_unread_or_a_genuinely_changed_preview(self):
class _Controller:
def __init__(self):
self.startup_flags = []
@asynccontextmanager
async def background_slot(self, *_args, **kwargs):
self.startup_flags.append(bool(kwargs.get("startup")))
yield
snapshots = [
[
{
"conversation_id": "0:1:10001:20001",
"peer_uid": "20001",
"sender_name": "历史会话",
"sender_avatar": "https://example.test/a.png",
"content": "历史消息",
"unread_count": 0,
},
{
"conversation_id": "0:1:10001:20002",
"peer_uid": "20002",
"sender_name": "未读会话",
"sender_avatar": "https://example.test/b.png",
"content": "新消息",
"unread_count": 1,
},
],
[
{
"conversation_id": "0:1:10001:20001",
"peer_uid": "20001",
"sender_name": "历史会话",
"sender_avatar": "https://example.test/a.png",
"content": "历史消息",
"unread_count": 0,
},
{
"conversation_id": "0:1:10001:20002",
"peer_uid": "20002",
"sender_name": "未读会话",
"sender_avatar": "https://example.test/b.png",
"content": "新消息",
"unread_count": 0,
},
],
[
{
"conversation_id": "0:1:10001:20001",
"peer_uid": "20001",
"sender_name": "历史会话",
"sender_avatar": "https://example.test/a.png",
"content": "真正发生变化",
"unread_count": 0,
},
{
"conversation_id": "0:1:10001:20002",
"peer_uid": "20002",
"sender_name": "未读会话",
"sender_avatar": "https://example.test/b.png",
"content": "新消息",
"unread_count": 0,
},
],
]
class _HttpClient:
def __init__(self):
self.get_conversations = AsyncMock(side_effect=snapshots)
self.conversation_list_unsupported = False
self.enter_count = 0
self.exit_count = 0
async def __aenter__(self):
self.enter_count += 1
return self
async def __aexit__(self, *_args):
self.exit_count += 1
return False
http = _HttpClient()
service = DouyinImService(
session=DouyinImSession(cookies={"sessionid": "test"}, my_uid=10001),
match_reply=AsyncMock(return_value=None),
log_fn=AsyncMock(),
account_id=9,
)
service._handle_incoming = AsyncMock()
controller = _Controller()
with (
patch.object(
service_module,
"get_traffic_controller",
return_value=controller,
),
patch.object(service_module, "DouyinImHttpClient", return_value=http) as factory,
):
# Startup indexes both previews, but only the unread conversation
# is allowed to enter the reply path.
await service._poll_conversations(initial=True)
self.assertEqual(service._handle_incoming.await_count, 1)
self.assertEqual(
service._handle_incoming.await_args.args[0]["peer_uid"],
"20002",
)
# The first normal reconciliation sees the exact same previews;
# it must not merely defer a historical-message reply explosion.
service._handle_incoming.reset_mock()
await service._poll_conversations()
service._handle_incoming.assert_not_awaited()
# A real preview transition is processed even if unread_count is
# unavailable/zero on the upstream response.
await service._poll_conversations()
service._handle_incoming.assert_awaited_once()
self.assertEqual(
service._handle_incoming.await_args.args[0]["peer_uid"],
"20001",
)
self.assertEqual(factory.call_count, 3)
self.assertEqual(http.enter_count, 3)
self.assertEqual(http.exit_count, 3)
self.assertEqual(controller.startup_flags, [True, False, False])
async def test_ready_is_not_blocked_by_slow_initial_unread_handler(self):
events: list[str] = []
ready = asyncio.Event()
handler_started = asyncio.Event()
handler_cancelled = asyncio.Event()
never_release = asyncio.Event()
def on_ready():
events.append("ready")
ready.set()
async def slow_handler(_message):
events.append("handler")
handler_started.set()
try:
await never_release.wait()
except asyncio.CancelledError:
handler_cancelled.set()
raise
class _WsClient:
connected = False
def __init__(self, *_args, **_kwargs):
self.start = AsyncMock()
self.stop = AsyncMock()
service = DouyinImService(
session=DouyinImSession(cookies={"sessionid": "test"}, my_uid=10001),
match_reply=AsyncMock(return_value=None),
log_fn=AsyncMock(),
account_id=901,
on_ready=on_ready,
)
service._verify_account_uid = AsyncMock()
service._poll_conversations = AsyncMock(
return_value=[
{
"conversation_id": "0:1:10001:29001",
"content": "startup unread",
"unread_count": 1,
}
]
)
service._handle_incoming = AsyncMock(side_effect=slow_handler)
service._reply_queue.start = AsyncMock()
service._reply_queue.stop = AsyncMock()
with (
patch.object(service_module, "DouyinImWsClient", _WsClient),
patch.object(service_module, "ensure_frontier_ws"),
patch.object(service_module.system_logger, "record"),
patch(
"rpa_engine.douyin_im.emoji_pack.is_fresh",
return_value=True,
),
):
run_task = asyncio.create_task(service.run())
try:
# If initial unread were still processed inline, this wait
# would time out because slow_handler never completes.
await asyncio.wait_for(ready.wait(), timeout=0.3)
await asyncio.wait_for(handler_started.wait(), timeout=0.3)
self.assertEqual(events[:2], ["ready", "handler"])
service._poll_conversations.assert_awaited_once_with(
initial=True,
defer_handlers=True,
)
# Account stop cancels its active deferred handler instead of
# leaving work detached from the service lifecycle.
await asyncio.wait_for(service.stop(), timeout=0.5)
self.assertTrue(handler_cancelled.is_set())
finally:
run_task.cancel()
await asyncio.gather(run_task, return_exceptions=True)
await service_module._shutdown_initial_unread_dispatcher()
async def test_initial_unread_dispatcher_has_process_wide_concurrency_limit(self):
dispatcher = service_module._InitialUnreadDispatcher(concurrency=2)
active = 0
maximum_active = 0
processed: list[int] = []
two_started = asyncio.Event()
release = asyncio.Event()
def make_service(account_id: int):
async def handle(_message):
nonlocal active, maximum_active
active += 1
maximum_active = max(maximum_active, active)
if active == 2:
two_started.set()
try:
await release.wait()
processed.append(account_id)
finally:
active -= 1
return SimpleNamespace(
account_id=account_id,
_running=True,
_handle_incoming=handle,
)
services = [make_service(index + 1) for index in range(6)]
try:
for service in services:
await dispatcher.submit(service, [{"unread_count": 1}])
await asyncio.wait_for(two_started.wait(), timeout=0.3)
await asyncio.sleep(0.03)
self.assertEqual(maximum_active, 2)
self.assertEqual(processed, [])
release.set()
await asyncio.wait_for(dispatcher.join(), timeout=0.5)
self.assertEqual(maximum_active, 2)
self.assertCountEqual(processed, range(1, 7))
finally:
await dispatcher.stop()
if __name__ == "__main__":
unittest.main()
@@ -152,6 +152,9 @@ class CookieCredentialLockTests(unittest.IsolatedAsyncioTestCase):
[
"lock-enter",
"authorize",
# Checks the pooled connection back in before cancel/stop,
# which may wait on an in-flight start.
"commit",
"cancel",
"stop",
"write-cookie",
@@ -245,6 +248,9 @@ class CookieCredentialLockTests(unittest.IsolatedAsyncioTestCase):
[
"lock-enter",
"authorize",
# Checks the pooled connection back in before cancel/stop,
# which may wait on an in-flight start.
"commit",
"cancel",
"stop",
"clear-cookie-file",
@@ -0,0 +1,63 @@
import threading
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from rpa_engine.credential import credential_egress_mismatch, validate_im_session
class CredentialResponsivenessTests(unittest.IsolatedAsyncioTestCase):
def test_legacy_egress_marker_comparison_is_diagnostic(self):
legacy = '{"cookies": []}'
self.assertFalse(credential_egress_mismatch(legacy, ""))
self.assertTrue(credential_egress_mismatch(legacy, "47.96.154.74"))
def test_stamped_egress_marker_comparison(self):
stamped = (
'{"cookies": [], '
'"credential_egress_public_ip": "47.96.154.74"}'
)
self.assertFalse(credential_egress_mismatch(stamped, "47.96.154.74"))
self.assertTrue(credential_egress_mismatch(stamped, "116.62.23.103"))
self.assertTrue(credential_egress_mismatch(stamped, ""))
async def test_uid_lookup_does_not_block_event_loop(self):
event_loop_thread_id = threading.get_ident()
lookup_thread_ids = []
def get_uid():
lookup_thread_ids.append(threading.get_ident())
return 123456
session = SimpleNamespace(
my_uid=0,
can_direct_im=lambda: True,
)
auth = SimpleNamespace(
get_uid=get_uid,
is_sign_ready=lambda: True,
)
with patch(
"rpa_engine.credential.DouyinAuth.from_im_session",
return_value=auth,
):
result = await validate_im_session(
session,
_bypass_global_limit=True,
)
self.assertTrue(result[0])
self.assertEqual(len(lookup_thread_ids), 1)
self.assertNotEqual(
lookup_thread_ids[0],
event_loop_thread_id,
"the synchronous UID lookup ran on the event-loop thread",
)
self.assertEqual(session.my_uid, 123456)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,369 @@
"""托管多个账号时的会话归属隔离回归测试。
复现的缺陷账号 A 的处理链路收到属于账号 B 的会话0:1:B:B的好友
resolve_peer_uid 把末段当成对方normalize_conversation_id 再拼成
0:1:A:B的好友于是账号 A 用自己的凭证把自动回复发给了账号 B 的好友
"""
from __future__ import annotations
import asyncio
import os
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock, patch
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ.setdefault("KEFU_DB_TYPE", "sqlite")
os.environ.setdefault("KEFU_DATABASE_URL", "")
os.environ.setdefault("KEFU_DB_PATH", str(BACKEND_DIR / "kefu.db"))
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im import hosted_registry
from rpa_engine.douyin_im import ws_client as ws_module
from rpa_engine.douyin_im.auth import DouyinAuth
from rpa_engine.douyin_im.conv_util import conversation_belongs_to
from rpa_engine.douyin_im.http_client import DouyinImHttpClient
from rpa_engine.douyin_im.service import DouyinImService
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.douyin_im.ws_client import DouyinImWsClient
ACCOUNT_A_UID = 7670159096859706425
ACCOUNT_B_UID = 7670157997767050299
PEER_OF_B = 66578464308
class ConversationOwnershipTests(unittest.TestCase):
def test_foreign_single_chat_is_rejected(self):
self.assertFalse(
conversation_belongs_to(
f"0:1:{ACCOUNT_B_UID}:{PEER_OF_B}", ACCOUNT_A_UID
)
)
def test_own_conversation_in_either_position(self):
self.assertTrue(
conversation_belongs_to(f"0:1:{ACCOUNT_A_UID}:{PEER_OF_B}", ACCOUNT_A_UID)
)
self.assertTrue(
conversation_belongs_to(f"0:1:{PEER_OF_B}:{ACCOUNT_A_UID}", ACCOUNT_A_UID)
)
def test_undecidable_shapes_pass_through(self):
# 缺 my_uid / 群聊 / 裸 UID:本来就判不了归属,保守放行
self.assertTrue(conversation_belongs_to(f"0:1:{ACCOUNT_B_UID}:{PEER_OF_B}", 0))
self.assertTrue(conversation_belongs_to("0:2:123:456", ACCOUNT_A_UID))
self.assertTrue(conversation_belongs_to(str(PEER_OF_B), ACCOUNT_A_UID))
self.assertTrue(conversation_belongs_to("", ACCOUNT_A_UID))
class ForeignMessageDropTests(unittest.IsolatedAsyncioTestCase):
def _service(self) -> DouyinImService:
service = DouyinImService(
session=DouyinImSession(cookies={"sessionid": "a"}, my_uid=ACCOUNT_A_UID),
match_reply=AsyncMock(return_value=["自动回复"]),
log_fn=AsyncMock(),
account_id=1,
)
service._running = True
return service
async def test_message_from_another_account_never_schedules_a_reply(self):
service = self._service()
service._resolve_peer_profile = AsyncMock(
return_value=("B 的好友", "", str(PEER_OF_B))
)
with patch(
"rpa_engine.douyin_im.service.system_logger.record", Mock()
) as record:
result = await service._prepare_incoming(
{
"conversation_id": f"0:1:{ACCOUNT_B_UID}:{PEER_OF_B}",
"sender_uid": str(PEER_OF_B),
"content": "在吗",
"server_message_id": "7665317099296081465",
}
)
self.assertIsNone(result)
service.match_reply.assert_not_awaited()
service.log_fn.assert_not_awaited()
self.assertEqual(service._conv_meta, {})
self.assertTrue(record.called)
async def test_own_message_is_still_processed(self):
service = self._service()
conv_id = f"0:1:{ACCOUNT_A_UID}:{PEER_OF_B}"
service._resolve_peer_profile = AsyncMock(
return_value=("我的好友", "", str(PEER_OF_B))
)
service._resolve_cooldown_seconds = AsyncMock(return_value=0)
service._resolve_reply_delay_seconds = AsyncMock(return_value=0)
service._send_auto_reply = AsyncMock()
with patch("rpa_engine.douyin_im.service.system_logger.record", Mock()):
send_reply = await service._prepare_incoming(
{
"conversation_id": conv_id,
"sender_uid": str(PEER_OF_B),
"content": "在吗",
"server_message_id": "7665317099296081466",
}
)
self.assertIsNotNone(send_reply)
service.match_reply.assert_awaited()
self.assertIn(conv_id, service._conv_meta)
class ForeignSendRefusalTests(unittest.IsolatedAsyncioTestCase):
async def test_send_refuses_a_conversation_owned_by_another_account(self):
client = DouyinImHttpClient(
DouyinImSession(cookies={"sessionid": "a"}, my_uid=ACCOUNT_A_UID),
account_id=1,
)
resolve_meta = AsyncMock()
with (
patch.object(
DouyinImHttpClient,
"_resolve_authoritative_uid",
return_value=ACCOUNT_A_UID,
),
patch.object(
DouyinImHttpClient, "resolve_conversation_meta", resolve_meta
),
patch("rpa_engine.douyin_im.http_client.system_logger.record", Mock()),
):
sent = await client.send_text_message(
f"0:1:{ACCOUNT_B_UID}:{PEER_OF_B}",
"你好",
_bypass_global_queue=True,
)
self.assertFalse(sent)
# 关键断言:拒发必须发生在解析 ticket / 真正写出去之前
resolve_meta.assert_not_awaited()
self.assertIn("不是本账号", client.last_error)
self.assertFalse(client.last_send_channel_retryable)
class ExpectedRecipientTests(unittest.IsolatedAsyncioTestCase):
"""手动发送必须打给调用方点选的那个人(昵称重复时会话可能匹配错)。"""
OTHER_PEER = 975976494279630
def _client(self) -> DouyinImHttpClient:
return DouyinImHttpClient(
DouyinImSession(cookies={"sessionid": "a"}, my_uid=ACCOUNT_A_UID),
account_id=1,
)
async def _send(self, client, conversation_id, expected_peer_uid):
resolve_meta = AsyncMock(return_value=("", "", ""))
with (
patch.object(
DouyinImHttpClient,
"_resolve_authoritative_uid",
return_value=ACCOUNT_A_UID,
),
# 本组用例只验收件人闸门,凭证是否齐全与它无关
patch.object(DouyinAuth, "is_sign_ready", return_value=True),
patch.object(
DouyinImHttpClient, "resolve_conversation_meta", resolve_meta
),
patch("rpa_engine.douyin_im.http_client.system_logger.record", Mock()),
):
sent = await client.send_text_message(
conversation_id,
"你好",
expected_peer_uid=expected_peer_uid,
_bypass_global_queue=True,
)
return sent, resolve_meta
async def test_refuses_when_the_conversation_points_at_someone_else(self):
client = self._client()
sent, resolve_meta = await self._send(
client,
f"0:1:{ACCOUNT_A_UID}:{self.OTHER_PEER}",
str(PEER_OF_B),
)
self.assertFalse(sent)
# 必须在解析 ticket / 发包之前就拒绝
resolve_meta.assert_not_awaited()
self.assertIn("发送目标与预期不一致", client.last_error)
self.assertFalse(client.last_send_channel_retryable)
async def test_allows_the_intended_recipient(self):
client = self._client()
sent, resolve_meta = await self._send(
client,
f"0:1:{ACCOUNT_A_UID}:{PEER_OF_B}",
str(PEER_OF_B),
)
# ticket 解析被 mock 成空 -> 发送仍会失败,但必须是「拿不到票据」而不是被闸门拦下
self.assertFalse(sent)
resolve_meta.assert_awaited()
self.assertNotIn("发送目标与预期不一致", client.last_error)
async def test_no_expectation_keeps_the_old_behaviour(self):
client = self._client()
_, resolve_meta = await self._send(
client, f"0:1:{ACCOUNT_A_UID}:{self.OTHER_PEER}", ""
)
resolve_meta.assert_awaited()
self.assertNotIn("发送目标与预期不一致", client.last_error)
class HostedPeerLoopTests(unittest.IsolatedAsyncioTestCase):
"""两个本系统托管的账号之间不得互相自动回复(无限回环 → 抖音风控)。"""
def _service(self) -> DouyinImService:
service = DouyinImService(
session=DouyinImSession(cookies={"sessionid": "a"}, my_uid=ACCOUNT_A_UID),
match_reply=AsyncMock(return_value=["自动回复"]),
log_fn=AsyncMock(),
account_id=1,
)
service._running = True
service._resolve_cooldown_seconds = AsyncMock(return_value=0)
service._resolve_reply_delay_seconds = AsyncMock(return_value=0)
return service
def tearDown(self):
hosted_registry.unregister(ACCOUNT_B_UID)
async def _incoming_from(self, service, peer_uid: int, message_id: str):
service._resolve_peer_profile = AsyncMock(
return_value=("对方", "", str(peer_uid))
)
with patch("rpa_engine.douyin_im.service.system_logger.record", Mock()):
return await service._prepare_incoming(
{
"conversation_id": f"0:1:{ACCOUNT_A_UID}:{peer_uid}",
"sender_uid": str(peer_uid),
"content": "在吗",
"server_message_id": message_id,
}
)
async def test_no_auto_reply_to_another_hosted_account(self):
hosted_registry.register(ACCOUNT_B_UID)
service = self._service()
result = await self._incoming_from(service, ACCOUNT_B_UID, "1")
self.assertIsNone(result)
service.match_reply.assert_not_awaited()
# 消息本身照常入库,只是标记为未回复
statuses = [
call.kwargs.get("status") for call in service.log_fn.await_args_list
]
self.assertIn("received", statuses)
self.assertIn("ignored", statuses)
async def test_ordinary_follower_still_gets_a_reply(self):
hosted_registry.register(ACCOUNT_B_UID)
service = self._service()
service._send_auto_reply = AsyncMock()
result = await self._incoming_from(service, PEER_OF_B, "2")
self.assertIsNotNone(result)
service.match_reply.assert_awaited()
class FrontierDeviceExclusivityTests(unittest.IsolatedAsyncioTestCase):
"""同一个 frontier 设备号同时只允许一个账号建连。"""
WS_URL = (
"wss://frontier-im.douyin.com/ws/v2?fpid=9&device_id=987654321&"
"token=shared-token"
)
def setUp(self):
ws_module._FRONTIER_DEVICE_OWNERS.clear()
def tearDown(self):
ws_module._FRONTIER_DEVICE_OWNERS.clear()
def _client(self, account_id: int) -> DouyinImWsClient:
client = DouyinImWsClient(
DouyinImSession(cookies={"sessionid": "s"}, ws_urls=[self.WS_URL]),
AsyncMock(),
account_id=account_id,
)
client._running = True
client._task = SimpleNamespace(done=lambda: False)
return client
def test_second_account_is_denied_while_the_first_holds_the_device(self):
first = self._client(11)
second = self._client(12)
self.assertTrue(first._claim_frontier_device(self.WS_URL))
self.assertFalse(second._claim_frontier_device(self.WS_URL))
self.assertEqual(second._blocked_device_owner_id, 11)
# 让出方不会被误标为已占用,重连时仍是 HTTP 轮询兜底
self.assertFalse(second.connected)
def test_device_is_taken_over_after_the_owner_stops(self):
first = self._client(11)
second = self._client(12)
self.assertTrue(first._claim_frontier_device(self.WS_URL))
first._running = False
first._release_frontier_device()
self.assertTrue(second._claim_frontier_device(self.WS_URL))
def test_same_account_reconnect_keeps_its_own_device(self):
client = self._client(11)
self.assertTrue(client._claim_frontier_device(self.WS_URL))
self.assertTrue(client._claim_frontier_device(self.WS_URL))
async def test_run_loop_does_not_open_a_second_connection(self):
owner = self._client(11)
self.assertTrue(owner._claim_frontier_device(self.WS_URL))
blocked = self._client(12)
blocked._prepare_url = AsyncMock(return_value=self.WS_URL)
run_connection = AsyncMock()
blocked._run_connection = run_connection
async def stop_after_first_backoff(_seconds):
blocked._running = False
with (
patch.object(ws_module, "_reconnect_delay", return_value=0.0),
patch.object(ws_module.system_logger, "record") as record,
patch.object(ws_module.asyncio, "sleep", stop_after_first_backoff),
):
await asyncio.wait_for(blocked._run_loop(self.WS_URL), timeout=1.0)
run_connection.assert_not_awaited()
self.assertFalse(blocked.connected)
self.assertTrue(record.called)
def test_url_without_device_id_is_not_blocked(self):
first = self._client(11)
second = self._client(12)
url = "wss://frontier-im.douyin.com/ws/v2?fpid=9&token=t"
self.assertTrue(first._claim_frontier_device(url))
self.assertTrue(second._claim_frontier_device(url))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,93 @@
from __future__ import annotations
import os
import sys
import unittest
from collections import namedtuple
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ["KEFU_DB_TYPE"] = "sqlite"
os.environ["KEFU_DATABASE_URL"] = ""
os.environ["KEFU_DB_PATH"] = str(BACKEND_DIR / "kefu.db")
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
import main
_AggregateRow = namedtuple(
"_AggregateRow",
(
"total_accounts",
"online_accounts",
"my_accounts",
"my_online_accounts",
),
)
class _AggregateResult:
def __init__(self, row):
self._row = row
def one(self):
return self._row
class DashboardAccountStatsTests(unittest.IsolatedAsyncioTestCase):
async def test_conditional_aggregate_maps_global_and_personal_counts(self):
row = _AggregateRow(
total_accounts=12,
online_accounts=5,
my_accounts=3,
my_online_accounts=2,
)
db = SimpleNamespace(
execute=AsyncMock(return_value=_AggregateResult(row))
)
user = SimpleNamespace(id=42, role="operator")
response = await main.get_dashboard_account_stats(db=db, user=user)
self.assertIsInstance(response, main.DashboardAccountStatsResponse)
self.assertEqual(response.total_accounts, 12)
self.assertEqual(response.online_accounts, 5)
self.assertEqual(response.my_accounts, 3)
self.assertEqual(response.my_online_accounts, 2)
db.execute.assert_awaited_once()
async def test_owner_scope_is_only_inside_personal_aggregates(self):
row = _AggregateRow(
total_accounts=8,
online_accounts=4,
my_accounts=2,
my_online_accounts=1,
)
db = SimpleNamespace(
execute=AsyncMock(return_value=_AggregateResult(row))
)
user = SimpleNamespace(id=73, role="viewer")
await main.get_dashboard_account_stats(db=db, user=user)
statement = db.execute.await_args.args[0]
sql = " ".join(str(statement).lower().split())
compiled_params = list(statement.compile().params.values())
# All roles receive the same global totals. The current user id may
# appear in CASE expressions for the two personal counters, but must
# never filter the entire aggregate query through a global WHERE.
self.assertIn("owner_id", sql)
self.assertGreaterEqual(sql.count("case when"), 3)
self.assertEqual(sql.count("accounts.owner_id"), 2)
self.assertIn(73, compiled_params)
self.assertNotIn(" where ", f" {sql} ")
self.assertEqual(db.execute.await_count, 1)
if __name__ == "__main__":
unittest.main()
@@ -171,7 +171,52 @@ class DesktopLoginSecUserIdTests(unittest.IsolatedAsyncioTestCase):
self.assertIs(result, response)
db.execute.assert_not_awaited()
build.assert_awaited_once_with(account, "cookie-json")
build.assert_awaited_once_with(
account,
"cookie-json",
runtime_check=False,
)
async def test_cookie_response_forwards_static_management_check(self):
account = self._account()
summary = {
"cookie_count": 2,
"cookie_valid": True,
"cookie_expired": False,
"reason": "ok",
"expires_at": None,
"key_names": ["sessionid"],
}
detail = {
"has_sessionid": True,
"sessionid": "sid",
"sessionid_ss": "",
"im_ready": False,
"im_status": "static",
"can_skip_browser": False,
"should_reset": False,
}
with (
patch.object(main, "cookie_summary", return_value=summary),
patch.object(
main,
"build_cookie_credential_detail",
new=AsyncMock(return_value=detail),
) as build_detail,
):
result = await main._build_cookie_response(
account,
"cookie-json",
runtime_check=False,
)
self.assertTrue(result.has_sessionid)
build_detail.assert_awaited_once_with(
"cookie-json",
None,
runtime_check=False,
)
async def test_legacy_cookie_request_is_guarded_for_existing_desktop_clients(self):
account = self._account()
+168
View File
@@ -0,0 +1,168 @@
from __future__ import annotations
import asyncio
import os
import sys
import time
import unittest
from pathlib import Path
from unittest.mock import AsyncMock, patch
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.dialects import mysql
from sqlalchemy.schema import CreateTable
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.egress_channels import (
EgressChannel,
EgressChannelUnavailable,
EgressSnapshot,
LocalAddress,
discover_egress_channels,
reset_egress_cache_for_tests,
resolve_send_channels,
)
from rpa_engine.source_bound_proxy import SourceBoundProxy
from models.db_migrate import migrate_accounts_table
from models.models import Account
class EgressChannelTests(unittest.IsolatedAsyncioTestCase):
def setUp(self):
reset_egress_cache_for_tests()
async def test_discovery_deduplicates_public_ip_and_keeps_bindable_source(self):
candidates = [
LocalAddress(None, "default", True),
LocalAddress("10.0.0.5", "eth0"),
LocalAddress("10.0.0.6", "eth0:1"),
]
async def probe(candidate):
public_ip = "203.0.113.10" if candidate.source_ip != "10.0.0.6" else "203.0.113.11"
return (
EgressChannel(
public_ip=public_ip,
source_ip=candidate.source_ip,
interface=candidate.interface,
is_default=candidate.is_default,
),
"",
)
with (
patch(
"rpa_engine.egress_channels.local_address_candidates",
return_value=candidates,
),
patch(
"rpa_engine.egress_channels._probe_local_address",
AsyncMock(side_effect=probe),
),
):
snapshot = await discover_egress_channels(force=True)
self.assertEqual([item.public_ip for item in snapshot.channels], ["203.0.113.10", "203.0.113.11"])
self.assertEqual(snapshot.channels[0].source_ip, "10.0.0.5")
self.assertTrue(snapshot.channels[0].is_default)
async def test_selected_channel_is_first_and_attempt_count_is_bounded(self):
snapshot = EgressSnapshot(
channels=(
EgressChannel("198.51.100.1", "10.0.0.1", "eth0", True),
EgressChannel("198.51.100.2", "10.0.0.2", "eth0:1"),
EgressChannel("198.51.100.3", "10.0.0.3", "eth0:2"),
),
errors=(),
detected_at=time.time(),
)
with patch(
"rpa_engine.egress_channels.discover_egress_channels",
AsyncMock(return_value=snapshot),
):
routes = await resolve_send_channels("198.51.100.2", 2)
self.assertEqual([item.public_ip for item in routes], ["198.51.100.2", "198.51.100.1"])
async def test_missing_selected_channel_fails_closed(self):
snapshot = EgressSnapshot(
channels=(EgressChannel("198.51.100.1", None, "default", True),),
errors=(),
detected_at=time.time(),
)
with patch(
"rpa_engine.egress_channels.discover_egress_channels",
AsyncMock(return_value=snapshot),
):
with self.assertRaises(EgressChannelUnavailable):
await resolve_send_channels("198.51.100.99", 2)
async def test_browser_proxy_binds_selected_source_address(self):
observed_peer = asyncio.get_running_loop().create_future()
async def target_handler(reader, writer):
if not observed_peer.done():
observed_peer.set_result(writer.get_extra_info("peername")[0])
payload = await reader.readexactly(4)
writer.write(payload)
await writer.drain()
writer.close()
await writer.wait_closed()
target = await asyncio.start_server(target_handler, "127.0.0.1", 0)
target_port = target.sockets[0].getsockname()[1]
proxy = await SourceBoundProxy("127.0.0.2").start()
writer = None
try:
reader, writer = await asyncio.open_connection(
"127.0.0.1",
int(proxy.server_url.rpartition(":")[2]),
)
writer.write(
(
f"CONNECT 127.0.0.1:{target_port} HTTP/1.1\r\n"
f"Host: 127.0.0.1:{target_port}\r\n\r\n"
).encode("ascii")
)
await writer.drain()
response = await reader.readuntil(b"\r\n\r\n")
self.assertIn(b"200 Connection Established", response)
writer.write(b"ping")
await writer.drain()
self.assertEqual(await reader.readexactly(4), b"ping")
self.assertEqual(await asyncio.wait_for(observed_peer, 1), "127.0.0.2")
finally:
if writer is not None:
writer.close()
await writer.wait_closed()
await proxy.close()
target.close()
await target.wait_closed()
class EgressMigrationTests(unittest.TestCase):
def test_mysql_accounts_uses_longtext_for_browser_payloads(self):
ddl = str(CreateTable(Account.__table__).compile(dialect=mysql.dialect()))
self.assertIn("cookie_data LONGTEXT", ddl)
self.assertIn("im_session_data LONGTEXT", ddl)
self.assertIn("qr_code_base64 LONGTEXT", ddl)
def test_old_accounts_table_receives_egress_columns(self):
engine = create_engine("sqlite:///:memory:")
with engine.begin() as connection:
connection.execute(text("CREATE TABLE accounts (id INTEGER PRIMARY KEY)"))
migrate_accounts_table(connection)
columns = {item["name"] for item in inspect(connection).get_columns("accounts")}
self.assertIn("egress_public_ip", columns)
self.assertIn("egress_auto_attempts", columns)
if __name__ == "__main__":
unittest.main()
+32
View File
@@ -0,0 +1,32 @@
from __future__ import annotations
import sys
import unittest
from pathlib import Path
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im.pb_decode import analyze_send_response
class AnalyzeSendResponseTests(unittest.TestCase):
def test_standalone_kick_json_is_not_decoded_as_protobuf(self):
result = analyze_send_response(b'{"decision": "KICK"}')
self.assertFalse(result["ok"])
self.assertEqual(result["decision"], "KICK")
self.assertEqual(result["summary"], "JSON响应 decision=KICK")
self.assertNotIn("unsupported wire type", result["summary"])
def test_malformed_json_still_returns_controlled_decode_summary(self):
result = analyze_send_response(b'{"decision":')
self.assertFalse(result["ok"])
self.assertIn("解码失败", result["summary"])
if __name__ == "__main__":
unittest.main()
+711
View File
@@ -0,0 +1,711 @@
"""接收私信链路的回归测试。
覆盖三个曾让托管中收不到抖音下发的私信的缺陷
1. frontier 长连接地址用了账号 UID 而不是设备号握手成功却订阅错地址
2. 浏览器本次登录抓到的真实 frontier 地址被 DB 里的旧地址挤掉
3. PushFrame 负载是 gzip / payloadType 不是 'pb' 时整帧被丢弃
另外覆盖会话列表接口被抖音拒绝时不能再伪装成收件箱为空
"""
from __future__ import annotations
import gzip
import json
import os
import sys
import unittest
from datetime import datetime
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ.setdefault("KEFU_DB_TYPE", "sqlite")
os.environ.setdefault("KEFU_DATABASE_URL", "")
os.environ.setdefault("KEFU_DB_PATH", str(BACKEND_DIR / "kefu.db"))
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im import frontier as frontier_module
from rpa_engine.douyin_im.auth import DouyinAuth
from rpa_engine.douyin_im.frontier import ensure_frontier_ws
from rpa_engine.douyin_im.http_client import DouyinImHttpClient
from rpa_engine.douyin_im.protocol import parse_ws_payload
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.douyin_im.static import Live_pb2, Response_pb2
from rpa_engine.playwright_worker import DouyinWorker
def _build_push_frame(
*,
conversation_id: str,
sender: int,
content: str,
message_type: int = 7,
server_message_id: int = 7665317099296081465,
encoding: str = "",
payload_type: str = "pb",
) -> bytes:
response = Response_pb2.Response()
notify = response.body.new_message_notify
notify.conversation_id = conversation_id
message = notify.message
message.conversation_id = conversation_id
message.conversation_type = 1
message.server_message_id = server_message_id
message.message_type = message_type
message.sender = sender
message.content = content
body = response.SerializeToString()
if encoding == "gzip":
body = gzip.compress(body)
frame = Live_pb2.PushFrame()
frame.seqId = 1
frame.service = 6
frame.method = 2
frame.payloadEncoding = encoding
frame.payloadType = payload_type
frame.payload = body
return frame.SerializeToString()
class FrontierAddressTests(unittest.TestCase):
"""frontier 按 device_id 寻址,不能用账号 UID 顶替。"""
def _session(self) -> DouyinImSession:
return DouyinImSession(
cookies={"sessionid": "6313fec013ec0000000000000000abcd"},
# query/user 返回的 id:本浏览器的设备注册号
device_id="7678285795559818786",
web_id="7678286623234475535",
my_uid=2609567359568155,
uid_verified=True,
)
def test_built_url_uses_device_id_not_account_uid(self):
session = self._session()
url = ensure_frontier_ws(session)
self.assertIsNotNone(url)
self.assertIn("device_id=7678285795559818786", url)
self.assertNotIn("device_id=2609567359568155", url)
def test_missing_device_id_falls_back_to_query_user_lookup(self):
session = self._session()
session.device_id = ""
session.web_id = ""
with patch.object(
frontier_module,
"fetch_device_id",
return_value="7678285795559818786",
) as fetch:
url = ensure_frontier_ws(session)
fetch.assert_called_once()
self.assertIn("device_id=7678285795559818786", url)
def test_proto_auth_keeps_device_id_for_verified_uid(self):
session = self._session()
auth = DouyinAuth.from_im_session(session)
self.assertEqual(auth.device_id, "7678285795559818786")
class CapturedFrontierUrlTests(unittest.IsolatedAsyncioTestCase):
"""浏览器本次抓到的真实地址必须压过 DB 里的旧地址。"""
async def test_fresh_browser_ws_url_wins_over_cached_url(self):
cached = (
"wss://frontier-im.douyin.com/ws/v2?aid=6383&device_platform=douyin_pc"
"&fpid=9&device_id=7678285795559818786&token=stale&access_key=stale"
)
captured = (
"wss://frontier31-normal.zijieapi.com/ws/v2?aid=6383&fpid=9"
"&device_id=7678285795559818786&access_key=realkey&token=realtoken"
)
saved = DouyinImSession(
cookies={"sessionid": "6313fec013ec0000000000000000abcd"},
ws_urls=[cached],
device_id="7678285795559818786",
my_uid=2609567359568155,
)
row = SimpleNamespace(
im_session_data=json.dumps(saved.to_dict()),
cookie_updated_at=datetime.utcnow(),
uid=None,
profile_updated_at=None,
)
result = MagicMock()
result.one_or_none.return_value = row
db = SimpleNamespace(execute=AsyncMock(return_value=result), close=AsyncMock())
worker = DouyinWorker(account_id=400, login_mode="browser")
worker.get_db = AsyncMock(return_value=db)
worker._load_raw_user_agent = AsyncMock(return_value="test-agent")
session = await worker._build_im_session_from_storage(
{"cookies": [{"name": "sessionid", "value": "6313fec013ec0000000000000000abcd"}]},
{"ws_urls": [captured]},
)
self.assertEqual(session.ws_urls[0], captured)
self.assertEqual(ensure_frontier_ws(session), captured)
class PushFramePayloadTests(unittest.TestCase):
"""PushFrame 负载的编码/类型不能再决定整帧被不被丢弃。"""
def test_plain_protobuf_frame_is_parsed(self):
raw = _build_push_frame(
conversation_id="0:1:869032150442612:2609567359568155",
sender=869032150442612,
content=json.dumps({"text": "你好", "aweType": 700}, ensure_ascii=False),
)
messages = parse_ws_payload(raw)
self.assertEqual(len(messages), 1)
self.assertEqual(messages[0]["sender_uid"], "869032150442612")
self.assertEqual(
messages[0]["conversation_id"], "0:1:869032150442612:2609567359568155"
)
def test_gzip_encoded_frame_is_parsed(self):
raw = _build_push_frame(
conversation_id="0:1:869032150442612:2609567359568155",
sender=869032150442612,
content=json.dumps({"text": "在吗", "aweType": 700}, ensure_ascii=False),
encoding="gzip",
)
messages = parse_ws_payload(raw)
self.assertEqual(len(messages), 1)
self.assertEqual(messages[0]["sender_uid"], "869032150442612")
def test_non_pb_payload_type_is_still_parsed(self):
# 现网 frontier 帧会带 payloadType='text/json';只认 'pb' 会整帧丢弃。
raw = _build_push_frame(
conversation_id="0:1:869032150442612:2609567359568155",
sender=869032150442612,
content=json.dumps({"text": "在吗", "aweType": 700}, ensure_ascii=False),
payload_type="text/json",
)
messages = parse_ws_payload(raw)
self.assertEqual(len(messages), 1)
self.assertEqual(messages[0]["sender_uid"], "869032150442612")
def test_empty_payload_control_frame_yields_no_message(self):
frame = Live_pb2.PushFrame()
frame.service = 6
frame.method = 2
frame.payloadEncoding = "utf-8"
frame.payloadType = "text/json"
self.assertEqual(parse_ws_payload(frame.SerializeToString()), [])
class InboxProtobufTests(unittest.TestCase):
"""imapi 只认 protobuf:解析真实响应形状,而不是 JSON。"""
@staticmethod
def _response(*, cmd=200, status=0, message="OK", messages=()):
from rpa_engine.douyin_im.http_client import _pb_int, _pb_msg, _pb_str
entries = b""
for m in messages:
entries += _pb_msg(
1,
_pb_str(1, m["conversation_id"])
+ _pb_int(3, m["server_message_id"])
+ _pb_int(5, m.get("conversation_short_id", 0))
+ _pb_int(6, m.get("message_type", 7))
+ _pb_int(7, m["sender"])
+ _pb_str(8, m.get("content", "")),
)
return (
_pb_int(1, cmd)
+ _pb_int(3, status)
+ _pb_str(4, message)
+ _pb_msg(6, _pb_msg(cmd, entries))
)
def test_status_is_read_from_the_protobuf_envelope(self):
from rpa_engine.douyin_im.http_client import _pb_response_status
ok = self._response()
self.assertEqual(_pb_response_status(ok), (0, "OK"))
rejected = self._response(status=1, message="unexepcted session length")
self.assertEqual(
_pb_response_status(rejected),
(1, "unexepcted session length"),
)
def test_message_bodies_are_extracted_from_the_inbox_response(self):
from rpa_engine.douyin_im.http_client import _pb_parse_inbox_messages
raw = self._response(
messages=[
{
"conversation_id": "0:1:2609567359568155:869032150442612",
"server_message_id": 7678140298052355621,
"conversation_short_id": 7654765893796266545,
"message_type": 7,
"sender": 869032150442612,
"content": '{"text":"你好"}',
}
]
)
parsed = _pb_parse_inbox_messages(raw, 200)
self.assertEqual(len(parsed), 1)
self.assertEqual(
parsed[0]["conversation_id"],
"0:1:2609567359568155:869032150442612",
)
self.assertEqual(parsed[0]["server_message_id"], "7678140298052355621")
self.assertEqual(parsed[0]["sender"], "869032150442612")
self.assertIn("你好", parsed[0]["content"])
def test_unrelated_protobuf_is_not_mistaken_for_a_message(self):
from rpa_engine.douyin_im.http_client import (
_pb_int, _pb_msg, _pb_parse_inbox_messages, _pb_str,
)
# 一段带字符串字段 1 但不是 conversation_id 的子消息
noise = _pb_msg(6, _pb_msg(200, _pb_msg(1, _pb_str(1, "not-a-conv") + _pb_int(3, 5))))
self.assertEqual(_pb_parse_inbox_messages(noise, 200), [])
def test_empty_inbox_yields_no_messages(self):
from rpa_engine.douyin_im.http_client import _pb_parse_inbox_messages
self.assertEqual(_pb_parse_inbox_messages(self._response(), 200), [])
class InboxCursorAndListTests(unittest.IsolatedAsyncioTestCase):
"""轮询用小窗口,用户点开列表用全量——同一个 cmd,只是游标不同。"""
def _client(self) -> DouyinImHttpClient:
session = DouyinImSession(
cookies={"sessionid": "s", "x_tt_token": "00" + "a" * 353},
device_id="7678285795559818786",
my_uid=2609567359568155,
)
return DouyinImHttpClient(session, account_id=405)
@staticmethod
def _cursor_from_payload(payload: bytes) -> int:
from rpa_engine.douyin_im.http_client import _pb_parse_fields
for fn, wt, val in _pb_parse_fields(payload):
if fn != 8 or wt != 2:
continue
for bfn, bwt, bval in _pb_parse_fields(val):
if bfn != 200 or bwt != 2:
continue
for cfn, cwt, cval in _pb_parse_fields(bval):
if cfn == 1 and cwt == 0:
return int(cval)
return -1
async def _capture_cursor(self, **kwargs) -> int:
from rpa_engine.douyin_im.http_client import _pb_int, _pb_str
client = self._client()
captured: dict = {}
async def fake_post(url, auth, payload, **_kw):
captured["payload"] = payload
return SimpleNamespace(
content=_pb_int(1, 200) + _pb_int(3, 0) + _pb_str(4, "OK"),
raise_for_status=lambda: None,
)
with patch.object(client, "_post_protobuf", fake_post):
await client.fetch_inbox_messages(**kwargs)
return self._cursor_from_payload(captured["payload"])
async def test_poll_window_sends_a_recent_microsecond_cursor(self):
import time as _time
cursor = await self._capture_cursor(lookback_seconds=1800)
now_us = int(_time.time() * 1_000_000)
self.assertGreater(cursor, 0)
# 游标应落在「大约半小时前」,允许几秒误差
self.assertLess(now_us - cursor, int(1810 * 1_000_000))
self.assertGreater(now_us - cursor, int(1790 * 1_000_000))
async def test_zero_lookback_means_no_cursor_not_now(self):
# lookback=0 若被算成 now,就只要「比此刻更新」的消息,永远是空列表。
self.assertEqual(await self._capture_cursor(lookback_seconds=0), 0)
@staticmethod
def _page(*, entries=(), next_cursor=0, has_more=False, cmd=200):
from rpa_engine.douyin_im.http_client import _pb_int, _pb_msg, _pb_str
inner = b""
for short_id, conv_id in entries:
inner += _pb_msg(6, _pb_int(1, short_id) + _pb_str(4, conv_id))
inner += _pb_int(2, next_cursor) + _pb_int(3, 1 if has_more else 0)
return (
_pb_int(1, cmd)
+ _pb_int(3, 0)
+ _pb_str(4, "OK")
+ _pb_msg(6, _pb_msg(cmd, inner))
)
async def test_paging_follows_the_cursor_and_dedupes_conversations(self):
client = self._client()
pages = [
self._page(
entries=[(1, "0:1:10001:20001"), (2, "0:1:10001:20002")],
next_cursor=111,
has_more=True,
),
self._page(
# 第二页重复一个、新增一个
entries=[(2, "0:1:10001:20002"), (3, "0:1:10001:20003")],
next_cursor=222,
has_more=True,
),
]
cursors: list[int] = []
async def fake_post(url, auth, payload, **_kw):
cursors.append(self._cursor_from_payload(payload))
return SimpleNamespace(
content=pages[len(cursors) - 1], raise_for_status=lambda: None
)
with patch.object(client, "_post_protobuf", fake_post):
await client.fetch_inbox_messages(lookback_seconds=0, max_pages=2)
self.assertEqual(cursors, [0, 111])
self.assertEqual(
[c["conversation_id"] for c in client._last_inbox_conversations],
["0:1:10001:20001", "0:1:10001:20002", "0:1:10001:20003"],
)
# 预算用完但抖音还说 has_more:必须承认列表不完整
self.assertTrue(client.inbox_truncated)
async def test_last_page_is_not_reported_as_truncated(self):
client = self._client()
page = self._page(entries=[(1, "0:1:10001:20001")], has_more=False)
async def fake_post(url, auth, payload, **_kw):
return SimpleNamespace(content=page, raise_for_status=lambda: None)
with patch.object(client, "_post_protobuf", fake_post):
await client.fetch_inbox_messages(lookback_seconds=0, max_pages=5)
self.assertFalse(client.inbox_truncated)
async def test_a_stalled_cursor_stops_paging(self):
client = self._client()
# 抖音回 has_more=1 但游标不前进:不能无限翻同一页
page = self._page(
entries=[(1, "0:1:10001:20001")], next_cursor=0, has_more=True
)
calls = {"n": 0}
async def fake_post(url, auth, payload, **_kw):
calls["n"] += 1
return SimpleNamespace(content=page, raise_for_status=lambda: None)
with patch.object(client, "_post_protobuf", fake_post):
await client.fetch_inbox_messages(lookback_seconds=0, max_pages=10)
self.assertEqual(calls["n"], 1)
async def test_conversations_without_recent_messages_still_listed(self):
client = self._client()
client.fetch_inbox_messages = AsyncMock(return_value=[])
client._last_inbox_conversations = [
{"conversation_id": "0:1:10001:20001", "conversation_short_id": "555"},
{"conversation_id": "0:1:10001:20002", "conversation_short_id": "666"},
]
rows = await client.get_conversations(
enrich_profiles=False, lookback_seconds=0
)
self.assertEqual(
{r["conversation_id"] for r in rows},
{"0:1:10001:20001", "0:1:10001:20002"},
)
self.assertEqual(
client.session.conv_meta["0:1:10001:20002"]["conversation_short_id"],
"666",
)
async def test_control_frames_never_become_a_conversation_preview(self):
client = self._client()
client.fetch_inbox_messages = AsyncMock(
return_value=[
{
"conversation_id": "0:1:10001:20001",
"server_message_id": "100",
"message_type": 7,
"sender": "20001",
"content": '{"text":"真实消息"}',
},
{
"conversation_id": "0:1:10001:20001",
"server_message_id": "200",
"message_type": 50001,
"sender": "20001",
"content": '{"command_type":6,"conversation_id":"0:1:10001:20001"}',
},
]
)
rows = await client.get_conversations(enrich_profiles=False)
# 控制帧 server_message_id 更大,但不能顶掉真实消息成为预览,
# 否则 _handle_incoming 会拿它去匹配自动回复。
self.assertEqual(len(rows), 1)
self.assertIn("真实消息", rows[0]["content"])
self.assertEqual(rows[0]["server_message_id"], "100")
class ReadRequestTokenTests(unittest.TestCase):
"""读接口的 Request.token 必须是 x_tt_token。
auth.ticket 时抖音照样回 status_code=0 "OK"但把调用方当匿名用户
正文恒为空收件箱没有消息完全无法区分是最难发现的那类故障
实测同一请求只换 tokenauth.ticket 73 字节 0 x_tt_token 113KB 47
"""
def _auth(self):
session = DouyinImSession(
cookies={
"sessionid": "6313fec013ec0000000000000000abcd",
"x_tt_token": "00" + "a" * 353,
},
device_id="7678285795559818786",
my_uid=2609567359568155,
)
return DouyinAuth.from_im_session(session)
def test_read_request_uses_x_tt_token(self):
from rpa_engine.douyin_im.proto_builder import ProtoBuilder
auth = self._auth()
request = ProtoBuilder.build_read_request(auth, 200)
self.assertEqual(request.token, "00" + "a" * 353)
self.assertNotEqual(request.token, auth.ticket)
def test_read_request_keeps_ticket_when_cookie_missing(self):
from rpa_engine.douyin_im.proto_builder import ProtoBuilder
session = DouyinImSession(
cookies={"sessionid": "6313fec013ec0000000000000000abcd"},
device_id="7678285795559818786",
my_uid=2609567359568155,
)
auth = DouyinAuth.from_im_session(session)
request = ProtoBuilder.build_read_request(auth, 200)
self.assertEqual(request.token, auth.ticket or "")
def test_send_request_is_left_on_the_normal_envelope(self):
from rpa_engine.douyin_im.proto_builder import ProtoBuilder
# 发送接口另有 bd-ticket-guard 签名且线上可用,不能顺手改掉它的 token。
auth = self._auth()
request = ProtoBuilder.build_normal_request(auth, 100)
self.assertEqual(request.token, auth.ticket or "")
class AuthoritativeUidTests(unittest.TestCase):
"""imapi 响应字段 13 是抖音认定的本账号 IM uid。"""
def test_response_uid_corrects_a_wrong_my_uid(self):
from rpa_engine.douyin_im.http_client import _pb_int, _pb_str
session = DouyinImSession(
cookies={"sessionid": "s"},
my_uid=7678285795559818786, # 误把 frontier 设备号当成了 IM uid
)
client = DouyinImHttpClient(session, account_id=404)
content = (
_pb_int(1, 200)
+ _pb_int(3, 0)
+ _pb_str(4, "OK")
+ _pb_int(13, 2609567359568155)
)
client._adopt_authoritative_uid(content)
self.assertEqual(session.my_uid, 2609567359568155)
self.assertTrue(session.uid_verified)
def test_matching_uid_is_left_alone(self):
from rpa_engine.douyin_im.http_client import _pb_int, _pb_str
session = DouyinImSession(cookies={"sessionid": "s"}, my_uid=2609567359568155)
client = DouyinImHttpClient(session, account_id=404)
content = _pb_int(1, 200) + _pb_int(3, 0) + _pb_str(4, "OK") + _pb_int(
13, 2609567359568155
)
client._adopt_authoritative_uid(content)
self.assertEqual(session.my_uid, 2609567359568155)
self.assertFalse(session.uid_verified)
class ConversationListRejectionTests(unittest.IsolatedAsyncioTestCase):
"""接口被拒不能再伪装成「收件箱为空」。"""
def _client(self) -> DouyinImHttpClient:
session = DouyinImSession(
cookies={"sessionid": "6313fec013ec0000000000000000abcd"},
device_id="7678285795559818786",
my_uid=2609567359568155,
)
return DouyinImHttpClient(session, account_id=401)
async def test_rejected_protobuf_response_is_reported(self):
client = self._client()
raw = InboxProtobufTests._response(
status=1, message="unexepcted session length"
)
post = AsyncMock(
return_value=SimpleNamespace(
content=raw, raise_for_status=lambda: None
)
)
with (
patch.object(client, "_post_protobuf", post),
patch(
"rpa_engine.douyin_im.auth.DouyinAuth.from_im_session",
return_value=SimpleNamespace(source_ip=""),
),
patch(
"rpa_engine.douyin_im.proto_builder.ProtoBuilder.build_normal_request",
return_value=SimpleNamespace(SerializeToString=lambda: b""),
),
patch.object(client, "_report_conversation_list_rejected") as report,
):
self.assertEqual(await client.fetch_inbox_messages(), [])
post.assert_awaited_once()
report.assert_called_once_with("unexepcted session length")
async def test_rejection_marks_the_endpoint_unsupported(self):
client = self._client()
client._report_conversation_list_rejected("unexepcted session length")
self.assertTrue(client.conversation_list_unsupported)
self.assertIn("unexepcted session length", client.last_error)
async def test_empty_but_successful_inbox_is_not_reported_as_failure(self):
client = self._client()
client.fetch_inbox_messages = AsyncMock(return_value=[])
with patch.object(
client, "_report_conversation_list_rejected"
) as report:
self.assertEqual(
await client.get_conversations(enrich_profiles=False), []
)
report.assert_not_called()
self.assertEqual(client.last_error, "")
self.assertFalse(client.conversation_list_unsupported)
class LoggedOutDetectionTests(unittest.IsolatedAsyncioTestCase):
"""抖音回「用户未登录」时必须明确报出来,不能当成资料接口抖动。"""
def test_status_code_8_is_reported_as_logged_out(self):
from rpa_engine import account_profile as ap
auth = SimpleNamespace(cookie={}, msToken="t", get_uid=lambda: "938334054809296")
payloads = [
{"status_code": 0, "user_uid": "938334054809296"},
{"status_code": 8, "status_msg": "用户未登录", "user": None},
{"status_code": 8, "status_msg": "用户未登录", "user": None},
]
responses = [SimpleNamespace(json=lambda v=v: v) for v in payloads]
with (
patch.object(ap, "_build_auth", return_value=(auth, "ua")),
patch.object(ap.requests, "get", side_effect=responses),
patch.object(ap, "generate_a_bogus", return_value="a-bogus"),
patch.object(ap, "generate_webid", return_value="web-id"),
patch.object(ap, "_requests_proxies", return_value=None),
):
detail = ap.fetch_douyin_profile_detail_sync("cookie-json", "ua")
self.assertTrue(detail["logged_out"])
self.assertFalse(detail["fetched"])
self.assertIn("用户未登录", detail["message"])
async def test_hosting_reports_logged_out_once(self):
worker = DouyinWorker(account_id=403, login_mode="im_direct")
with patch(
"rpa_engine.playwright_worker.system_logger.record"
) as record:
await worker._report_douyin_logged_out("抖音返回「用户未登录」")
await worker._report_douyin_logged_out("抖音返回「用户未登录」")
record.assert_called_once()
self.assertEqual(record.call_args.kwargs["level"], "error")
class ReconciliationBackoffTests(unittest.IsolatedAsyncioTestCase):
"""被抖音拒绝过的接口不能每 120 秒再白打一次。"""
def _service(self):
from rpa_engine.douyin_im.service import DouyinImService
return DouyinImService(
session=DouyinImSession(
cookies={"sessionid": "6313fec013ec0000000000000000abcd"},
device_id="7678285795559818786",
my_uid=2609567359568155,
),
match_reply=AsyncMock(return_value=[]),
log_fn=AsyncMock(),
account_id=402,
)
async def test_second_poll_skips_a_rejected_endpoint(self):
service = self._service()
client = MagicMock()
client.get_conversations = AsyncMock(return_value=[])
client.conversation_list_unsupported = True
client.__aenter__ = AsyncMock(return_value=client)
client.__aexit__ = AsyncMock(return_value=False)
with patch(
"rpa_engine.douyin_im.service.DouyinImHttpClient",
return_value=client,
):
self.assertEqual(await service._poll_conversations(), [])
self.assertTrue(service._conversation_list_unsupported)
# 第二轮完全不再构造 HTTP 客户端 / 发请求
self.assertEqual(await service._poll_conversations(), [])
client.get_conversations.assert_awaited_once()
if __name__ == "__main__":
unittest.main()
+281
View File
@@ -0,0 +1,281 @@
from __future__ import annotations
import json
import logging.handlers
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from sqlalchemy import text
from sqlalchemy.pool import StaticPool
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from models.db_config import DatabaseConfig, create_database_engine, engine_kwargs_for_url
from models.db_migrate import migrate_message_logs_table
from models.models import MessageLog
from rpa_engine.douyin_im import protocol
from rpa_engine.douyin_im.static import Live_pb2, Response_pb2
from utils import system_logger
from utils.log_limits import (
TRUNCATION_MARKER,
bound_message_log_content,
bound_raw_message_log_content,
)
class LogLimitTests(unittest.TestCase):
def tearDown(self):
system_logger.clear()
def test_system_log_caps_persisted_and_console_detail(self):
with patch.dict(os.environ, {"KEFU_SYSTEM_LOG_MAX_CHARS": "512"}):
with self.assertLogs("douyin_im.system", level="INFO") as captured:
entry = system_logger.record("event", "x" * 5000)
self.assertLessEqual(len(entry["detail"]), 512)
self.assertIn(TRUNCATION_MARKER.strip(), entry["detail"])
self.assertLess(len(captured.output[0]), 700)
def test_oversized_media_log_remains_valid_compact_json(self):
payload = json.dumps(
{
"type": "sticker",
"url": "https://example.invalid/sticker.webp",
"text": "x" * 20000,
"unused_blob": "y" * 20000,
},
ensure_ascii=False,
)
with patch.dict(os.environ, {"KEFU_MESSAGE_LOG_MAX_CHARS": "4096"}):
bounded = bound_message_log_content(payload)
decoded = json.loads(bounded)
self.assertEqual(decoded["type"], "sticker")
self.assertEqual(decoded["url"], "https://example.invalid/sticker.webp")
self.assertTrue(decoded["_log_truncated"])
self.assertNotIn("unused_blob", decoded)
self.assertLessEqual(len(bounded), 4096)
def test_message_model_validator_caps_all_insert_paths(self):
with patch.dict(os.environ, {"KEFU_MESSAGE_LOG_MAX_CHARS": "2048"}):
row = MessageLog(message_content="m" * 10000, reply_content="r" * 10000)
self.assertLessEqual(len(row.message_content), 2048)
self.assertLessEqual(len(row.reply_content), 2048)
def test_raw_message_log_is_bounded(self):
with patch.dict(os.environ, {"KEFU_RAW_MESSAGE_LOG_MAX_CHARS": "4096"}):
bounded = bound_raw_message_log_content("z" * 20000)
self.assertLessEqual(len(bounded), 4096)
self.assertIn(TRUNCATION_MARKER.strip(), bounded)
class SqliteIoTests(unittest.IsolatedAsyncioTestCase):
async def test_short_memory_url_uses_one_static_connection(self):
kwargs = engine_kwargs_for_url("sqlite+aiosqlite://")
self.assertIs(kwargs["poolclass"], StaticPool)
self.assertNotIn("pool_size", kwargs)
engine = create_database_engine(
DatabaseConfig(
db_type="sqlite",
database_url="sqlite+aiosqlite://",
)
)
try:
async with engine.begin() as conn:
await conn.execute(text("CREATE TABLE memory_probe (id INTEGER)"))
async with engine.begin() as conn:
await conn.execute(text("INSERT INTO memory_probe VALUES (1)"))
count = (
await conn.execute(text("SELECT count(*) FROM memory_probe"))
).scalar_one()
self.assertEqual(count, 1)
finally:
await engine.dispose()
async def test_file_sqlite_uses_bounded_pool_and_wal_pragmas(self):
kwargs = engine_kwargs_for_url("sqlite+aiosqlite:///example.db")
self.assertEqual(kwargs["pool_size"], 5)
self.assertEqual(kwargs["max_overflow"], 0)
self.assertEqual(kwargs["connect_args"]["timeout"], 30.0)
with tempfile.TemporaryDirectory() as temp_dir:
db_path = Path(temp_dir) / "io.db"
engine = create_database_engine(
DatabaseConfig(db_type="sqlite", db_path=str(db_path))
)
try:
async with engine.connect() as conn:
journal_mode = (await conn.execute(text("PRAGMA journal_mode"))).scalar_one()
synchronous = (await conn.execute(text("PRAGMA synchronous"))).scalar_one()
busy_timeout = (await conn.execute(text("PRAGMA busy_timeout"))).scalar_one()
self.assertEqual(str(journal_mode).lower(), "wal")
self.assertEqual(synchronous, 1)
self.assertEqual(busy_timeout, 30000)
finally:
await engine.dispose()
async def test_existing_log_tables_receive_composite_indexes(self):
with tempfile.TemporaryDirectory() as temp_dir:
db_path = Path(temp_dir) / "migration.db"
engine = create_database_engine(
DatabaseConfig(db_type="sqlite", db_path=str(db_path))
)
try:
async with engine.begin() as conn:
await conn.execute(
text(
"CREATE TABLE message_logs ("
"id INTEGER PRIMARY KEY, account_id INTEGER, "
"created_at DATETIME, sender_avatar TEXT, status VARCHAR(50))"
)
)
await conn.execute(
text(
"CREATE TABLE received_message_logs ("
"id INTEGER PRIMARY KEY, account_id INTEGER, "
"created_at DATETIME)"
)
)
await conn.execute(
text(
"CREATE TABLE system_logs ("
"id INTEGER PRIMARY KEY, account_id INTEGER, "
"created_at DATETIME)"
)
)
await conn.run_sync(migrate_message_logs_table)
async with engine.connect() as conn:
message_indexes = {
row[1]
for row in (await conn.execute(text("PRAGMA index_list(message_logs)"))).all()
}
received_indexes = {
row[1]
for row in (
await conn.execute(text("PRAGMA index_list(received_message_logs)"))
).all()
}
system_indexes = {
row[1]
for row in (await conn.execute(text("PRAGMA index_list(system_logs)"))).all()
}
latest_plan = " ".join(
str(row[-1])
for row in (
await conn.execute(
text(
"EXPLAIN QUERY PLAN SELECT * FROM message_logs "
"ORDER BY created_at DESC LIMIT 50"
)
)
).all()
)
status_plan = " ".join(
str(row[-1])
for row in (
await conn.execute(
text(
"EXPLAIN QUERY PLAN SELECT count(*) FROM message_logs "
"WHERE status = 'replied'"
)
)
).all()
)
account_plan = " ".join(
str(row[-1])
for row in (
await conn.execute(
text(
"EXPLAIN QUERY PLAN SELECT * FROM message_logs "
"WHERE account_id = 1 ORDER BY created_at DESC LIMIT 50"
)
)
).all()
)
system_plan = " ".join(
str(row[-1])
for row in (
await conn.execute(
text(
"EXPLAIN QUERY PLAN SELECT * FROM system_logs "
"WHERE account_id = 1 ORDER BY created_at DESC LIMIT 50"
)
)
).all()
)
self.assertIn("ix_message_logs_account_created_at", message_indexes)
self.assertIn("ix_message_logs_created_at", message_indexes)
self.assertIn("ix_message_logs_status_account_id", message_indexes)
self.assertIn(
"ix_received_message_logs_account_created_at",
received_indexes,
)
self.assertIn("ix_system_logs_account_created_at", system_indexes)
self.assertIn("ix_message_logs_created_at", latest_plan)
self.assertIn("ix_message_logs_status_account_id", status_plan)
self.assertIn("ix_message_logs_account_created_at", account_plan)
self.assertIn("ix_system_logs_account_created_at", system_plan)
finally:
await engine.dispose()
class WebSocketDebugTests(unittest.TestCase):
def _frame(self, *, message_type: int, content: str) -> bytes:
response = Response_pb2.Response()
message = response.body.new_message_notify.message
message.conversation_id = "0:1:200:100"
message.server_message_id = 123
message.message_type = message_type
message.sender = 200
message.content = content
frame = Live_pb2.PushFrame()
frame.payloadType = "pb"
frame.payload = response.SerializeToString()
return frame.SerializeToString()
def test_control_frame_is_filtered_before_debug_writer(self):
with patch.object(protocol, "_dump_ws_message") as dump:
result = protocol.parse_ws_payload(
self._frame(message_type=50001, content='{"command_type":6}')
)
self.assertEqual(result, [])
dump.assert_not_called()
def test_debug_writer_uses_non_blocking_rotating_queue(self):
# Inspect construction without writing chat data to the repository.
with tempfile.TemporaryDirectory() as temp_dir:
old_path = protocol._WS_DEBUG_PATH
protocol._WS_DEBUG_PATH = str(Path(temp_dir) / "ws.log")
protocol._WS_DEBUG_LOGGER = None
try:
with patch.dict(os.environ, {"KEFU_WS_DEBUG": "1"}):
protocol._dump_ws_message(1, "conv", "hello")
logger = protocol._WS_DEBUG_LOGGER
self.assertIsNotNone(logger)
self.assertIsInstance(logger.handlers[0], logging.handlers.QueueHandler)
self.assertIsInstance(
logger._kefu_rotating_handler,
logging.handlers.RotatingFileHandler,
)
finally:
logger = protocol._WS_DEBUG_LOGGER
if logger is not None:
logger._kefu_queue_listener.stop()
logger._kefu_rotating_handler.close()
logger.handlers.clear()
protocol._WS_DEBUG_LOGGER = None
protocol._WS_DEBUG_PATH = old_path
if __name__ == "__main__":
unittest.main()
+136 -4
View File
@@ -30,36 +30,127 @@ class AccountReplyQueueTests(unittest.IsolatedAsyncioTestCase):
await asyncio.sleep(0.002)
self.assertEqual(queue.pending_count, 0)
async def test_one_account_runs_three_jobs_at_successive_fifo_slots(self):
async def test_immediate_if_idle_runs_first_now_then_successive_fifo_slots(self):
queue = await self._start_queue(account_id=101)
interval = 0.05
loop = asyncio.get_running_loop()
started_at = loop.time()
calls: list[tuple[int, float]] = []
finished = asyncio.Event()
first_started = asyncio.Event()
release_first = asyncio.Event()
def callback_for(index: int):
async def callback() -> None:
calls.append((index, loop.time() - started_at))
if index == 0:
first_started.set()
await release_first.wait()
if len(calls) == 3:
finished.set()
return callback
for index in range(3):
await queue.enqueue(interval, callback_for(index), description=str(index))
await queue.enqueue(
interval,
callback_for(0),
description="0",
immediate_if_idle=True,
)
await asyncio.wait_for(first_started.wait(), timeout=0.1)
for index in (1, 2):
await queue.enqueue(
interval,
callback_for(index),
description=str(index),
immediate_if_idle=True,
)
release_first.set()
await asyncio.wait_for(finished.wait(), timeout=0.75)
await self._wait_until_idle(queue)
self.assertEqual([index for index, _ in calls], [0, 1, 2])
elapsed = [timestamp for _, timestamp in calls]
for timestamp, expected in zip(elapsed, (interval, interval * 2, interval * 3)):
for timestamp, expected in zip(elapsed, (0, interval, interval * 2)):
self.assertGreaterEqual(timestamp, expected - 0.015)
self.assertLess(timestamp, expected + 0.15)
self.assertGreaterEqual(elapsed[1] - elapsed[0], interval - 0.02)
self.assertGreaterEqual(elapsed[2] - elapsed[1], interval - 0.02)
async def test_immediate_if_idle_does_not_bypass_active_send(self):
queue = await self._start_queue(account_id=102)
interval = 1.0
first_started = asyncio.Event()
release_first = asyncio.Event()
second_started = asyncio.Event()
async def first_callback() -> None:
first_started.set()
await release_first.wait()
async def second_callback() -> None:
second_started.set()
await queue.enqueue(
interval,
first_callback,
description="first",
immediate_if_idle=True,
)
await asyncio.wait_for(first_started.wait(), timeout=0.1)
await queue.enqueue(
interval,
second_callback,
description="second",
immediate_if_idle=True,
)
snapshot = await queue.snapshot()
self.assertEqual([item["status"] for item in snapshot], ["sending", "waiting"])
self.assertEqual(snapshot[0]["interval_seconds"], 0)
self.assertEqual(snapshot[1]["interval_seconds"], int(interval))
release_first.set()
await asyncio.sleep(0.02)
self.assertFalse(second_started.is_set())
async def test_immediate_if_idle_resets_after_queue_drains(self):
queue = await self._start_queue(account_id=103)
interval = 1.0
first_finished = asyncio.Event()
second_started = asyncio.Event()
release_second = asyncio.Event()
async def first_callback() -> None:
first_finished.set()
async def second_callback() -> None:
second_started.set()
await release_second.wait()
await queue.enqueue(
interval,
first_callback,
description="first wave",
immediate_if_idle=True,
)
await asyncio.wait_for(first_finished.wait(), timeout=0.1)
await self._wait_until_idle(queue)
await queue.enqueue(
interval,
second_callback,
description="second wave",
immediate_if_idle=True,
)
await asyncio.wait_for(second_started.wait(), timeout=0.1)
snapshot = await queue.snapshot()
self.assertEqual(len(snapshot), 1)
self.assertEqual(snapshot[0]["status"], "sending")
self.assertEqual(snapshot[0]["interval_seconds"], 0)
release_second.set()
async def test_separate_account_queues_reach_first_slot_without_blocking(self):
first_queue = await self._start_queue(account_id=201)
second_queue = await self._start_queue(account_id=202)
@@ -194,6 +285,47 @@ class AccountReplyQueueTests(unittest.IsolatedAsyncioTestCase):
new_due = datetime.fromisoformat(new["scheduled_at"]).timestamp()
self.assertAlmostEqual(old_due - new_due, interval, delta=0.05)
async def test_send_now_on_zero_slot_does_not_claim_later_jobs_shifted(self):
queue = await self._start_queue(account_id=512)
interval = 1.0
async def noop() -> None:
pass
# Both enqueues complete without yielding to the consumer, preserving
# the narrow management-API window where the zero-slot first job is
# still waiting and can be selected by send-now.
await queue.enqueue(
interval,
noop,
description="immediate first",
immediate_if_idle=True,
)
await queue.enqueue(
interval,
noop,
description="scheduled second",
immediate_if_idle=True,
)
before = await queue.snapshot()
second_before = next(
item for item in before if item["description"] == "scheduled second"
)
result = await queue.send_now(before[0]["job_id"])
after = await queue.snapshot()
second_after = next(
item for item in after if item["description"] == "scheduled second"
)
self.assertEqual(result["status"], "accepted")
self.assertEqual(result["shifted_count"], 0)
second_due_before = datetime.fromisoformat(
second_before["scheduled_at"]
).timestamp()
second_due_after = datetime.fromisoformat(second_after["scheduled_at"]).timestamp()
self.assertAlmostEqual(second_due_after, second_due_before, delta=0.01)
async def test_send_now_middle_runs_first_and_only_shifts_jobs_behind_it(self):
queue = await self._start_queue(account_id=503)
interval = 0.12
+21
View File
@@ -84,6 +84,27 @@ class ReplyQueueApiTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(response.total_pending, 1)
self.assertEqual([item.account_id for item in response.items], [1])
async def test_summary_with_account_ids_only_scans_requested_page(self):
service_one = _FakeService(1, [_queue_item(1)])
service_two = _FakeService(2, [_queue_item(2, "job-2")])
service_one.get_reply_queue_snapshot = AsyncMock(return_value=service_one.items)
service_two.get_reply_queue_snapshot = AsyncMock(return_value=service_two.items)
main.manager.workers = {
1: SimpleNamespace(is_running=True, _im_service=service_one),
2: SimpleNamespace(is_running=True, _im_service=service_two),
}
response = await main.get_reply_queue_summaries(
account_ids="2",
db=object(),
user=SimpleNamespace(id=1, role="admin"),
)
service_one.get_reply_queue_snapshot.assert_not_awaited()
service_two.get_reply_queue_snapshot.assert_awaited_once()
self.assertEqual(response.total_pending, 1)
self.assertEqual([item.account_id for item in response.items], [2])
async def test_offline_account_detail_returns_empty_snapshot(self):
account = SimpleNamespace(id=1, reply_delay_seconds=0)
with (
+118 -2
View File
@@ -18,12 +18,16 @@ if str(BACKEND_DIR) not in sys.path:
from auth.system_settings import SystemSettingsData, set_cached_settings
from rpa_engine.douyin_im.service import DouyinImService
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.douyin_im import service as service_module
from rpa_engine.playwright_worker import DouyinWorker
class _RecordingQueue:
def __init__(self) -> None:
self.jobs: list[tuple[float, object, str, dict, frozenset[str]]] = []
self.jobs: list[
tuple[float, object, str, dict, frozenset[str], bool]
] = []
async def enqueue(
self,
@@ -33,11 +37,19 @@ class _RecordingQueue:
details=None,
merge_key="",
merge_keys=None,
immediate_if_idle=False,
) -> int:
keys = merge_keys if merge_keys is not None else [merge_key]
normalized_keys = frozenset(str(key) for key in keys if str(key or "").strip())
self.jobs.append(
(delay_seconds, callback, description, dict(details or {}), normalized_keys)
(
delay_seconds,
callback,
description,
dict(details or {}),
normalized_keys,
bool(immediate_if_idle),
)
)
return len(self.jobs)
@@ -75,6 +87,7 @@ class _RecordingQueue:
job[2],
merged_details,
frozenset(job[4] | incoming_keys),
job[5],
)
return {
"status": "merged",
@@ -103,6 +116,107 @@ def _build_service(delay_seconds: int = 60):
class ReplyQueueIntegrationTests(unittest.IsolatedAsyncioTestCase):
async def test_kick_does_not_replay_through_browser_fallback(self):
callback = AsyncMock()
fallback = AsyncMock(return_value=(True, "must not run"))
session = DouyinImSession(cookies={"sessionid": "test"}, my_uid=999)
service = DouyinImService(
session=session,
match_reply=AsyncMock(),
log_fn=AsyncMock(),
account_id=1,
send_fallback=fallback,
on_session_invalid=callback,
)
service._running = True
kicked_http = SimpleNamespace(
send_text_message=AsyncMock(return_value=False),
last_error="decision=KICK",
last_send_needs_refresh=False,
)
context = AsyncMock()
context.__aenter__.return_value = kicked_http
context.__aexit__.return_value = None
with (
unittest.mock.patch.object(
service_module, "DouyinImHttpClient", return_value=context
),
unittest.mock.patch(
"rpa_engine.douyin_im.service.system_logger.record", Mock()
),
):
sent, _ = await service._send_text("0:1:999:123", "hello")
self.assertFalse(sent)
fallback.assert_not_awaited()
callback.assert_awaited_once()
async def test_fresh_session_replaces_send_and_ws_state_atomically(self):
current = DouyinImSession(
cookies={"sessionid": "old"},
my_uid=999,
conv_meta={"old": {"ticket": "one"}},
)
current.egress_public_ip = "203.0.113.10"
current.egress_source_ip = "10.0.0.10"
current.egress_auto_attempts = 2
service = DouyinImService(
session=current,
match_reply=AsyncMock(),
log_fn=AsyncMock(),
account_id=1,
)
service._ws_client = SimpleNamespace(session=current)
fresh = DouyinImSession(
cookies={"sessionid": "fresh"},
my_uid=999,
conv_meta={"new": {"ticket": "two"}},
)
await service.replace_session(fresh)
self.assertIs(service.session, fresh)
self.assertIs(service._ws_client.session, fresh)
self.assertEqual(service.session.cookies["sessionid"], "fresh")
self.assertEqual(set(service.session.conv_meta), {"old", "new"})
self.assertEqual(service.session.egress_public_ip, "203.0.113.10")
self.assertEqual(service.session.egress_source_ip, "10.0.0.10")
self.assertEqual(service.session.egress_auto_attempts, 2)
async def test_kick_response_takes_account_offline_immediately(self):
callback = AsyncMock()
service, _, _ = _build_service()
service.on_session_invalid = callback
with unittest.mock.patch(
"rpa_engine.douyin_im.service.system_logger.record", Mock()
):
await service._note_session_invalid(
"抖音安全网关返回 decision=KICK,当前登录态已失效"
)
self.assertFalse(service._running)
self.assertTrue(service._session_invalid_fired)
callback.assert_awaited_once()
self.assertIn("decision=KICK", callback.await_args.args[0])
async def test_invalid_request_still_requires_two_consecutive_failures(self):
callback = AsyncMock()
service, _, _ = _build_service()
service.on_session_invalid = callback
with unittest.mock.patch(
"rpa_engine.douyin_im.service.system_logger.record", Mock()
):
await service._note_session_invalid("INVALID_REQUEST")
self.assertTrue(service._running)
callback.assert_not_awaited()
await service._note_session_invalid("INVALID_REQUEST")
self.assertFalse(service._running)
callback.assert_awaited_once()
async def test_same_message_from_ws_and_poll_is_queued_once(self):
service, match_reply, _ = _build_service()
message = {
@@ -123,6 +237,8 @@ class ReplyQueueIntegrationTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(len(service._reply_queue.jobs), 1)
self.assertEqual(match_reply.await_count, 1)
self.assertEqual(service._reply_queue.jobs[0][0], 60)
self.assertTrue(service._reply_queue.jobs[0][5])
details = service._reply_queue.jobs[0][3]
self.assertEqual(details["sender_name"], "张三")
self.assertEqual(details["conversation_id"], "conv-1")
+191
View File
@@ -0,0 +1,191 @@
from __future__ import annotations
import os
import sys
import unittest
from pathlib import Path
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import sessionmaker
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ["KEFU_DB_TYPE"] = "sqlite"
os.environ["KEFU_DATABASE_URL"] = "sqlite+aiosqlite:///:memory:"
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from models.models import Base, Role, User # noqa: E402
from auth.passwords import hash_password # noqa: E402
from auth.permissions import ( # noqa: E402
ACCOUNTS_WRITE,
ALL_PERMISSIONS,
MENU_USERS,
USERS_MANAGE,
expand_paired_permissions,
)
from auth.role_service import ( # noqa: E402
create_role,
delete_role,
guard_last_admin_change,
seed_builtin_roles,
update_role,
)
from auth.roles import has_permission, is_admin, permissions_for_role # noqa: E402
from auth import router as auth_router # noqa: E402
class RolesRbacTests(unittest.IsolatedAsyncioTestCase):
async def asyncSetUp(self):
self.engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with self.engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
self.session_factory = sessionmaker(
self.engine, class_=AsyncSession, expire_on_commit=False
)
self.db = self.session_factory()
await seed_builtin_roles(self.db)
async def asyncTearDown(self):
await self.db.close()
await self.engine.dispose()
async def test_seed_creates_builtin_roles(self):
result = await self.db.execute(select(Role))
codes = {row.code for row in result.scalars().all()}
self.assertEqual(codes, {"admin", "operator", "viewer"})
self.assertTrue(is_admin("admin"))
self.assertFalse(is_admin("operator"))
self.assertIn(MENU_USERS, permissions_for_role("admin"))
self.assertNotIn(MENU_USERS, permissions_for_role("operator"))
self.assertFalse(has_permission("viewer", ACCOUNTS_WRITE))
async def test_custom_role_crud(self):
created = await create_role(
self.db,
code="ops_leader",
label="运营主管",
description="可管账号",
permissions=[MENU_USERS, ACCOUNTS_WRITE],
)
self.assertEqual(created.code, "ops_leader")
self.assertTrue(has_permission("ops_leader", ACCOUNTS_WRITE))
self.assertFalse(is_admin("ops_leader"))
updated = await update_role(
self.db,
"ops_leader",
label="主管",
permissions=[ACCOUNTS_WRITE],
)
self.assertEqual(updated.label, "主管")
self.assertFalse(has_permission("ops_leader", MENU_USERS))
await delete_role(self.db, "ops_leader")
self.assertFalse(has_permission("ops_leader", ACCOUNTS_WRITE))
async def test_cannot_delete_system_role(self):
with self.assertRaises(HTTPException) as caught:
await delete_role(self.db, "operator")
self.assertEqual(caught.exception.status_code, 400)
async def test_admin_permissions_always_full(self):
await update_role(
self.db,
"admin",
label="管理员",
permissions=[ACCOUNTS_WRITE],
)
self.assertEqual(permissions_for_role("admin"), list(ALL_PERMISSIONS))
async def test_me_payload_includes_permissions(self):
user = User(
username="u1",
password_hash=hash_password("password1"),
display_name="U1",
role="operator",
is_active=True,
email_verified=True,
)
self.db.add(user)
await self.db.commit()
await self.db.refresh(user)
payload = await auth_router._build_user_response(self.db, user)
self.assertEqual(payload.role_label, "运营")
self.assertFalse(payload.is_admin)
self.assertIn("accounts.create", payload.permissions)
self.assertNotIn(MENU_USERS, payload.permissions)
self.assertNotIn("data.scope_all", payload.permissions)
async def test_legacy_accounts_write_implies_granular(self):
created = await create_role(
self.db,
code="legacy_ops",
label="旧版运营",
description=None,
permissions=[ACCOUNTS_WRITE, "menu.accounts"],
)
self.assertTrue(has_permission("legacy_ops", "accounts.start"))
self.assertTrue(has_permission("legacy_ops", "accounts.cookie"))
self.assertIn(ACCOUNTS_WRITE, created.permissions)
async def test_data_scope_all_for_custom_role(self):
from auth.roles import has_global_scope
await create_role(
self.db,
code="auditor",
label="审计",
description=None,
permissions=["menu.accounts", "data.scope_all"],
)
self.assertTrue(has_global_scope("auditor"))
self.assertFalse(has_global_scope("operator"))
self.assertTrue(has_global_scope("admin"))
async def test_last_admin_cannot_be_demoted(self):
admin = User(
username="admin1",
password_hash=hash_password("password1"),
role="admin",
is_active=True,
email_verified=True,
)
self.db.add(admin)
await self.db.commit()
await self.db.refresh(admin)
with self.assertRaises(HTTPException) as caught:
await guard_last_admin_change(self.db, user=admin, new_role="operator")
self.assertEqual(caught.exception.status_code, 400)
async def test_menu_action_pairs_expand(self):
expanded = expand_paired_permissions([MENU_USERS])
self.assertIn(MENU_USERS, expanded)
self.assertIn(USERS_MANAGE, expanded)
created = await create_role(
self.db,
code="hr_desk",
label="人事台",
description=None,
permissions=[MENU_USERS],
)
self.assertIn(USERS_MANAGE, created.permissions)
self.assertNotIn("menu.roles", created.permissions)
self.assertNotIn("roles.manage", created.permissions)
roles_only = await create_role(
self.db,
code="role_editor",
label="角色编辑",
description=None,
permissions=["menu.roles"],
)
self.assertIn("roles.manage", roles_only.permissions)
self.assertNotIn(USERS_MANAGE, roles_only.permissions)
if __name__ == "__main__":
unittest.main()
+285 -29
View File
@@ -1,9 +1,11 @@
from __future__ import annotations
import json
import os
import sys
import unittest
from contextlib import asynccontextmanager
from datetime import datetime, timedelta
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
@@ -18,13 +20,114 @@ if str(BACKEND_DIR) not in sys.path:
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine import account_profile as account_profile_module
from rpa_engine.credential import build_im_session_from_storage
from rpa_engine.playwright_worker import DouyinWorker
class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
async def test_im_direct_missing_sec_user_id_exits_before_online_state(self):
async def test_current_profile_uid_overrides_browser_runtime_uid(self):
now = datetime.utcnow()
row = SimpleNamespace(
im_session_data=None,
cookie_updated_at=now,
uid="2609567359568155",
profile_updated_at=now,
)
result = MagicMock()
result.one_or_none.return_value = row
db = SimpleNamespace(
execute=AsyncMock(return_value=result),
close=AsyncMock(),
)
worker = DouyinWorker(account_id=300, login_mode="im_direct")
worker.get_db = AsyncMock(return_value=db)
worker._load_raw_user_agent = AsyncMock(return_value="test-agent")
storage = {
"cookies": [{"name": "sessionid", "value": "test-session"}],
"my_uid": 7678285795559818786,
"origins": [
{
"localStorage": [
{
"name": "web_runtime_security_uid",
"value": "7678285795559818786",
}
]
}
],
}
session = await worker._build_im_session_from_storage(storage)
self.assertEqual(session.my_uid, 2609567359568155)
# device_id 必须与权威 UID 同步(protobuf 发送时 device_id 优先取
# session.device_id,残留的浏览器采集值会导致 device_id != my_uid -> KICK
self.assertEqual(session.device_id, "2609567359568155")
self.assertTrue(session.uid_verified)
db.close.assert_awaited_once()
async def test_stale_profile_uid_does_not_override_new_cookie(self):
now = datetime.utcnow()
row = SimpleNamespace(
im_session_data=None,
cookie_updated_at=now,
uid="2609567359568155",
profile_updated_at=now - timedelta(seconds=1),
)
result = MagicMock()
result.one_or_none.return_value = row
db = SimpleNamespace(
execute=AsyncMock(return_value=result),
close=AsyncMock(),
)
worker = DouyinWorker(account_id=300, login_mode="im_direct")
worker.get_db = AsyncMock(return_value=db)
worker._load_raw_user_agent = AsyncMock(return_value="test-agent")
session = await worker._build_im_session_from_storage(
{
"cookies": [{"name": "sessionid", "value": "test-session"}],
"my_uid": 7678285795559818786,
}
)
self.assertEqual(session.my_uid, 7678285795559818786)
self.assertFalse(session.uid_verified)
def test_persisted_verified_uid_wins_in_generic_session_builder(self):
saved = DouyinImSession(
cookies={"sessionid": "test-session"},
my_uid=2609567359568155,
device_id="7678285795559818786",
uid_verified=True,
)
session = build_im_session_from_storage(
{
"cookies": [{"name": "sessionid", "value": "test-session"}],
"my_uid": 7678285795559818786,
"origins": [
{
"localStorage": [
{
"name": "web_runtime_security_uid",
"value": "7678285795559818786",
}
]
}
],
},
json.dumps(saved.to_dict()),
)
self.assertEqual(session.my_uid, 2609567359568155)
# device_id 同步为已核验 UID,避免凭证残留设备号导致 device_id != my_uid
self.assertEqual(session.device_id, "2609567359568155")
self.assertTrue(session.uid_verified)
async def test_im_direct_missing_sec_user_id_continues_im_hosting(self):
worker = DouyinWorker(account_id=301, login_mode="im_direct")
worker._require_sec_user_id = AsyncMock(return_value=False)
worker._best_effort_sec_user_id = AsyncMock(return_value="")
worker._load_user_agent = AsyncMock(return_value="test-agent")
im_session = DouyinImSession(
cookies={"sessionid": "test-session"},
@@ -42,34 +145,48 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
{"cookies": [{"name": "sessionid", "value": "test-session"}]}
)
self.assertFalse(started)
self.assertIn("sec_user_id", reason)
worker._require_sec_user_id.assert_awaited_once()
require_call = worker._require_sec_user_id.await_args
self.assertTrue(require_call.kwargs["refresh_if_missing"])
self.assertTrue(require_call.kwargs["refresh_if_stale"])
self.assertTrue(started)
self.assertEqual(reason, "")
worker._best_effort_sec_user_id.assert_awaited_once_with(
refresh_if_missing=True,
refresh_if_stale=True,
)
worker._build_im_session_from_storage.assert_awaited_once()
validate.assert_awaited_once_with(im_session)
worker._persist_im_session.assert_not_awaited()
worker._run_im_direct_service.assert_not_awaited()
worker._persist_im_session.assert_awaited_once()
worker._run_im_direct_service.assert_awaited_once_with(im_session)
async def test_running_guard_precedes_disabled_follow_welcome_setting(self):
async def test_disabled_follow_welcome_does_not_require_identity(self):
worker = DouyinWorker(account_id=302, login_mode="im_direct")
worker.is_running = True
worker._im_service = SimpleNamespace(_running=True)
worker._require_sec_user_id = AsyncMock(return_value=False)
worker._best_effort_sec_user_id = AsyncMock()
worker._refresh_follow_welcome_config = AsyncMock(
return_value=(False, "", "")
)
worker.get_db = AsyncMock()
await worker.follow_welcome_tick()
worker._require_sec_user_id.assert_awaited_once()
require_call = worker._require_sec_user_id.await_args
self.assertFalse(require_call.kwargs.get("refresh_if_missing", False))
# The identity guard must run before the account's follow-welcome flag
# is queried. Otherwise accounts with that feature disabled could stay
# hosted indefinitely without a sec_user_id.
worker._refresh_follow_welcome_config.assert_awaited_once_with()
worker._best_effort_sec_user_id.assert_not_awaited()
worker.get_db.assert_not_awaited()
async def test_cached_disabled_follow_setting_skips_missing_identity(self):
worker = DouyinWorker(account_id=311, login_mode="im_direct")
worker.is_running = True
worker._im_service = SimpleNamespace(_running=True)
worker._follow_config_loaded = True
worker._refresh_follow_welcome_config = AsyncMock(
return_value=(False, "", "")
)
worker._best_effort_sec_user_id = AsyncMock(return_value="")
await worker.follow_welcome_tick()
worker._refresh_follow_welcome_config.assert_awaited_once_with()
worker._best_effort_sec_user_id.assert_not_awaited()
async def test_blank_sec_user_id_is_missing_and_stops_hosting(self):
worker = DouyinWorker(account_id=303, login_mode="im_direct")
worker._load_sec_user_id = AsyncMock(return_value=" ")
@@ -280,6 +397,141 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
self.assertIn("托管已自动退出", status_call.kwargs["error_msg"])
record_system_log.assert_called_once()
def test_profile_payload_uid_outranks_query_user_and_cookie_uid(self):
"""资料接口的 UID 必须压过 query/user 的 user_uid 和 cookie 兜底。
实测同一个 Cookiequery/user 返回 user_uid=938334054809296
而账号资料 UID 2609567359568155以前前者先占位导致账号卡片
一直显示用户938334054809296反查 sec_user_id 也永远失败
"""
auth = SimpleNamespace(
cookie={},
msToken="test-ms-token",
get_uid=lambda: "938334054809296",
)
payloads = [
# 1) query/user:只有 user_uid,属于弱兜底
{"status_code": 0, "user_uid": "938334054809296"},
# 2) user/profile/self:权威账号资料
{
"status_code": 0,
"user": {
"uid": "2609567359568155",
"nickname": "凤的心",
"sec_uid": "MS4wLjABAAAA-real-sec-user-id",
"unique_id": "39688250979",
"aweme_count": 50,
"follower_count": 154,
},
},
]
responses = [SimpleNamespace(json=lambda value=value: value) for value in payloads]
with (
patch.object(
account_profile_module, "_build_auth",
return_value=(auth, "test-agent"),
),
patch.object(
account_profile_module.requests, "get", side_effect=responses,
),
patch.object(
account_profile_module, "generate_a_bogus", return_value="a-bogus",
),
patch.object(
account_profile_module, "generate_webid", return_value="web-id",
),
patch.object(
account_profile_module, "_requests_proxies", return_value=None,
),
):
detail = account_profile_module.fetch_douyin_profile_detail_sync(
"cookie-json", "test-agent",
)
self.assertEqual(detail["uid"], "2609567359568155")
self.assertEqual(detail["nickname"], "凤的心")
self.assertTrue(detail["fetched"])
async def test_sync_writes_one_identity_to_both_account_and_profile(self):
"""卡片与详细资料必须来自同一次抓取,不能各抓一次抓出两个身份。"""
detail = {
"uid": "2609567359568155",
"nickname": "凤的心",
"avatar_url": "https://example.test/avatar.png",
"unique_id": "39688250979",
"signature": "",
"sec_user_id": "MS4wLjABAAAA-real-sec-user-id",
"sec_user_id_status": "found",
"video_count": 50,
"follower_count": 154,
"following_count": 162,
"total_favorited": 612,
"favoriting_count": 0,
"fetched": True,
"message": "",
}
account = SimpleNamespace(
id=1,
username="用户938334054809296",
douyin_uid="938334054809296",
avatar_url=None,
user_agent="test-agent",
)
profile = SimpleNamespace(
account_id=1, uid=None, nickname=None, avatar_url=None,
unique_id=None, signature=None, sec_user_id=None,
follower_count=None, following_count=None, total_favorited=None,
favoriting_count=None, video_count=None, synced_at=None,
sync_message=None,
)
profile_result = MagicMock()
profile_result.scalar_one_or_none.return_value = profile
username_result = MagicMock()
username_result.scalar_one_or_none.return_value = None
db = SimpleNamespace(
execute=AsyncMock(side_effect=lambda stmt: (
username_result if "accounts.username" in str(stmt).lower()
or "username" in str(stmt).lower() else profile_result
)),
add=MagicMock(),
commit=AsyncMock(),
refresh=AsyncMock(),
)
with (
patch.object(
account_profile_module,
"fetch_douyin_profile_detail_with_sec_user_id",
new=AsyncMock(return_value=detail),
),
patch.object(
account_profile_module,
"fetch_douyin_user_videos",
new=AsyncMock(return_value={"videos": [], "message": "无作品"}),
),
patch.object(
account_profile_module,
"fetch_douyin_profile",
new=AsyncMock(side_effect=AssertionError("must not re-fetch")),
),
patch.object(
account_profile_module,
"load_account_profile_from_db",
new=AsyncMock(return_value={}),
),
):
await account_profile_module.sync_account_profile_to_db(
db, account, "cookie-json",
)
# 同一个身份同时写进 accounts 行和 account_profile_details 行
self.assertEqual(account.douyin_uid, "2609567359568155")
self.assertEqual(account.username, "凤的心")
self.assertEqual(account.avatar_url, "https://example.test/avatar.png")
self.assertEqual(profile.uid, "2609567359568155")
self.assertEqual(profile.nickname, "凤的心")
async def test_cookie_uid_without_valid_profile_payload_stays_unknown(self):
auth = SimpleNamespace(
cookie={},
@@ -473,7 +725,7 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
),
patch.object(
account_profile_module,
"apply_douyin_profile",
"apply_profile_to_account",
apply_profile,
),
patch.object(
@@ -501,19 +753,19 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
"profile endpoint temporarily unavailable",
)
async def test_browser_login_missing_sec_user_id_closes_browser_and_skips_im(self):
async def test_browser_login_missing_sec_user_id_still_starts_im(self):
worker = DouyinWorker(account_id=305, login_mode="browser")
worker.is_running = True
events: list[str] = []
async def reject_identity(*_args, **_kwargs):
events.append("require-sec-user-id")
return False
return ""
worker._load_user_agent = AsyncMock(return_value="test-agent")
worker._probe_existing_login = AsyncMock(return_value=True)
worker._finalize_login_session = AsyncMock()
worker._require_sec_user_id = AsyncMock(side_effect=reject_identity)
worker._best_effort_sec_user_id = AsyncMock(side_effect=reject_identity)
worker._setup_im_network_listener = AsyncMock()
worker._navigate_to_message_center = AsyncMock()
worker._harvest_im_credentials = AsyncMock()
@@ -578,8 +830,8 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
)
worker._finalize_login_session.assert_awaited_once_with()
worker._require_sec_user_id.assert_awaited_once()
require_call = worker._require_sec_user_id.await_args
worker._best_effort_sec_user_id.assert_awaited_once()
require_call = worker._best_effort_sec_user_id.await_args
self.assertTrue(require_call.kwargs["force_refresh"])
self.assertEqual(
events,
@@ -590,8 +842,10 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
worker._harvest_im_credentials.assert_awaited_once_with(timeout=25)
worker._persist_cookies.assert_awaited_once_with()
worker._build_im_session.assert_awaited_once_with()
worker._persist_im_session.assert_not_awaited()
worker._run_im_direct_service.assert_not_awaited()
# sec_user_id 只服务关注欢迎语;缺它不能阻断私信托管,
# 否则浏览器登录后账号立刻下线、永远不会自动回复。
worker._persist_im_session.assert_awaited_once()
worker._run_im_direct_service.assert_awaited_once_with(im_session)
worker._close_browser_only.assert_awaited_once_with()
async def test_browser_login_with_sec_user_id_continues_to_im(self):
@@ -600,7 +854,7 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
worker._load_user_agent = AsyncMock(return_value="test-agent")
worker._probe_existing_login = AsyncMock(return_value=True)
worker._finalize_login_session = AsyncMock()
worker._require_sec_user_id = AsyncMock(return_value=True)
worker._best_effort_sec_user_id = AsyncMock(return_value="sec-uid-306")
worker._setup_im_network_listener = AsyncMock()
worker._navigate_to_message_center = AsyncMock()
worker._harvest_im_credentials = AsyncMock()
@@ -657,8 +911,10 @@ class SecUserIdGuardTests(unittest.IsolatedAsyncioTestCase):
},
)
worker._require_sec_user_id.assert_awaited_once()
self.assertTrue(worker._require_sec_user_id.await_args.kwargs["force_refresh"])
worker._best_effort_sec_user_id.assert_awaited_once()
self.assertTrue(
worker._best_effort_sec_user_id.await_args.kwargs["force_refresh"]
)
worker._setup_im_network_listener.assert_awaited_once_with()
worker._navigate_to_message_center.assert_awaited_once_with()
worker._harvest_im_credentials.assert_awaited_once_with(timeout=25)
@@ -18,8 +18,12 @@ if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im import http_client as http_client_module
from rpa_engine.douyin_im.auth import DouyinAuth
from rpa_engine.douyin_im.frontier import ensure_frontier_ws
from rpa_engine.douyin_im.http_client import DouyinImHttpClient
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.egress_channels import EgressChannel
from rpa_engine import playwright_worker as playwright_worker_module
from rpa_engine.playwright_worker import DouyinWorker
@@ -32,6 +36,75 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase):
)
return DouyinImHttpClient(session, account_id=account_id)
def test_query_user_does_not_replace_existing_im_uid(self):
client = self._make_client()
auth = SimpleNamespace(
source_ip="",
get_uid=MagicMock(return_value=938334054809296),
)
resolved = client._resolve_authoritative_uid(auth)
self.assertEqual(resolved, 10001)
self.assertEqual(client.session.my_uid, 10001)
auth.get_uid.assert_not_called()
def test_verified_uid_does_not_replace_the_runtime_device_id(self):
# device_id 是 query/user 返回的设备注册号,my_uid 是账号 UID。
# 拿 my_uid 顶替 device_id 会让 frontier 订阅到另一个地址:握手照样
# 成功,却永远收不到这个账号的私信。
session = DouyinImSession(
cookies={"sessionid": "test-session"},
my_uid=2609567359568155,
device_id="7678285795559818786",
web_id="7678286623234475535",
uid_verified=True,
)
auth = DouyinAuth.from_im_session(session)
self.assertEqual(auth.device_id, "7678285795559818786")
self.assertEqual(session.device_id, "7678285795559818786")
def test_captured_tokenless_browser_frontier_url_is_kept(self):
url = (
"wss://frontier100-normal.zijieapi.com/ws/v2?aid=6383&"
"device_platform=web&fpid=9&device_id=7678285795559818786&"
"access_key=0123456789abcdef0123456789abcdef&version_code=fws_1.0.0"
)
session = DouyinImSession(
cookies={"sessionid": "test-session"},
ws_urls=[url],
my_uid=2609567359568155,
device_id="7678285795559818786",
)
resolved = ensure_frontier_ws(session)
self.assertEqual(resolved, url)
self.assertEqual(session.ws_urls, [url])
def test_non_im_frontier_product_is_rejected(self):
url = (
"wss://frontier100-normal.zijieapi.com/ws/v2?aid=6383&"
"device_platform=web&fpid=971&device_id=7678285795559818786&"
"access_key=0123456789abcdef0123456789abcdef"
)
session = DouyinImSession(
cookies={"sessionid": "test-session"},
ws_urls=[url],
my_uid=2609567359568155,
device_id="7678285795559818786",
uid_verified=True,
)
resolved = ensure_frontier_ws(session)
self.assertIn("frontier-im.douyin.com", resolved)
self.assertIn("fpid=9", resolved)
self.assertIn("device_id=7678285795559818786", resolved)
self.assertNotIn("fpid=971", resolved)
async def test_public_entry_submits_the_whole_send_to_outbound_queue(self):
client = self._make_client(account_id=73)
submit = AsyncMock(return_value=True)
@@ -85,6 +158,7 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase):
last_send_meta=queued_meta,
last_error="credential expired",
last_send_needs_refresh=True,
last_send_channel_retryable=False,
last_request_debug="response status=401",
)
queued_context = MagicMock()
@@ -114,15 +188,18 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase):
"0:1:10001:20002",
"queued hello",
conversation_short_id="short-before-send",
expected_peer_uid="20002",
)
self.assertFalse(sent)
submit.assert_awaited_once()
queued_factory.assert_called_once_with(client.session, account_id=88)
# 收件人期望必须原样传给真正写出去的那个 client:排队调度层不能把它吃掉
queued_client.send_text_message.assert_awaited_once_with(
"0:1:10001:20002",
"queued hello",
conversation_short_id="short-before-send",
expected_peer_uid="20002",
_bypass_global_queue=True,
)
self.assertEqual(client.last_send_meta, queued_meta)
@@ -131,8 +208,194 @@ class SendTextMessageEntryTests(unittest.IsolatedAsyncioTestCase):
self.assertTrue(client.last_send_needs_refresh)
self.assertEqual(client.last_request_debug, "response status=401")
async def test_retryable_network_failure_switches_channels_serially(self):
client = self._make_client(account_id=89)
client.session.egress_auto_attempts = 2
routes = [
EgressChannel("198.51.100.10", "10.0.0.10", "eth0", True),
EgressChannel("198.51.100.11", "10.0.0.11", "eth0:1", False),
]
first = SimpleNamespace(
send_text_message=AsyncMock(return_value=False),
last_send_meta={},
last_error="connect timeout",
last_send_needs_refresh=False,
last_send_channel_retryable=True,
last_request_debug="first route",
)
second = SimpleNamespace(
send_text_message=AsyncMock(return_value=True),
last_send_meta={"conv": {"ticket": "ok"}},
last_error="",
last_send_needs_refresh=False,
last_send_channel_retryable=False,
last_request_debug="second route",
)
def context_for(value):
context = MagicMock()
context.__aenter__ = AsyncMock(return_value=value)
context.__aexit__ = AsyncMock(return_value=None)
return context
queued_factory = MagicMock(side_effect=[context_for(first), context_for(second)])
async def execute_submission(account_id, operation, description=""):
self.assertEqual(account_id, 89)
return await operation()
with (
patch(
"rpa_engine.douyin_im.traffic_control.submit_outbound",
AsyncMock(side_effect=execute_submission),
),
patch(
"rpa_engine.douyin_im.http_client.resolve_send_channels",
AsyncMock(return_value=routes),
),
patch.object(http_client_module, "DouyinImHttpClient", queued_factory),
patch.object(http_client_module.system_logger, "record"),
):
sent = await client.send_text_message("0:1:10001:20002", "hello")
self.assertTrue(sent)
self.assertEqual(queued_factory.call_count, 2)
self.assertEqual(queued_factory.call_args_list[0].kwargs["source_ip"], "10.0.0.10")
self.assertEqual(queued_factory.call_args_list[1].kwargs["source_ip"], "10.0.0.11")
first.send_text_message.assert_awaited_once()
second.send_text_message.assert_awaited_once()
self.assertEqual(client.last_request_debug, "second route")
async def test_kick_never_switches_public_channels(self):
client = self._make_client(account_id=90)
client.session.egress_auto_attempts = 2
routes = [
EgressChannel("198.51.100.10", "10.0.0.10", "eth0", True),
EgressChannel("198.51.100.11", "10.0.0.11", "eth1", False),
]
kicked = SimpleNamespace(
send_text_message=AsyncMock(return_value=False),
last_send_meta={},
last_error="decision=KICK",
last_send_needs_refresh=False,
last_send_channel_retryable=False,
last_request_debug="terminal kick",
)
context = MagicMock()
context.__aenter__ = AsyncMock(return_value=kicked)
context.__aexit__ = AsyncMock(return_value=None)
queued_factory = MagicMock(return_value=context)
async def execute_submission(account_id, operation, description=""):
return await operation()
with (
patch(
"rpa_engine.douyin_im.traffic_control.submit_outbound",
AsyncMock(side_effect=execute_submission),
),
patch(
"rpa_engine.douyin_im.http_client.resolve_send_channels",
AsyncMock(return_value=routes),
),
patch.object(http_client_module, "DouyinImHttpClient", queued_factory),
):
sent = await client.send_text_message("0:1:10001:20002", "hello")
self.assertFalse(sent)
queued_factory.assert_called_once()
kicked.send_text_message.assert_awaited_once()
class WorkerLifecycleTests(unittest.IsolatedAsyncioTestCase):
async def test_browser_launch_uses_selected_source_proxy(self):
launch = AsyncMock(return_value="browser")
pw = SimpleNamespace(chromium=SimpleNamespace(launch=launch))
source_proxy = AsyncMock(return_value={"server": "http://127.0.0.1:43210"})
with (
patch.object(
playwright_worker_module,
"ensure_browser_display",
AsyncMock(),
),
patch.object(
playwright_worker_module,
"playwright_proxy_for_source",
source_proxy,
),
patch.object(playwright_worker_module, "playwright_proxy") as global_proxy,
):
browser = await playwright_worker_module._launch_chromium(
pw,
["--no-sandbox"],
headless=True,
source_ip="10.0.0.6",
)
self.assertEqual(browser, "browser")
source_proxy.assert_awaited_once_with("10.0.0.6")
global_proxy.assert_not_called()
launch.assert_awaited_once_with(
headless=True,
args=["--no-sandbox"],
proxy={"server": "http://127.0.0.1:43210"},
)
async def test_virtual_display_starts_before_playwright_driver(self):
ensure_display = AsyncMock()
async def start_driver():
ensure_display.assert_awaited_once_with(False)
return "playwright-driver"
manager = SimpleNamespace(start=AsyncMock(side_effect=start_driver))
with (
patch.object(
playwright_worker_module,
"resolve_headless",
return_value=False,
),
patch.object(
playwright_worker_module,
"ensure_browser_display",
ensure_display,
),
patch.object(
playwright_worker_module,
"async_playwright",
return_value=manager,
),
):
driver, headless = await playwright_worker_module._start_playwright_for_browser()
self.assertEqual(driver, "playwright-driver")
self.assertFalse(headless)
manager.start.assert_awaited_once_with()
async def test_visible_login_prompt_overrides_stale_sessionid(self):
worker = DouyinWorker(account_id=917, login_mode="browser")
worker._has_visible_login_prompt = AsyncMock(return_value=True)
worker.check_homepage_login_status = AsyncMock(return_value=True)
worker.check_logged_in_by_cookie = AsyncMock(return_value=True)
logged_in = await worker._verify_login_state()
self.assertFalse(logged_in)
worker.check_homepage_login_status.assert_not_awaited()
worker.check_logged_in_by_cookie.assert_not_awaited()
async def test_homepage_message_entry_is_not_login_evidence(self):
worker = DouyinWorker(account_id=918, login_mode="browser")
worker._has_visible_login_prompt = AsyncMock(return_value=False)
worker.page = SimpleNamespace(query_selector=AsyncMock(return_value=None))
logged_in = await worker.check_homepage_login_status()
self.assertFalse(logged_in)
worker.page.query_selector.assert_awaited_once()
async def test_start_saves_task_and_stop_waits_until_it_is_done(self):
worker = DouyinWorker(account_id=919, login_mode="im_direct")
loop_started = asyncio.Event()
+145
View File
@@ -389,6 +389,10 @@ class GlobalSendQueueTests(unittest.IsolatedAsyncioTestCase):
class BackgroundTrafficLimitTests(unittest.IsolatedAsyncioTestCase):
async def _wait_until(self, predicate) -> None:
while not predicate():
await asyncio.sleep(0.001)
async def test_background_slot_respects_configured_concurrency_limit(self):
with patch.dict(
os.environ,
@@ -458,6 +462,147 @@ class BackgroundTrafficLimitTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(len(tasks_seen), 3)
self.assertTrue(all(task is tasks_seen[0] for task in tasks_seen))
async def test_startup_traffic_leaves_one_slot_for_recurring_work(self):
with patch.dict(
os.environ,
{"KEFU_BACKGROUND_NETWORK_CONCURRENCY": "3"},
):
controller = TrafficController()
self.addAsyncCleanup(controller.stop)
entered: list[str] = []
release = asyncio.Event()
async def request(name: str, *, startup: bool = False) -> None:
async with controller.background_slot(1, name, startup=startup):
entered.append(name)
await release.wait()
starts = [
asyncio.create_task(request(f"startup-{index}", startup=True))
for index in range(3)
]
while len(entered) < 2:
await asyncio.sleep(0)
await asyncio.sleep(0.01)
# A flood of startups may occupy at most capacity - 1 slots, so a
# recurring poll still gets in while the fleet is coming online.
self.assertEqual(len(entered), 2)
self.assertEqual(controller.background_startup_active, 2)
poll = asyncio.create_task(request("poll"))
await asyncio.wait_for(
self._wait_until(lambda: "poll" in entered),
timeout=0.5,
)
release.set()
await asyncio.wait_for(asyncio.gather(*starts, poll), timeout=0.5)
self.assertEqual(controller.background_active, 0)
self.assertEqual(controller.background_startup_active, 0)
async def test_recurring_work_is_not_deferred_indefinitely_by_startups(self):
"""Pending startup work may delay a recurring poll, never block it.
Starting several hundred accounts keeps startup requests queued for the
whole run. Yielding to that queue without a deadline left every hosted
account silent until the last account had finished coming online.
"""
with patch.dict(
os.environ,
{
"KEFU_BACKGROUND_NETWORK_CONCURRENCY": "1",
"KEFU_BACKGROUND_NORMAL_MAX_DEFER_SECONDS": "0.02",
},
):
controller = TrafficController()
self.addAsyncCleanup(controller.stop)
entered: list[str] = []
release = asyncio.Event()
async def poll() -> None:
async with controller.background_slot(1, "conversation poll"):
entered.append("poll")
await release.wait()
# Stands in for a batch whose startup requests never stop arriving.
controller._background_startup_clear.clear()
task = asyncio.create_task(poll())
await asyncio.wait_for(
self._wait_until(lambda: entered == ["poll"]),
timeout=1.0,
)
release.set()
await asyncio.wait_for(task, timeout=0.5)
async def test_startup_request_is_not_buried_behind_normal_backlog(self):
with patch.dict(
os.environ,
{"KEFU_BACKGROUND_NETWORK_CONCURRENCY": "1"},
):
controller = TrafficController()
self.addAsyncCleanup(controller.stop)
entered: list[str] = []
releases = {
name: asyncio.Event()
for name in ("active", "normal-1", "normal-2", "startup")
}
async def request(name: str, *, startup: bool = False) -> None:
async with controller.background_slot(
1,
name,
startup=startup,
):
entered.append(name)
await releases[name].wait()
active = asyncio.create_task(request("active"))
while entered != ["active"]:
await asyncio.sleep(0)
normal_one = asyncio.create_task(request("normal-1"))
normal_two = asyncio.create_task(request("normal-2"))
# Let one normal request reach the shared semaphore while the other is
# held at normal admission, then add the priority startup request.
await asyncio.sleep(0)
await asyncio.sleep(0)
startup = asyncio.create_task(request("startup", startup=True))
await asyncio.sleep(0)
releases["active"].set()
while len(entered) < 2:
await asyncio.sleep(0)
self.assertEqual(entered[:2], ["active", "normal-1"])
releases["normal-1"].set()
while len(entered) < 3:
await asyncio.sleep(0)
self.assertEqual(entered[:3], ["active", "normal-1", "startup"])
releases["startup"].set()
while len(entered) < 4:
await asyncio.sleep(0)
releases["normal-2"].set()
await asyncio.wait_for(
asyncio.gather(active, normal_one, normal_two, startup),
timeout=0.2,
)
self.assertEqual(
entered,
["active", "normal-1", "startup", "normal-2"],
)
self.assertEqual(controller.background_active, 0)
self.assertEqual(controller.background_waiting, 0)
self.assertEqual(controller.background_startup_active, 0)
self.assertEqual(controller.background_startup_waiting, 0)
class TrafficControllerLoopIsolationTests(unittest.TestCase):
def test_get_traffic_controller_does_not_reuse_asyncio_primitives(self):
+192
View File
@@ -0,0 +1,192 @@
from __future__ import annotations
import json
import os
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
BACKEND_DIR = Path(__file__).resolve().parents[1]
os.environ["KEFU_DB_TYPE"] = "sqlite"
os.environ["KEFU_DATABASE_URL"] = ""
os.environ["KEFU_DB_PATH"] = str(BACKEND_DIR / "kefu.db")
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.playwright_worker import DouyinWorker
class WorkerScaleControlTests(unittest.IsolatedAsyncioTestCase):
async def test_storage_load_selects_only_cookie_column(self):
class _ScalarResult:
def scalar_one_or_none(self):
return '{"cookies": [{"name": "sessionid", "value": "ok"}]}'
db = SimpleNamespace(
execute=AsyncMock(return_value=_ScalarResult()),
close=AsyncMock(),
)
worker = DouyinWorker(account_id=499)
worker.get_db = AsyncMock(return_value=db)
storage = await worker._load_storage_state()
self.assertEqual(storage["cookies"][0]["value"], "ok")
statement = db.execute.await_args.args[0]
selected_names = [
item.get("name") for item in statement.column_descriptions
]
self.assertEqual(selected_names, ["cookie_data"])
self.assertNotIn("im_session_data", str(statement).lower())
async def test_direct_service_marks_worker_ready_after_initialization(self):
worker = DouyinWorker(account_id=500, login_mode="im_direct")
worker._refresh_follow_welcome_config = AsyncMock(
return_value=(False, "", "sec-user")
)
worker.get_reply_delay = AsyncMock(return_value=None)
fake_service = SimpleNamespace(run=AsyncMock(), stop=AsyncMock())
async def complete_initialization():
service_factory.call_args.kwargs["on_ready"]()
fake_service.run.side_effect = complete_initialization
session = DouyinImSession(
cookies={"sessionid": "session"},
my_uid=0,
)
with patch(
"rpa_engine.playwright_worker.DouyinImService",
return_value=fake_service,
) as service_factory:
await worker._run_im_direct_service(session)
await worker.wait_until_ready()
worker._refresh_follow_welcome_config.assert_awaited_once_with(force=True)
fake_service.run.assert_awaited_once_with()
fake_service.stop.assert_awaited_once_with()
async def test_direct_start_failure_releases_readiness_waiter(self):
worker = DouyinWorker(account_id=501, login_mode="im_direct")
worker._load_storage_state = AsyncMock(return_value=None)
worker.update_account_status = AsyncMock()
worker.cleanup = AsyncMock()
await worker.start()
with self.assertRaisesRegex(RuntimeError, "未保存 Cookie"):
await worker.wait_until_ready()
if worker._task:
await worker._task
worker.update_account_status.assert_awaited_once_with(
"error",
error_msg="未保存 Cookie,无法直连 IM",
)
async def test_prevalidated_start_skips_duplicate_remote_validation(self):
worker = DouyinWorker(
account_id=502,
login_mode="im_direct",
credential_prevalidated=True,
)
session = DouyinImSession(
cookies={"sessionid": "session"},
my_uid=10001,
keys_str=json.dumps({"ec_privateKey": "private"}),
web_protect_str=json.dumps(
{
"ticket": "ticket",
"ts_sign": "sign",
"client_cert": "certificate",
}
),
)
worker._load_user_agent = AsyncMock(return_value="test-agent")
worker._build_im_session_from_storage = AsyncMock(return_value=session)
worker._best_effort_sec_user_id = AsyncMock(return_value="sec-user")
worker._persist_im_session = AsyncMock()
worker._run_im_direct_service = AsyncMock()
with patch(
"rpa_engine.playwright_worker.validate_im_session",
new_callable=AsyncMock,
) as validate:
started, reason = await worker._try_cookie_only_im_start(
{"cookies": []}
)
self.assertTrue(started)
self.assertEqual(reason, "")
validate.assert_not_awaited()
worker._best_effort_sec_user_id.assert_awaited_once_with(
refresh_if_missing=True,
refresh_if_stale=True,
)
worker._run_im_direct_service.assert_awaited_once_with(session)
async def test_disabled_follow_welcome_uses_cached_lightweight_config(self):
class _Result:
def first(self):
return False, "", "sec-user"
db = SimpleNamespace(
execute=AsyncMock(return_value=_Result()),
close=AsyncMock(),
)
worker = DouyinWorker(account_id=503)
worker.get_db = AsyncMock(return_value=db)
first = await worker._refresh_follow_welcome_config()
second = await worker._refresh_follow_welcome_config()
self.assertEqual(first, (False, "", "sec-user"))
self.assertEqual(second, first)
worker.get_db.assert_awaited_once_with()
db.execute.assert_awaited_once()
db.close.assert_awaited_once_with()
# The account PUT endpoint calls this synchronous hook so enabling the
# feature does not wait for the disabled-account ten-minute TTL.
worker.invalidate_follow_welcome_config()
await worker._refresh_follow_welcome_config()
self.assertEqual(worker.get_db.await_count, 2)
self.assertEqual(db.execute.await_count, 2)
async def test_disabled_follow_tick_does_not_read_follower_log(self):
worker = DouyinWorker(account_id=504)
worker._im_service = SimpleNamespace(session=object())
worker._follow_config_loaded = True
worker._refresh_follow_welcome_config = AsyncMock(
return_value=(False, "", "sec-user")
)
worker.get_db = AsyncMock()
worker._require_sec_user_id = AsyncMock()
await worker.follow_welcome_tick()
worker._refresh_follow_welcome_config.assert_awaited_once_with()
worker.get_db.assert_not_awaited()
worker._require_sec_user_id.assert_not_awaited()
async def test_disabled_follow_welcome_ignores_missing_cached_sec_user_id(self):
worker = DouyinWorker(account_id=505)
worker._im_service = SimpleNamespace(session=object())
worker._follow_config_loaded = True
worker._refresh_follow_welcome_config = AsyncMock(
return_value=(False, "", "")
)
worker._best_effort_sec_user_id = AsyncMock(return_value="")
await worker.follow_welcome_tick()
worker._best_effort_sec_user_id.assert_not_awaited()
if __name__ == "__main__":
unittest.main()
+573
View File
@@ -0,0 +1,573 @@
from __future__ import annotations
import asyncio
import os
import sys
import threading
import unittest
from pathlib import Path
from unittest.mock import AsyncMock, Mock, patch
from websockets.legacy.server import serve
BACKEND_DIR = Path(__file__).resolve().parents[1]
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from rpa_engine.douyin_im import ws_client as ws_module
from rpa_engine.douyin_im.session import DouyinImSession
from rpa_engine.douyin_im.ws_client import DouyinImWsClient, _reconnect_delay
TEST_WS_URL = (
"wss://frontier-im.douyin.com/ws/v2?fpid=9&device_id=10001&"
"token=test-token-value"
)
class _FakeWebSocket:
def __init__(self, frames=()):
self.frames = list(frames)
self.next_calls = 0
self.close_code = 1000
self.close_reason = "test complete"
self.close_calls: list[tuple[int, str]] = []
self.fail_calls = 0
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, traceback):
return False
def __aiter__(self):
return self
async def __anext__(self):
self.next_calls += 1
if not self.frames:
raise StopAsyncIteration
return self.frames.pop(0)
async def close(self, code=1000, reason=""):
self.close_calls.append((code, reason))
def fail_connection(self):
self.fail_calls += 1
class WebSocketScalingTests(unittest.IsolatedAsyncioTestCase):
def _make_client(self, handler=None, account_id=23):
session = DouyinImSession(
cookies={"sessionid": "session-value", "sid_tt": "sid-value"},
ws_urls=[TEST_WS_URL],
user_agent="test-agent/1.0",
)
return DouyinImWsClient(
session,
handler or AsyncMock(),
account_id=account_id,
)
async def test_async_connection_preserves_handshake_and_ping_options(self):
received: list[bytes] = []
async def handler(item):
received.append(item["payload"])
client = self._make_client(handler)
client._running = True
fake_ws = _FakeWebSocket([b"binary-frame", "text-frame"])
connect_mock = Mock(return_value=fake_ws)
with (
patch.object(ws_module, "websocket_connect", connect_mock),
patch.object(
ws_module,
"parse_ws_payload",
side_effect=lambda payload: [{"payload": payload}],
),
patch.object(ws_module.system_logger, "record"),
):
await client._run_connection(TEST_WS_URL)
queue = client._message_queue
self.assertIsNotNone(queue)
await asyncio.wait_for(queue.join(), timeout=0.5)
await client.stop()
self.assertEqual(received, [b"binary-frame", b"text-frame"])
kwargs = connect_mock.call_args.kwargs
self.assertEqual(kwargs["origin"], "https://www.douyin.com")
self.assertEqual(kwargs["subprotocols"], ["binary", "base64", "pbbp2"])
self.assertEqual(kwargs["user_agent_header"], "test-agent/1.0")
self.assertEqual(kwargs["ping_interval"], 20)
self.assertEqual(kwargs["ping_timeout"], ws_module._PING_TIMEOUT_SECONDS)
self.assertEqual(kwargs["max_queue"], ws_module._TRANSPORT_MAX_QUEUE)
self.assertEqual(kwargs["max_size"], ws_module._INCOMING_MAX_SIZE)
headers = dict(kwargs["extra_headers"])
self.assertEqual(headers["Cookie"], "sessionid=session-value; sid_tt=sid-value")
self.assertNotIn("Sec-WebSocket-Protocol", headers)
self.assertFalse(client.connected)
self.assertIsNone(client._connection)
async def test_browser_frontier_uses_text_heartbeat_and_filters_ack(self):
client = self._make_client()
client._running = True
websocket = AsyncMock()
heartbeat = asyncio.create_task(client._run_browser_heartbeat(websocket))
for _ in range(20):
if websocket.send.await_count:
break
await asyncio.sleep(0)
heartbeat.cancel()
with self.assertRaises(asyncio.CancelledError):
await heartbeat
websocket.send.assert_awaited_once_with("hi")
with patch.object(ws_module, "parse_ws_payload") as parse:
await client._dispatch("hi")
parse.assert_not_called()
async def test_starting_500_clients_does_not_create_os_threads(self):
parked = asyncio.Event()
async def parked_run_loop(_client, _url):
await parked.wait()
clients = [self._make_client(account_id=index + 1) for index in range(500)]
before_threads = threading.active_count()
with patch.object(DouyinImWsClient, "_run_loop", parked_run_loop):
await asyncio.gather(*(client.start() for client in clients))
await asyncio.sleep(0)
self.assertEqual(threading.active_count(), before_threads)
self.assertEqual(sum(client._task is not None for client in clients), 500)
self.assertEqual(
sum(client._dispatcher_task is not None for client in clients),
500,
)
await asyncio.gather(*(client.stop() for client in clients))
self.assertTrue(all(client._task is None for client in clients))
self.assertTrue(all(client._dispatcher_task is None for client in clients))
async def test_global_handler_concurrency_is_shared_across_clients(self):
limit = 3
release_handlers = asyncio.Event()
limit_reached = asyncio.Event()
active = 0
maximum_active = 0
started = 0
completed = 0
async def handler(_item):
nonlocal active, maximum_active, started, completed
active += 1
started += 1
maximum_active = max(maximum_active, active)
if started == limit:
limit_reached.set()
try:
await release_handlers.wait()
finally:
active -= 1
completed += 1
clients = [
self._make_client(handler, account_id=index + 1000)
for index in range(12)
]
queues = []
with patch.dict(
os.environ,
{ws_module._HANDLER_CONCURRENCY_ENV: str(limit)},
):
try:
for index, client in enumerate(clients):
client._running = True
client._ensure_dispatcher()
queue = client._message_queue
self.assertIsNotNone(queue)
queues.append(queue)
await queue.put({"index": index})
await asyncio.wait_for(limit_reached.wait(), timeout=0.5)
# Give every other account a chance to contend for the same
# process/event-loop-wide semaphore.
await asyncio.sleep(0)
self.assertEqual(started, limit)
self.assertEqual(maximum_active, limit)
release_handlers.set()
await asyncio.wait_for(
asyncio.gather(*(queue.join() for queue in queues)),
timeout=1.0,
)
finally:
release_handlers.set()
await asyncio.gather(*(client.stop() for client in clients))
self.assertEqual(completed, len(clients))
self.assertEqual(maximum_active, limit)
async def test_global_handler_limit_preserves_single_client_fifo(self):
received: list[int] = []
async def handler(item):
await asyncio.sleep(0)
received.append(item["sequence"])
client = self._make_client(handler, account_id=2001)
client._running = True
client._ensure_dispatcher()
queue = client._message_queue
self.assertIsNotNone(queue)
for sequence in range(20):
await queue.put({"sequence": sequence})
await asyncio.wait_for(queue.join(), timeout=0.5)
await client.stop()
self.assertEqual(received, list(range(20)))
async def test_stop_cancels_dispatcher_waiting_for_global_handler_slot(self):
holder_started = asyncio.Event()
release_holder = asyncio.Event()
waiter_handler = AsyncMock()
async def holder_handler(_item):
holder_started.set()
await release_holder.wait()
holder = self._make_client(holder_handler, account_id=3001)
waiter = self._make_client(waiter_handler, account_id=3002)
with patch.dict(
os.environ,
{ws_module._HANDLER_CONCURRENCY_ENV: "1"},
):
holder._running = True
holder._ensure_dispatcher()
holder_queue = holder._message_queue
self.assertIsNotNone(holder_queue)
await holder_queue.put({"id": "holder"})
await asyncio.wait_for(holder_started.wait(), timeout=0.5)
waiter._running = True
waiter._ensure_dispatcher()
waiter_queue = waiter._message_queue
self.assertIsNotNone(waiter_queue)
await waiter_queue.put({"id": "waiter"})
state = ws_module._get_loop_state()
async def wait_until_slot_has_waiter():
while not state.handler_slots._waiters:
await asyncio.sleep(0)
await asyncio.wait_for(wait_until_slot_has_waiter(), timeout=0.5)
joined = asyncio.create_task(waiter_queue.join())
await asyncio.wait_for(waiter.stop(), timeout=0.5)
await asyncio.wait_for(joined, timeout=0.5)
waiter_handler.assert_not_awaited()
self.assertTrue(waiter_queue.empty())
release_holder.set()
await asyncio.wait_for(holder_queue.join(), timeout=0.5)
await holder.stop()
async def test_bounded_dispatch_queue_keeps_receiver_responsive(self):
first_handler_started = asyncio.Event()
release_first_handler = asyncio.Event()
received: list[bytes] = []
async def handler(item):
received.append(item["payload"])
if len(received) == 1:
first_handler_started.set()
await release_first_handler.wait()
client = self._make_client(handler)
client._running = True
fake_ws = _FakeWebSocket([b"first", b"second", b"third", b"fourth"])
async def wait_until_third_frame_is_read():
while fake_ws.next_calls < 3:
await asyncio.sleep(0)
with (
patch.object(ws_module, "_APPLICATION_QUEUE_SIZE", 1),
patch.object(ws_module, "websocket_connect", return_value=fake_ws),
patch.object(
ws_module,
"parse_ws_payload",
side_effect=lambda payload: [{"payload": payload}],
),
patch.object(ws_module.system_logger, "record"),
):
task = asyncio.create_task(client._run_connection(TEST_WS_URL))
await asyncio.wait_for(first_handler_started.wait(), timeout=0.5)
await asyncio.wait_for(wait_until_third_frame_is_read(), timeout=0.5)
# The receiver keeps consuming while the business handler is
# blocked, but stops after the bounded application queue fills.
# It has pulled the third frame and is blocked enqueueing it; the
# fourth frame hasn't been requested and memory remains bounded.
self.assertEqual(fake_ws.next_calls, 3)
self.assertEqual(received, [b"first"])
self.assertEqual(client._message_queue.qsize(), 1)
self.assertIsNotNone(client._dispatcher_task)
release_first_handler.set()
await asyncio.wait_for(task, timeout=0.5)
queue = client._message_queue
self.assertIsNotNone(queue)
await asyncio.wait_for(queue.join(), timeout=0.5)
await client.stop()
self.assertEqual(received, [b"first", b"second", b"third", b"fourth"])
self.assertEqual(fake_ws.next_calls, 5) # four frames + end-of-stream
async def test_real_async_handshake_receives_binary_frame(self):
received: list[bytes] = []
request: dict[str, str | None] = {}
async def handler(item):
received.append(item["payload"])
async def server_handler(websocket, _path):
request["origin"] = websocket.request_headers.get("Origin")
request["cookie"] = websocket.request_headers.get("Cookie")
request["user_agent"] = websocket.request_headers.get("User-Agent")
request["subprotocol"] = websocket.subprotocol
await websocket.send(b"protobuf-frame")
await websocket.close(code=1000, reason="test complete")
client = self._make_client(handler)
client._running = True
with (
patch.object(
ws_module,
"parse_ws_payload",
side_effect=lambda payload: [{"payload": payload}],
),
patch.object(ws_module.system_logger, "record"),
):
async with serve(
server_handler,
"127.0.0.1",
0,
origins=["https://www.douyin.com"],
subprotocols=["pbbp2"],
) as server:
port = server.sockets[0].getsockname()[1]
await asyncio.wait_for(
client._run_connection(f"ws://127.0.0.1:{port}"),
timeout=1.0,
)
queue = client._message_queue
self.assertIsNotNone(queue)
await asyncio.wait_for(queue.join(), timeout=0.5)
await client.stop()
self.assertEqual(received, [b"protobuf-frame"])
self.assertEqual(request["origin"], "https://www.douyin.com")
self.assertEqual(request["cookie"], "sessionid=session-value; sid_tt=sid-value")
self.assertEqual(request["user_agent"], "test-agent/1.0")
self.assertEqual(request["subprotocol"], "pbbp2")
async def test_stop_cancels_slow_handler_and_drains_pending_messages(self):
handler_started = asyncio.Event()
handler_cancelled = asyncio.Event()
never_release = asyncio.Event()
async def handler(_item):
handler_started.set()
try:
await never_release.wait()
except asyncio.CancelledError:
handler_cancelled.set()
raise
client = self._make_client(handler)
client._running = True
client._ensure_dispatcher()
queue = client._message_queue
self.assertIsNotNone(queue)
await queue.put({"id": 1})
await queue.put({"id": 2})
await asyncio.wait_for(handler_started.wait(), timeout=0.5)
joined = asyncio.create_task(queue.join())
await asyncio.wait_for(client.stop(), timeout=0.5)
await asyncio.wait_for(joined, timeout=0.5)
self.assertTrue(handler_cancelled.is_set())
self.assertTrue(queue.empty())
self.assertIsNone(client._message_queue)
self.assertIsNone(client._dispatcher_task)
async def test_first_connection_uses_captured_url_without_refresh(self):
client = self._make_client(account_id=24)
client._running = True
refreshed_url = TEST_WS_URL + "&refreshed=1"
client._prepare_url = AsyncMock(return_value=refreshed_url)
connected_urls: list[str] = []
async def connection(url):
connected_urls.append(url)
client._last_connection_lifetime = 1.0
if len(connected_urls) == 2:
client._running = False
client._run_connection = connection
with (
patch.object(ws_module, "_reconnect_delay", return_value=0.0),
patch.object(ws_module.system_logger, "record"),
):
await asyncio.wait_for(client._run_loop(TEST_WS_URL), timeout=0.5)
self.assertEqual(connected_urls, [TEST_WS_URL, refreshed_url])
client._prepare_url.assert_awaited_once_with(TEST_WS_URL)
async def test_stop_closes_connection_and_cancels_receive_task(self):
client = self._make_client()
client._running = True
client.connected = True
fake_ws = _FakeWebSocket()
client._connection = fake_ws
task = asyncio.create_task(asyncio.sleep(30))
client._task = task
await client.stop()
self.assertEqual(fake_ws.close_calls, [(1000, "client stopping")])
self.assertTrue(task.cancelled())
self.assertFalse(client.connected)
self.assertIsNone(client._connection)
self.assertIsNone(client._task)
async def test_stop_aborts_connection_when_close_handshake_stalls(self):
client = self._make_client()
client._running = True
client.connected = True
fake_ws = _FakeWebSocket()
async def stalled_close(code=1000, reason=""):
fake_ws.close_calls.append((code, reason))
await asyncio.Event().wait()
fake_ws.close = stalled_close
client._connection = fake_ws
task = asyncio.create_task(asyncio.sleep(30))
client._task = task
with patch.object(ws_module, "_CLOSE_GRACE_SECONDS", 0.01):
await asyncio.wait_for(client.stop(), timeout=0.2)
self.assertEqual(fake_ws.fail_calls, 1)
self.assertTrue(task.cancelled())
self.assertIsNone(client._connection)
async def test_short_normal_closes_continue_exponential_retry(self):
client = self._make_client(account_id=41)
client._running = True
client._prepare_url = AsyncMock(return_value=TEST_WS_URL)
attempts = 0
async def short_connection(_url):
nonlocal attempts
attempts += 1
client._last_connection_lifetime = 1.0
if attempts == 4:
client._running = False
client._run_connection = short_connection
with (
patch.object(ws_module, "_reconnect_delay", return_value=0.0) as delay,
patch.object(ws_module.system_logger, "record"),
):
await asyncio.wait_for(client._run_loop(TEST_WS_URL), timeout=0.5)
self.assertEqual([call.args[1] for call in delay.call_args_list], [1, 2, 3])
async def test_stable_connection_resets_retry_counter(self):
client = self._make_client(account_id=42)
client._running = True
client._prepare_url = AsyncMock(return_value=TEST_WS_URL)
lifetimes = [1.0, 1.0, ws_module._STABLE_CONNECTION_SECONDS + 1.0, 1.0]
attempts = 0
async def connection(_url):
nonlocal attempts
client._last_connection_lifetime = lifetimes[attempts]
attempts += 1
if attempts == len(lifetimes):
client._running = False
client._run_connection = connection
with (
patch.object(ws_module, "_reconnect_delay", return_value=0.0) as delay,
patch.object(ws_module.system_logger, "record"),
):
await asyncio.wait_for(client._run_loop(TEST_WS_URL), timeout=0.5)
self.assertEqual([call.args[1] for call in delay.call_args_list], [1, 2, 1])
async def test_repeated_connection_failures_throttle_system_logs_per_account(self):
client = self._make_client(account_id=4041)
client._running = True
client._prepare_url = AsyncMock(return_value=TEST_WS_URL)
attempts = 0
async def failing_connection(_url):
nonlocal attempts
attempts += 1
if attempts == 6:
client._running = False
raise ConnectionError("frontier unavailable")
client._run_connection = failing_connection
with (
patch.object(ws_module, "_reconnect_delay", return_value=0.0),
patch.object(
ws_module,
"_system_log_throttle_seconds",
return_value=300.0,
),
patch.object(ws_module.system_logger, "record") as system_record,
patch.object(ws_module.logger, "warning") as ordinary_warning,
):
await asyncio.wait_for(client._run_loop(TEST_WS_URL), timeout=0.5)
# Ordinary diagnostics remain available for every live failure, while
# the database-backed system log receives one row for the repeated
# failure/reconnect cycle of this account.
self.assertEqual(attempts, 6)
self.assertEqual(ordinary_warning.call_count, 5)
self.assertEqual(system_record.call_count, 1)
self.assertEqual(system_record.call_args.kwargs["account_id"], 4041)
def test_handler_concurrency_environment_value_is_safely_clamped(self):
with patch.dict(os.environ, {ws_module._HANDLER_CONCURRENCY_ENV: "0"}):
self.assertEqual(ws_module._handler_concurrency_limit(), 1)
with patch.dict(os.environ, {ws_module._HANDLER_CONCURRENCY_ENV: "500"}):
self.assertEqual(ws_module._handler_concurrency_limit(), 32)
with patch.dict(os.environ, {ws_module._HANDLER_CONCURRENCY_ENV: "invalid"}):
self.assertEqual(ws_module._handler_concurrency_limit(), 8)
def test_reconnect_delay_is_bounded_and_spread_by_account(self):
waits = [_reconnect_delay(10, retry) for retry in range(1, 10)]
self.assertGreater(waits[1], waits[0])
self.assertGreater(waits[2], waits[1])
self.assertTrue(all(2.0 <= wait < 90.0 for wait in waits))
self.assertNotEqual(_reconnect_delay(10, 8), _reconnect_delay(11, 8))
if __name__ == "__main__":
unittest.main()
+23
View File
@@ -128,6 +128,10 @@ def _parse_tea_from_ls(origins: list) -> tuple[str, str]:
parsed = json.loads(entry.get("value") or "{}")
except Exception:
continue
# __tea_cache_first_6383 等条目的值是纯数字(如 1),json.loads 返回 int
# 不是 dict,必须跳过,否则 parsed.get() 抛 AttributeError 导致保存凭证 500。
if not isinstance(parsed, dict):
continue
uid = str(parsed.get("user_unique_id") or "").strip()
wid = str(parsed.get("web_id") or "").strip()
if uid and wid and uid == wid and len(uid) > 12:
@@ -233,6 +237,25 @@ def validate_cookie_json(cookie_data: str) -> dict:
return normalize_storage_state_for_im(data)
def extract_user_agent_from_cookie_data(cookie_data: Optional[str]) -> str:
"""从已保存的 Cookie JSON 提取顶层 user_agent(登录头)。
Playwright storage_state 导出时顶层带 user_agent采集浏览器当前 UA
DYCRED 凭证经 convert_dycred_to_storage_state 后也统一落到 user_agent
返回完整 UA 字符串缺失/无效时返回空串由调用方决定是否回填账号
"""
if not cookie_data:
return ""
try:
data = json.loads(cookie_data)
except Exception:
return ""
if not isinstance(data, dict):
return ""
ua = str(data.get("user_agent") or data.get("ua") or "").strip()
return ua
def write_cookie_file(account_id: int, cookie_data: str) -> str:
data = validate_cookie_json(cookie_data)
path = get_cookie_path(account_id)
+101
View File
@@ -0,0 +1,101 @@
"""Bound diagnostic payloads without changing live message processing."""
from __future__ import annotations
import json
import os
from typing import Any
TRUNCATION_MARKER = "\n...[日志内容过长,已截断]"
_MESSAGE_FIELDS = (
"type",
"url",
"uri",
"text",
"name",
"width",
"height",
"duration",
"sticker_id",
"mime_type",
)
def _env_limit(name: str, default: int, minimum: int, maximum: int) -> int:
try:
value = int(os.getenv(name, str(default)) or default)
except (TypeError, ValueError):
value = default
return max(minimum, min(maximum, value))
def truncate_text(value: Any, limit: int) -> str:
text = "" if value is None else str(value)
if len(text) <= limit:
return text
keep = max(0, limit - len(TRUNCATION_MARKER))
return text[:keep] + TRUNCATION_MARKER
def bound_system_log_detail(value: Any) -> str:
return truncate_text(
value,
_env_limit("KEFU_SYSTEM_LOG_MAX_CHARS", 4096, 512, 65536),
)
def _compact_media_message(value: Any) -> Any:
if not isinstance(value, dict):
return truncate_text(value, 2048)
compact: dict[str, Any] = {}
for key in _MESSAGE_FIELDS:
if key not in value:
continue
item = value[key]
compact[key] = truncate_text(item, 2048) if isinstance(item, str) else item
compact["_log_truncated"] = True
return compact
def bound_message_log_content(value: Any) -> str:
"""Keep a valid compact media JSON payload when a log entry is oversized."""
limit = _env_limit("KEFU_MESSAGE_LOG_MAX_CHARS", 16384, 2048, 262144)
text = "" if value is None else str(value)
if len(text) <= limit:
return text
try:
parsed = json.loads(text)
except (TypeError, ValueError, json.JSONDecodeError):
return truncate_text(text, limit)
if isinstance(parsed, dict) and parsed.get("type"):
compact = _compact_media_message(parsed)
elif isinstance(parsed, dict) and isinstance(parsed.get("messages"), list):
compact = {
"messages": [
_compact_media_message(item)
for item in parsed["messages"][:20]
],
"_log_truncated": True,
}
else:
return truncate_text(text, limit)
encoded = json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
return encoded if len(encoded) <= limit else truncate_text(encoded, limit)
def bound_raw_message_log_content(value: Any) -> str:
return truncate_text(
value,
_env_limit("KEFU_RAW_MESSAGE_LOG_MAX_CHARS", 32768, 4096, 262144),
)
def bound_error_log_content(value: Any) -> str:
return truncate_text(
value,
_env_limit("KEFU_ERROR_LOG_MAX_CHARS", 4096, 512, 65536),
)
+4 -2
View File
@@ -8,6 +8,7 @@ from typing import Optional
from models.database import AsyncSessionLocal
from models.models import ReceivedMessageLog
from utils.log_limits import bound_raw_message_log_content
logger = logging.getLogger("received_message_log")
@@ -23,8 +24,9 @@ async def record_received_message(
message_type: Optional[int] = None,
server_message_id: Optional[str] = None,
) -> None:
# 原样落库:不做 strip / parse / serialize,空字符串也记录
store_content = raw_content if raw_content is not None else ""
# The live message object remains untouched for matching/replying. Only
# this diagnostic copy is bounded before persistence.
store_content = bound_raw_message_log_content(raw_content)
async with AsyncSessionLocal() as db:
try:
+7 -3
View File
@@ -15,6 +15,8 @@ from collections import deque
from datetime import datetime
from typing import Optional
from .log_limits import bound_system_log_detail, truncate_text
logger = logging.getLogger("douyin_im.system")
_VALID_LEVELS = ("info", "success", "warning", "error")
@@ -43,14 +45,16 @@ def record(
"account_id": account_id,
"level": level,
"category": category,
"event": str(event or ""),
"detail": str(detail or ""),
"event": truncate_text(event, 255),
"detail": bound_system_log_detail(detail),
"created_at": datetime.utcnow().isoformat(),
}
_buffer.appendleft(entry)
_pending.append(entry)
msg = f"[{category}] {event}" + (f" | {detail}" if detail else "")
msg = f"[{category}] {entry['event']}" + (
f" | {entry['detail']}" if entry["detail"] else ""
)
if level == "error":
logger.error(msg)
elif level == "warning":
+30 -60
View File
@@ -1,5 +1,5 @@
<script setup>
import { ref, computed } from 'vue'
import { computed } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { message } from 'ant-design-vue'
import {
@@ -12,6 +12,7 @@ import {
MessageOutlined,
BugOutlined,
TeamOutlined,
SafetyCertificateOutlined,
LogoutOutlined,
ControlOutlined,
PayCircleOutlined,
@@ -22,6 +23,25 @@ import {
InboxOutlined
} from '@ant-design/icons-vue'
import { useAuthStore } from './stores/auth'
import { HEADER_TITLES } from './config/menus'
const ICON_MAP = {
DashboardOutlined,
UserOutlined,
SettingOutlined,
FileTextOutlined,
MessageOutlined,
BugOutlined,
TeamOutlined,
SafetyCertificateOutlined,
ControlOutlined,
PayCircleOutlined,
UnorderedListOutlined,
QuestionCircleOutlined,
RocketOutlined,
CloudDownloadOutlined,
InboxOutlined
}
const route = useRoute()
const router = useRouter()
@@ -29,6 +49,9 @@ const auth = useAuthStore()
const selectedKeys = computed(() => [route.path])
const isLoginPage = computed(() => route.path === '/login')
const headerTitle = computed(
() => HEADER_TITLES[route.name] || route.name || '工作台'
)
const navigate = ({ key }) => {
router.push(key)
@@ -57,61 +80,11 @@ const handleLogout = () => {
@click="navigate"
class="custom-menu"
>
<a-menu-item key="/">
<template #icon><DashboardOutlined /></template>
<span>数据概览</span>
</a-menu-item>
<a-menu-item key="/accounts">
<template #icon><UserOutlined /></template>
<span>账号管理</span>
</a-menu-item>
<a-menu-item key="/messages">
<template #icon><MessageOutlined /></template>
<span>私信收发</span>
</a-menu-item>
<a-menu-item key="/rules">
<template #icon><SettingOutlined /></template>
<span>自动回复规则</span>
</a-menu-item>
<a-menu-item key="/logs">
<template #icon><FileTextOutlined /></template>
<span>回复日志面板</span>
</a-menu-item>
<a-menu-item key="/received-messages">
<template #icon><InboxOutlined /></template>
<span>接收消息日志</span>
</a-menu-item>
<a-menu-item key="/system-logs">
<template #icon><BugOutlined /></template>
<span>系统诊断日志</span>
</a-menu-item>
<a-menu-item key="/download">
<template #icon><CloudDownloadOutlined /></template>
<span>软件下载</span>
</a-menu-item>
<a-menu-item key="/help">
<template #icon><QuestionCircleOutlined /></template>
<span>帮助中心</span>
</a-menu-item>
<a-menu-item v-if="auth.isAdmin" key="/users">
<template #icon><TeamOutlined /></template>
<span>用户与角色</span>
</a-menu-item>
<a-menu-item v-if="auth.isAdmin" key="/settings">
<template #icon><ControlOutlined /></template>
<span>系统设置</span>
</a-menu-item>
<a-menu-item v-if="auth.isAdmin" key="/desktop-update">
<template #icon><RocketOutlined /></template>
<span>桌面端升级</span>
</a-menu-item>
<a-menu-item v-if="auth.isAdmin" key="/payment-settings">
<template #icon><PayCircleOutlined /></template>
<span>支付配置</span>
</a-menu-item>
<a-menu-item v-if="!auth.isAdmin" key="/payment-orders">
<template #icon><UnorderedListOutlined /></template>
<span>我的订单</span>
<a-menu-item v-for="item in auth.visibleMenus" :key="item.path">
<template #icon>
<component :is="ICON_MAP[item.icon]" />
</template>
<span>{{ item.title }}</span>
</a-menu-item>
</a-menu>
</a-layout-sider>
@@ -119,9 +92,7 @@ const handleLogout = () => {
<a-layout>
<a-layout-header class="app-header">
<div class="header-left">
<h2 class="header-title">
{{ route.name === 'Dashboard' ? '数据中心' : route.name === 'Accounts' ? '账号中心' : route.name === 'Messages' ? '私信中心' : route.name === 'Rules' ? '策略中心' : route.name === 'ReceivedMessages' ? '接收消息日志' : route.name === 'SystemLogs' ? '诊断中心' : route.name === 'Users' ? '权限中心' : route.name === 'Settings' ? '系统设置' : route.name === 'DesktopUpdate' ? '桌面端升级' : route.name === 'PaymentSettings' ? '支付配置' : route.name === 'MyPaymentOrders' ? '我的订单' : route.name === 'Download' ? '软件下载' : route.name === 'Help' ? '帮助中心' : '日志中心' }}
</h2>
<h2 class="header-title">{{ headerTitle }}</h2>
</div>
<div class="header-right">
<a-space size="middle">
@@ -180,7 +151,6 @@ const handleLogout = () => {
-webkit-text-fill-color: transparent;
}
/* 侧边栏折叠时只保留图标,避免标题文字竖排变形 */
.app-sider.ant-layout-sider-collapsed .logo-container {
justify-content: center;
padding: 0;
+36 -2
View File
@@ -21,6 +21,14 @@ const props = defineProps({
showHeader: {
type: Boolean,
default: true
},
readonly: {
type: Boolean,
default: false
},
canUploadCards: {
type: Boolean,
default: true
}
})
@@ -39,10 +47,12 @@ const typeOptions = replyTypeOptions.map((opt) => ({
const cardUploading = ref({})
const addReplyItem = () => {
if (props.readonly) return
replyItems.value = [...replyItems.value, emptyReplyForm()]
}
const removeReplyItem = (index) => {
if (props.readonly) return
if (replyItems.value.length <= 1) {
message.warning('至少保留一条回复消息')
return
@@ -53,6 +63,7 @@ const removeReplyItem = (index) => {
}
const updateReplyField = (index, field, value) => {
if (props.readonly) return
const next = replyItems.value.map((item, i) =>
i === index ? { ...item, [field]: value } : item
)
@@ -60,6 +71,7 @@ const updateReplyField = (index, field, value) => {
}
const updateReplyFields = (index, fields) => {
if (props.readonly) return
const next = replyItems.value.map((item, i) =>
i === index ? { ...item, ...fields } : item
)
@@ -67,6 +79,11 @@ const updateReplyFields = (index, fields) => {
}
const uploadCardImage = async (index, options) => {
if (props.readonly || !props.canUploadCards) {
message.warning('当前账号无卡片上传权限')
options?.onError?.(new Error('no permission'))
return
}
const { file, onSuccess, onError } = options
cardUploading.value = { ...cardUploading.value, [index]: true }
try {
@@ -104,7 +121,13 @@ const copyPageUrl = async (url) => {
<div class="reply-rule-editor">
<div v-if="showHeader" class="reply-list-header">
<span class="reply-list-title">自动回复消息</span>
<a-button type="dashed" size="small" class="reply-add-btn" @click="addReplyItem">
<a-button
v-if="!readonly"
type="dashed"
size="small"
class="reply-add-btn"
@click="addReplyItem"
>
<template #icon><PlusOutlined /></template>
添加一条消息
</a-button>
@@ -114,7 +137,7 @@ const copyPageUrl = async (url) => {
<div class="reply-item-header">
<span>消息 {{ replyIndex + 1 }}</span>
<a-button
v-if="replyItems.length > 1"
v-if="!readonly && replyItems.length > 1"
type="text"
danger
size="small"
@@ -130,6 +153,7 @@ const copyPageUrl = async (url) => {
:value="reply.reply_type"
button-style="solid"
class="reply-type-group"
:disabled="readonly"
@update:value="(v) => updateReplyField(replyIndex, 'reply_type', v)"
>
<a-radio-button v-for="opt in typeOptions" :key="opt.value" :value="opt.value">
@@ -193,6 +217,7 @@ const copyPageUrl = async (url) => {
<a-form-item label="封面图片" required>
<div class="card-upload-row">
<a-upload
v-if="canUploadCards && !readonly"
name="file"
list-type="picture-card"
:show-upload-list="false"
@@ -214,6 +239,15 @@ const copyPageUrl = async (url) => {
<div>上传封面</div>
</div>
</a-upload>
<div v-else class="card-upload-placeholder">
<img
v-if="reply.reply_card_cover_url || reply.reply_card_image_path"
:src="reply.reply_card_cover_url || reply.reply_card_image_path"
alt="封面"
class="card-cover-preview"
/>
<template v-else>无上传权限</template>
</div>
<span class="card-upload-tip">上传后自动转为 PNG favicon32×32与卡片封面256×256</span>
</div>
</a-form-item>
+150
View File
@@ -0,0 +1,150 @@
/**
* Sidebar menu catalog. Visibility is driven by permission codes from /auth/me.
* Admin pages also require paired action permissions (alsoRequires).
*/
export const MENU_ITEMS = [
{
path: '/',
name: 'Dashboard',
title: '数据概览',
permission: 'menu.dashboard',
icon: 'DashboardOutlined'
},
{
path: '/accounts',
name: 'Accounts',
title: '我的账号',
adminTitle: '账号管理',
permission: 'menu.accounts',
icon: 'UserOutlined'
},
{
path: '/messages',
name: 'Messages',
title: '私信收发',
permission: 'menu.messages',
icon: 'MessageOutlined'
},
{
path: '/rules',
name: 'Rules',
title: '自动回复规则',
permission: 'menu.rules',
icon: 'SettingOutlined'
},
{
path: '/logs',
name: 'Logs',
title: '回复日志面板',
permission: 'menu.logs',
alsoRequires: ['logs.read'],
icon: 'FileTextOutlined'
},
{
path: '/received-messages',
name: 'ReceivedMessages',
title: '接收消息日志',
permission: 'menu.received_messages',
alsoRequires: ['received_messages.read'],
icon: 'InboxOutlined'
},
{
path: '/system-logs',
name: 'SystemLogs',
title: '系统诊断日志',
permission: 'menu.system_logs',
alsoRequires: ['system_logs.read'],
icon: 'BugOutlined'
},
{
path: '/download',
name: 'Download',
title: '软件下载',
permission: 'menu.download',
icon: 'CloudDownloadOutlined'
},
{
path: '/help',
name: 'Help',
title: '帮助中心',
permission: 'menu.help',
icon: 'QuestionCircleOutlined'
},
{
path: '/users',
name: 'Users',
title: '用户管理',
permission: 'menu.users',
alsoRequires: ['users.manage'],
icon: 'TeamOutlined'
},
{
path: '/roles',
name: 'Roles',
title: '角色设定',
permission: 'menu.roles',
alsoRequires: ['roles.manage'],
icon: 'SafetyCertificateOutlined'
},
{
path: '/settings',
name: 'Settings',
title: '系统设置',
permission: 'menu.settings',
alsoRequires: ['settings.manage'],
icon: 'ControlOutlined'
},
{
path: '/desktop-update',
name: 'DesktopUpdate',
title: '桌面端升级',
permission: 'menu.desktop_update',
alsoRequires: ['desktop.manage'],
icon: 'RocketOutlined'
},
{
path: '/payment-settings',
name: 'PaymentSettings',
title: '支付配置',
permission: 'menu.payment_settings',
alsoRequires: ['payments.manage'],
icon: 'PayCircleOutlined'
},
{
path: '/payment-orders',
name: 'MyPaymentOrders',
title: '我的订单',
permission: 'menu.payment_orders',
alsoRequires: ['orders.read'],
icon: 'UnorderedListOutlined'
}
]
export const HEADER_TITLES = {
Dashboard: '数据中心',
Accounts: '账号中心',
Messages: '私信中心',
Rules: '策略中心',
ReceivedMessages: '接收消息日志',
SystemLogs: '诊断中心',
Users: '用户管理',
Roles: '角色设定',
Settings: '系统设置',
DesktopUpdate: '桌面端升级',
PaymentSettings: '支付配置',
MyPaymentOrders: '我的订单',
Download: '软件下载',
Help: '帮助中心',
Logs: '日志中心'
}
export function menuAccessible(item, hasPermission) {
if (!item || !hasPermission(item.permission)) return false
const extra = item.alsoRequires || []
return extra.every((code) => hasPermission(code))
}
export function firstAccessiblePath(hasPermission) {
const hit = MENU_ITEMS.find((item) => menuAccessible(item, hasPermission))
return hit?.path || '/help'
}
+58 -19
View File
@@ -8,6 +8,7 @@ import SystemLogs from '../views/SystemLogs.vue'
import ReceivedMessages from '../views/ReceivedMessages.vue'
import Login from '../views/Login.vue'
import Users from '../views/Users.vue'
import Roles from '../views/Roles.vue'
import Settings from '../views/Settings.vue'
import DesktopUpdate from '../views/DesktopUpdate.vue'
import PaymentSettings from '../views/PaymentSettings.vue'
@@ -15,23 +16,51 @@ import MyPaymentOrders from '../views/MyPaymentOrders.vue'
import Help from '../views/Help.vue'
import Download from '../views/Download.vue'
import { useAuthStore } from '../stores/auth'
import { firstAccessiblePath, MENU_ITEMS, menuAccessible } from '../config/menus'
const routes = [
{ path: '/login', component: Login, name: 'Login', meta: { public: true } },
{ path: '/', component: Dashboard, name: 'Dashboard' },
{ path: '/accounts', component: Accounts, name: 'Accounts', meta: { write: true } },
{ path: '/messages', component: Messages, name: 'Messages', meta: { write: true } },
{ path: '/rules', component: Rules, name: 'Rules', meta: { write: true } },
{ path: '/logs', component: Logs, name: 'Logs' },
{ path: '/received-messages', component: ReceivedMessages, name: 'ReceivedMessages' },
{ path: '/system-logs', component: SystemLogs, name: 'SystemLogs' },
{ path: '/users', component: Users, name: 'Users', meta: { admin: true } },
{ path: '/settings', component: Settings, name: 'Settings', meta: { admin: true } },
{ path: '/desktop-update', component: DesktopUpdate, name: 'DesktopUpdate', meta: { admin: true } },
{ path: '/payment-settings', component: PaymentSettings, name: 'PaymentSettings', meta: { admin: true } },
{ path: '/payment-orders', component: MyPaymentOrders, name: 'MyPaymentOrders' },
{ path: '/help', component: Help, name: 'Help' },
{ path: '/download', component: Download, name: 'Download' }
{ path: '/', component: Dashboard, name: 'Dashboard', meta: { permission: 'menu.dashboard' } },
{ path: '/accounts', component: Accounts, name: 'Accounts', meta: { permission: 'menu.accounts' } },
{ path: '/messages', component: Messages, name: 'Messages', meta: { permission: 'menu.messages' } },
{ path: '/rules', component: Rules, name: 'Rules', meta: { permission: 'menu.rules' } },
{ path: '/logs', component: Logs, name: 'Logs', meta: { permission: 'menu.logs' } },
{
path: '/received-messages',
component: ReceivedMessages,
name: 'ReceivedMessages',
meta: { permission: 'menu.received_messages' }
},
{
path: '/system-logs',
component: SystemLogs,
name: 'SystemLogs',
meta: { permission: 'menu.system_logs' }
},
{ path: '/users', component: Users, name: 'Users', meta: { permission: 'menu.users' } },
{ path: '/roles', component: Roles, name: 'Roles', meta: { permission: 'menu.roles' } },
{ path: '/settings', component: Settings, name: 'Settings', meta: { permission: 'menu.settings' } },
{
path: '/desktop-update',
component: DesktopUpdate,
name: 'DesktopUpdate',
meta: { permission: 'menu.desktop_update' }
},
{
path: '/payment-settings',
component: PaymentSettings,
name: 'PaymentSettings',
meta: { permission: 'menu.payment_settings' }
},
{
path: '/payment-orders',
component: MyPaymentOrders,
name: 'MyPaymentOrders',
meta: { permission: 'menu.payment_orders' }
},
{ path: '/help', component: Help, name: 'Help', meta: { permission: 'menu.help' } },
{ path: '/download', component: Download, name: 'Download', meta: { permission: 'menu.download' } }
]
const router = createRouter({
@@ -44,7 +73,7 @@ router.beforeEach(async (to) => {
if (to.meta.public) {
if (auth.isLoggedIn && to.path === '/login') {
return '/'
return firstAccessiblePath((code) => auth.hasPermission(code))
}
return true
}
@@ -60,14 +89,24 @@ router.beforeEach(async (to) => {
auth.clearSession()
return '/login'
}
} else if (!Array.isArray(auth.user.permissions)) {
// Old localStorage sessions lack the permission list.
try {
await auth.fetchMe()
} catch {
auth.clearSession()
return '/login'
}
}
if (to.meta.admin && !auth.isAdmin) {
return '/'
const required = to.meta.permission
if (required && !auth.hasPermission(required)) {
return firstAccessiblePath((code) => auth.hasPermission(code))
}
if (to.meta.write && auth.isViewer) {
return '/'
const menuItem = MENU_ITEMS.find((item) => item.path === to.path)
if (menuItem && !menuAccessible(menuItem, (code) => auth.hasPermission(code))) {
return firstAccessiblePath((code) => auth.hasPermission(code))
}
return true
+96 -105
View File
@@ -1,197 +1,188 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import api from '../api'
import { MENU_ITEMS, menuAccessible } from '../config/menus'
export const useAuthStore = defineStore('auth', () => {
const token = ref(localStorage.getItem('kefu_token') || '')
const user = ref(JSON.parse(localStorage.getItem('kefu_user') || 'null'))
const isLoggedIn = computed(() => !!token.value)
const isAdmin = computed(() => user.value?.role === 'admin')
const canWrite = computed(() => ['admin', 'operator'].includes(user.value?.role))
const isViewer = computed(() => user.value?.role === 'viewer')
const roleLabel = computed(() => {
const map = { admin: '管理员', operator: '运营', viewer: '只读' }
return map[user.value?.role] || user.value?.role || ''
const permissions = computed(() => {
const list = user.value?.permissions
return Array.isArray(list) ? list : []
})
const permissionSet = computed(() => new Set(permissions.value))
const isAdmin = computed(() => {
if (typeof user.value?.is_admin === 'boolean') return user.value.is_admin
return user.value?.role === 'admin'
})
const hasPermission = (code) => {
if (!code) return true
if (isAdmin.value) return true
if (permissionSet.value.has(code)) return true
// Legacy accounts.write covers all granular account write buttons.
if (
code.startsWith('accounts.') &&
code !== 'accounts.write' &&
permissionSet.value.has('accounts.write')
) {
return true
}
return false
}
const canWrite = computed(() =>
hasPermission('accounts.write') ||
hasPermission('accounts.create') ||
hasPermission('accounts.update') ||
hasPermission('accounts.delete') ||
hasPermission('accounts.start') ||
hasPermission('accounts.stop') ||
hasPermission('accounts.cookie')
)
const canCreateAccounts = computed(() => hasPermission('accounts.create'))
const canUpdateAccounts = computed(() => hasPermission('accounts.update'))
const canDeleteAccounts = computed(() => hasPermission('accounts.delete'))
const canStartAccounts = computed(() => hasPermission('accounts.start'))
const canStopAccounts = computed(() => hasPermission('accounts.stop'))
const canManageCookies = computed(() => hasPermission('accounts.cookie'))
const canWriteMessages = computed(() => hasPermission('messages.write'))
const canWriteRules = computed(() => hasPermission('rules.write'))
const canWriteLinkCards = computed(() => hasPermission('link_cards.write'))
const canClearSystemLogs = computed(() => hasPermission('system_logs.clear'))
const canManageUsers = computed(() => hasPermission('users.manage'))
const canManageRoles = computed(() => hasPermission('roles.manage'))
const canManagePayments = computed(() => hasPermission('payments.manage'))
const canManageSettings = computed(() => hasPermission('settings.manage'))
const canManageDatabase = computed(() => hasPermission('settings.database'))
const canCreateOrders = computed(() => hasPermission('orders.create'))
const hasGlobalDataScope = computed(
() => isAdmin.value || hasPermission('data.scope_all')
)
const isViewer = computed(
() =>
!canWrite.value &&
!canWriteMessages.value &&
!canWriteRules.value &&
!isAdmin.value
)
const roleLabel = computed(
() => user.value?.role_label || user.value?.role || ''
)
const visibleMenus = computed(() =>
MENU_ITEMS.filter((item) => menuAccessible(item, hasPermission)).map((item) => ({
...item,
title: isAdmin.value && item.adminTitle ? item.adminTitle : item.title
}))
)
const setSession = (accessToken, userData) => {
token.value = accessToken
user.value = userData
localStorage.setItem('kefu_token', accessToken)
localStorage.setItem('kefu_user', JSON.stringify(userData))
}
const clearSession = () => {
token.value = ''
user.value = null
localStorage.removeItem('kefu_token')
localStorage.removeItem('kefu_user')
}
const login = async (username, password) => {
const res = await api.post('/auth/login', { username, password })
const accessToken = res.data.access_token
const me = await api.get('/auth/me', {
headers: { Authorization: `Bearer ${accessToken}` }
})
setSession(accessToken, me.data)
return me.data
}
const register = async (payload) => {
const res = await api.post('/auth/register', payload)
return res.data
}
const verifyEmail = async (verifyToken) => {
const res = await api.post('/auth/verify-email', { token: verifyToken })
return res.data
}
const resendVerification = async (payload) => {
const res = await api.post('/auth/resend-verification', payload)
return res.data
}
const forgotPassword = async (payload) => {
const res = await api.post('/auth/forgot-password', payload)
return res.data
}
const resetPassword = async (token, password) => {
const res = await api.post('/auth/reset-password', { token, password })
const resetPassword = async (resetToken, password) => {
const res = await api.post('/auth/reset-password', {
token: resetToken,
password
})
return res.data
}
const fetchMe = async () => {
if (!token.value) return null
const res = await api.get('/auth/me')
user.value = res.data
localStorage.setItem('kefu_user', JSON.stringify(res.data))
return res.data
}
const logout = () => {
clearSession()
}
return {
token,
user,
isLoggedIn,
permissions,
isAdmin,
canWrite,
canCreateAccounts,
canUpdateAccounts,
canDeleteAccounts,
canStartAccounts,
canStopAccounts,
canManageCookies,
canWriteMessages,
canWriteRules,
canWriteLinkCards,
canClearSystemLogs,
canManageUsers,
canManageRoles,
canManagePayments,
canManageSettings,
canManageDatabase,
canCreateOrders,
hasGlobalDataScope,
isViewer,
roleLabel,
visibleMenus,
hasPermission,
login,
register,
verifyEmail,
resendVerification,
forgotPassword,
resetPassword,
fetchMe,
logout,
clearSession
}
})
+138 -7
View File
@@ -19,12 +19,12 @@
--accent-green: hsl(150, 75%, 50%);
--accent-red: hsl(360, 75%, 60%);
--text-primary: hsl(0, 0%, 95%);
--text-secondary: hsl(230, 10%, 65%);
--text-muted: hsl(230, 10%, 45%);
--text-primary: hsl(0, 0%, 96%);
--text-secondary: hsl(230, 12%, 72%);
--text-muted: hsl(230, 10%, 58%);
--border-light: rgba(255, 255, 255, 0.06);
--border-glow: hsla(270, 85%, 65%, 0.2);
--border-light: rgba(255, 255, 255, 0.1);
--border-glow: hsla(270, 85%, 65%, 0.28);
--glass-shadow: 0 8px 32px 0 rgba(0, 0, 0, 0.37);
--transition-smooth: all 0.3s cubic-bezier(0.25, 0.8, 0.25, 1);
@@ -287,17 +287,148 @@ h1, h2, h3, h4, h5, h6 {
.ant-modal .ant-input-password .ant-input,
.ant-modal textarea.ant-input,
.ant-modal .ant-select-selector {
background: rgba(255, 255, 255, 0.05) !important;
border-color: var(--border-light) !important;
background: rgba(255, 255, 255, 0.08) !important;
border-color: rgba(255, 255, 255, 0.16) !important;
color: var(--text-primary) !important;
box-shadow: none !important;
}
.ant-modal .ant-input:hover,
.ant-modal .ant-input-affix-wrapper:hover,
.ant-modal .ant-select-selector:hover,
.ant-modal .ant-input-number:hover {
border-color: rgba(192, 132, 252, 0.45) !important;
}
.ant-modal .ant-input:focus,
.ant-modal .ant-input-focused,
.ant-modal .ant-input-affix-wrapper-focused,
.ant-modal .ant-select-focused .ant-select-selector,
.ant-modal .ant-input-number-focused {
border-color: rgba(192, 132, 252, 0.65) !important;
box-shadow: 0 0 0 2px rgba(170, 59, 255, 0.18) !important;
}
.ant-modal .ant-input::placeholder,
.ant-modal .ant-input-number-input::placeholder,
.ant-modal textarea.ant-input::placeholder {
color: hsl(230, 10%, 62%) !important;
opacity: 1 !important;
}
.ant-modal .ant-input-affix-wrapper input::placeholder {
color: hsl(230, 10%, 62%) !important;
opacity: 1 !important;
}
.ant-modal .ant-checkbox-wrapper {
color: var(--text-primary) !important;
}
.ant-modal .ant-checkbox-inner {
background: rgba(255, 255, 255, 0.06) !important;
border-color: rgba(255, 255, 255, 0.35) !important;
}
.ant-modal .ant-checkbox-checked .ant-checkbox-inner {
background: #7c3aed !important;
border-color: #a78bfa !important;
}
.ant-modal .ant-select-arrow,
.ant-modal .ant-select-selection-placeholder {
color: var(--text-muted) !important;
}
.ant-modal .ant-select-selection-item {
color: var(--text-primary) !important;
}
.ant-modal .ant-alert-info {
background: rgba(59, 130, 246, 0.12) !important;
border: 1px solid rgba(96, 165, 250, 0.35) !important;
}
.ant-modal .ant-alert-message {
color: #dbeafe !important;
}
/* Tabs / 搜索框 / 分页 — 暗色可读性 */
.ant-tabs-top > .ant-tabs-nav::before {
border-bottom-color: rgba(255, 255, 255, 0.08) !important;
}
.ant-tabs .ant-tabs-tab {
color: var(--text-secondary) !important;
}
.ant-tabs .ant-tabs-tab:hover {
color: #e9d5ff !important;
}
.ant-tabs .ant-tabs-tab-active .ant-tabs-tab-btn {
color: #f3e8ff !important;
text-shadow: none;
}
.ant-tabs .ant-tabs-ink-bar {
background: linear-gradient(90deg, #aa3bff, #c084fc) !important;
}
.ant-input,
.ant-input-affix-wrapper,
.ant-select:not(.ant-select-customize-input) .ant-select-selector {
background: rgba(255, 255, 255, 0.06) !important;
border-color: rgba(255, 255, 255, 0.14) !important;
color: var(--text-primary) !important;
}
.ant-input::placeholder,
.ant-input-affix-wrapper input::placeholder {
color: var(--text-muted) !important;
opacity: 1 !important;
}
.ant-pagination {
color: var(--text-secondary) !important;
}
.ant-pagination .ant-pagination-item {
background: rgba(255, 255, 255, 0.04) !important;
border-color: rgba(255, 255, 255, 0.12) !important;
}
.ant-pagination .ant-pagination-item a {
color: var(--text-secondary) !important;
}
.ant-pagination .ant-pagination-item-active {
background: rgba(147, 51, 234, 0.25) !important;
border-color: rgba(192, 132, 252, 0.55) !important;
}
.ant-pagination .ant-pagination-item-active a {
color: #f3e8ff !important;
}
.ant-pagination .ant-pagination-prev .ant-pagination-item-link,
.ant-pagination .ant-pagination-next .ant-pagination-item-link,
.ant-pagination .ant-select-selector {
background: rgba(255, 255, 255, 0.04) !important;
border-color: rgba(255, 255, 255, 0.12) !important;
color: var(--text-secondary) !important;
}
.ant-pagination-options-quick-jumper input {
background: rgba(255, 255, 255, 0.06) !important;
border-color: rgba(255, 255, 255, 0.14) !important;
color: var(--text-primary) !important;
}
.ant-table-pagination.ant-pagination {
margin: 16px 20px !important;
}
.ant-modal .ant-radio-button-wrapper {
color: var(--text-secondary) !important;
background: rgba(255, 255, 255, 0.03) !important;
+97
View File
@@ -0,0 +1,97 @@
/**
* 日志聚合出来的会话对应回账号真实会话列表里的那一条
*
* 这条链路决定手动发送打给谁所以匹配必须是分级的宁缺毋滥的
* 抖音昵称大量重复还有用户1234567未知用户这类占位名
* 早先版本把 UID / 会话 ID / 昵称写在同一个 OR 谓词里交给 Array.find
* 于是列表里靠前的一条只要昵称相同就胜出哪怕后面有 UID 精确匹配的那条
* 结果就是消息发给了同名的另一个人
*/
export const isGenericPeerName = (name, peerUid = '') => {
const value = String(name || '').trim()
if (!value) return true
if (value === '未知用户') return true
if (peerUid && value === String(peerUid)) return true
if (/^\d+$/.test(value)) return true
if (/^用户\d+$/.test(value)) return true
return false
}
export const isConvId = (id) => /^0:1:\d+:\d+$/.test(String(id || '').trim())
export const extractPeerUid = (conv) => {
const raw = String(conv?.sender_id || '').trim()
if (!raw) return ''
if (/^\d+$/.test(raw)) return raw
if (isConvId(raw)) return raw.split(':')[3] || ''
const last = raw.split(':').pop()
return /^\d+$/.test(last || '') ? last : ''
}
/** 会话条目代表的对方 UID:sender_id 优先,其次会话 ID 末段。 */
export const conversationPeerUid = (item) =>
extractPeerUid(item) || extractPeerUid({ sender_id: item?.conversation_id })
/**
* 分级匹配命中一级就返回绝不降级
* 1) 对方 UID 精确相等 UID 时只认 UID
* 2) 会话 ID 完全相等
* 3) 会话 ID 末段等于 sender_id且全列表唯一
* 4) 昵称相等且昵称不是占位名且全列表唯一
* 任何一级出现多个候选都返回 null宁可不匹配也不能猜错人
*/
export const matchPeerConversation = (list, conv) => {
const items = Array.isArray(list) ? list : []
if (!items.length || !conv) return null
const peerUid = extractPeerUid(conv)
if (peerUid) {
return items.find((item) => conversationPeerUid(item) === peerUid) || null
}
const rawId = String(conv.sender_id || '').trim()
if (rawId) {
const exact = items.find(
(item) => String(item.conversation_id || '').trim() === rawId
)
if (exact) return exact
const suffix = items.filter((item) =>
String(item.conversation_id || '').endsWith(`:${rawId}`)
)
if (suffix.length === 1) return suffix[0]
if (suffix.length > 1) return null
}
const name = String(conv.sender_name || '').trim()
if (!name || isGenericPeerName(name)) return null
const byName = items.filter(
(item) => String(item.sender_name || '').trim() === name
)
return byName.length === 1 ? byName[0] : null
}
/**
* 解析手动发送要用的 conversation_id无法确认对方身份时返回空串
* 由调用方提示用户而不是拿一个差不多的会话把消息发出去
*/
export const resolveConversationId = (list, conv) => {
if (!conv) return ''
const peerUid = extractPeerUid(conv)
const match = matchPeerConversation(list, conv)
if (match?.conversation_id) {
const matchedPeer = conversationPeerUid(match)
// 最后一道断言:匹配到的会话必须和当前会话指向同一个人。
if (!peerUid || !matchedPeer || matchedPeer === peerUid) {
return String(match.conversation_id)
}
return ''
}
const raw = String(conv.sender_id || '').trim()
if (isConvId(raw)) return raw
// 裸 UID 是安全的:后端会用「当前账号 UID + 该 UID」拼出本账号的会话。
if (/^\d+$/.test(raw)) return raw
return ''
}
+415 -64
View File
@@ -1,6 +1,6 @@
<script setup>
import { ref, onMounted, onUnmounted, computed, watch } from 'vue'
import { useRouter } from 'vue-router'
import { useRoute, useRouter } from 'vue-router'
import api from '../api'
import { message } from 'ant-design-vue'
import { useAuthStore } from '../stores/auth'
@@ -39,6 +39,7 @@ import {
import { useIsMobile } from '../composables/useIsMobile'
const auth = useAuthStore()
const route = useRoute()
const router = useRouter()
const isMobile = useIsMobile()
const profileModalWidth = computed(() => (isMobile.value ? 'calc(100vw - 32px)' : 860))
@@ -67,14 +68,25 @@ const accountPageSize = ref(9)
const accountTotal = ref(0)
const rules = ref([])
const deviceProfiles = ref([])
const egressChannels = ref([])
const egressChannelsLoading = ref(false)
const egressChannelsError = ref('')
const CUSTOM_UA_PROFILE = '__custom__'
const loading = ref(false)
const batchStarting = ref(false)
const batchDeleting = ref(false)
const activeStartBatchId = ref(null)
let batchStatusTimer = null
let batchStatusRequestActive = false
let batchStatusGeneration = 0
const BATCH_STATUS_POLL_MS = 1500
const BATCH_STATUS_MAX_POLL_MS = 5000
const BATCH_STATUS_BACKOFF_STEP_MS = 750
let batchStatusPollDelayMs = BATCH_STATUS_POLL_MS
let batchStatusLastFinished = null
let batchStatusLastTotal = null
let batchStatusLastProcessing = null
let batchStatusLastQueued = null
const selectedIds = ref([])
const addVisible = ref(false)
const addSaving = ref(false)
@@ -354,6 +366,8 @@ const editForm = ref({
follow_welcome_content: '',
user_agent_profile: 'chrome_win120',
user_agent_custom: '',
egress_public_ip: '',
egress_auto_attempts: 1,
})
const profileSelectOptions = computed(() => {
@@ -365,6 +379,25 @@ const profileSelectOptions = computed(() => {
return opts
})
const egressChannelOptions = computed(() => {
const options = [
{ value: '', label: '自动选择(服务器默认公网出口)' }
]
for (const channel of egressChannels.value || []) {
const source = channel.source_ip ? `本地 ${channel.source_ip}` : '默认路由'
const suffix = channel.is_default ? ' · 当前默认' : ''
options.push({
value: channel.public_ip,
label: `${channel.public_ip}${source}${suffix}`
})
}
const selected = (editForm.value.egress_public_ip || '').trim()
if (selected && !options.some((item) => item.value === selected)) {
options.push({ value: selected, label: `${selected}(当前未检测到)`, disabled: true })
}
return options
})
const accountQuota = computed(() => {
const user = auth.user
const count = accountTotal.value
@@ -387,7 +420,12 @@ const accountQuotaLabel = computed(() => {
})
const canPurchaseSlots = computed(() => {
return paymentConfig.value?.payment_enabled && accountQuota.value.limited && !canAddAccount.value
return (
auth.canCreateOrders &&
paymentConfig.value?.payment_enabled &&
accountQuota.value.limited &&
!canAddAccount.value
)
})
const startableAccounts = computed(() =>
@@ -433,6 +471,18 @@ const selectedStartableCount = computed(() =>
startableAccounts.value.filter((a) => selectedIds.value.includes(a.id)).length
)
const selectableAccounts = computed(() => {
if (auth.canDeleteAccounts) return accounts.value
if (auth.canStartAccounts) return startableAccounts.value
return []
})
const selectedAccounts = computed(() =>
selectableAccounts.value.filter((a) => selectedIds.value.includes(a.id))
)
const selectedAccountCount = computed(() => selectedAccounts.value.length)
const isAccountSelected = (id) => selectedIds.value.includes(id)
const toggleAccountSelect = (id) => {
@@ -443,8 +493,8 @@ const toggleAccountSelect = (id) => {
}
}
const selectAllStartable = () => {
selectedIds.value = startableAccounts.value.map((a) => a.id)
const selectAllAccounts = () => {
selectedIds.value = selectableAccounts.value.map((a) => a.id)
}
const clearSelection = () => {
@@ -452,11 +502,11 @@ const clearSelection = () => {
}
const onSelectAllChange = (e) => {
if (e.target.checked) selectAllStartable()
if (e.target.checked) selectAllAccounts()
else clearSelection()
}
const showMyOrdersEntry = computed(() => !auth.isAdmin)
const showMyOrdersEntry = computed(() => auth.hasPermission('menu.payment_orders'))
const goMyOrders = () => {
router.push('/payment-orders')
@@ -534,6 +584,26 @@ const fetchDeviceProfiles = async () => {
}
}
const fetchEgressChannels = async (refresh = false) => {
if (!auth.canUpdateAccounts || egressChannelsLoading.value) return
egressChannelsLoading.value = true
egressChannelsError.value = ''
try {
const res = await api.get('/network/egress-channels', {
params: { refresh },
timeout: 20000
})
egressChannels.value = res.data?.channels || []
if (!egressChannels.value.length) {
egressChannelsError.value = '未探测到可用公网出口,将继续使用服务器默认路由'
}
} catch (error) {
egressChannelsError.value = error.response?.data?.detail || '公网通道检测失败'
} finally {
egressChannelsLoading.value = false
}
}
const fetchAccounts = async () => {
try {
loading.value = true
@@ -547,6 +617,8 @@ const fetchAccounts = async () => {
})
accounts.value = res.data.items || []
accountTotal.value = res.data.total || 0
const visibleIds = new Set(selectableAccounts.value.map((a) => a.id))
selectedIds.value = selectedIds.value.filter((id) => visibleIds.has(id))
// /退
const maxPage = Math.max(1, Math.ceil(accountTotal.value / accountPageSize.value) || 1)
if (accountPage.value > maxPage) {
@@ -710,9 +782,18 @@ const applyQueueSnapshot = (data, accountId) => {
const fetchReplyQueueSummaries = async ({ silent = true } = {}) => {
if (queueSummaryLoading.value) return
const accountIds = accounts.value
.map((account) => Number(account?.id))
.filter((accountId) => Number.isInteger(accountId) && accountId > 0)
if (!accountIds.length) {
replyQueueSummaries.value = {}
return
}
queueSummaryLoading.value = true
try {
const res = await api.get('/reply-queues')
const res = await api.get('/reply-queues', {
params: { account_ids: accountIds.join(',') }
})
const rows = Array.isArray(res.data?.items) ? res.data.items : []
const next = {}
for (const row of rows) {
@@ -840,7 +921,7 @@ const queueActionLabel = (item) => {
}
const isQueueSendDisabled = (item) => {
if (auth.user?.role === 'viewer') return true
if (!auth.canWriteMessages) return true
if (!queueSnapshot.value.running) return true
if (queueSendingJobId.value) return true
return item?.status !== 'waiting' || !!item?.expedited
@@ -983,6 +1064,10 @@ const goAccountRulesPage = (accountId) => {
}
const openAddModal = () => {
if (!auth.canCreateAccounts) {
message.warning('当前账号无添加账号权限')
return
}
if (!canAddAccount.value) {
if (canPurchaseSlots.value) {
purchaseVisible.value = true
@@ -1037,10 +1122,74 @@ const handleAddAccount = async () => {
const handleDeleteAccount = async (id) => {
try {
await api.delete(`/accounts/${id}`)
selectedIds.value = selectedIds.value.filter((item) => item !== id)
message.success('删除成功')
fetchAccounts()
await Promise.all([fetchAccounts(), auth.fetchMe()])
} catch (error) {
message.error('删除账号失败')
message.error(error.response?.data?.detail || '删除账号失败')
}
}
const handleBatchDeleteAccounts = async () => {
if (batchDeleting.value || batchStarting.value) return
const targets = selectedAccounts.value.map((account) => account.id)
if (!targets.length) {
message.warning('请先勾选要删除的账号')
return
}
batchDeleting.value = true
const deletedIds = []
const failures = []
message.loading({
content: `正在删除 0/${targets.length} 个账号...`,
key: 'batch_delete',
duration: 0
})
try {
// SQLite
for (let index = 0; index < targets.length; index += 1) {
const id = targets[index]
try {
await api.delete(`/accounts/${id}`)
deletedIds.push(id)
} catch (error) {
failures.push({
id,
reason: error.response?.data?.detail || error.message || '删除失败'
})
}
message.loading({
content: `正在删除 ${index + 1}/${targets.length} 个账号...`,
key: 'batch_delete',
duration: 0
})
}
const deletedSet = new Set(deletedIds)
selectedIds.value = selectedIds.value.filter((id) => !deletedSet.has(id))
await Promise.all([fetchAccounts(), auth.fetchMe()])
if (!failures.length) {
message.success({
content: `已删除 ${deletedIds.length} 个账号`,
key: 'batch_delete'
})
} else if (deletedIds.length) {
message.warning({
content: `已删除 ${deletedIds.length} 个账号,${failures.length} 个失败,可重新勾选后重试`,
key: 'batch_delete',
duration: 6
})
} else {
message.error({
content: failures[0]?.reason || '批量删除失败',
key: 'batch_delete',
duration: 6
})
}
} finally {
batchDeleting.value = false
}
}
@@ -1147,6 +1296,11 @@ const stopBatchStatusPolling = () => {
batchStatusTimer = null
}
batchStatusRequestActive = false
batchStatusPollDelayMs = BATCH_STATUS_POLL_MS
batchStatusLastFinished = null
batchStatusLastTotal = null
batchStatusLastProcessing = null
batchStatusLastQueued = null
}
const finishBatchStart = async (snapshot) => {
@@ -1176,6 +1330,7 @@ const finishBatchStart = async (snapshot) => {
}
await fetchAccounts()
batchStarting.value = false
startReplyQueueSummaryPolling()
}
const pollBatchStartStatus = (batchId, initialSnapshot = null) => {
@@ -1191,11 +1346,31 @@ const pollBatchStartStatus = (batchId, initialSnapshot = null) => {
Math.max(0, Number(snapshot?.failed_count) || 0) +
Math.max(0, Number(snapshot?.skipped_count) || 0) +
Math.max(0, Number(snapshot?.cancelled_count) || 0)
message.loading({
content: `账号启动队列处理中:${Math.min(finished, total)}/${total}`,
key: 'batch_start',
duration: 0
})
const processing = Math.max(0, Number(snapshot?.processing_count) || 0)
const queued = Math.max(0, Number(snapshot?.queued_count) || 0)
const visibleFinished = Math.min(finished, total)
const progressChanged =
visibleFinished !== batchStatusLastFinished ||
total !== batchStatusLastTotal ||
processing !== batchStatusLastProcessing ||
queued !== batchStatusLastQueued
if (progressChanged) {
batchStatusLastFinished = visibleFinished
batchStatusLastTotal = total
batchStatusLastProcessing = processing
batchStatusLastQueued = queued
batchStatusPollDelayMs = BATCH_STATUS_POLL_MS
message.loading({
content: `账号启动队列处理中:${visibleFinished}/${total}(正在处理 ${processing},等待 ${queued},系统正错峰启动)`,
key: 'batch_start',
duration: 0
})
} else {
batchStatusPollDelayMs = Math.min(
BATCH_STATUS_MAX_POLL_MS,
batchStatusPollDelayMs + BATCH_STATUS_BACKOFF_STEP_MS
)
}
if (snapshot?.complete) {
await finishBatchStart(snapshot)
return true
@@ -1225,12 +1400,13 @@ const pollBatchStartStatus = (batchId, initialSnapshot = null) => {
duration: 5
})
await fetchAccounts()
startReplyQueueSummaryPolling()
return
} finally {
if (generation === batchStatusGeneration) batchStatusRequestActive = false
}
if (generation === batchStatusGeneration && activeStartBatchId.value === batchId) {
batchStatusTimer = setTimeout(poll, BATCH_STATUS_POLL_MS)
batchStatusTimer = setTimeout(poll, batchStatusPollDelayMs)
}
}
@@ -1240,15 +1416,16 @@ const pollBatchStartStatus = (batchId, initialSnapshot = null) => {
generation === batchStatusGeneration &&
activeStartBatchId.value === batchId
) {
batchStatusTimer = setTimeout(poll, BATCH_STATUS_POLL_MS)
batchStatusTimer = setTimeout(poll, batchStatusPollDelayMs)
}
})
}
//
const runBatchStart = async ({ accountIds = [], allAccounts = false }) => {
if (batchStarting.value) return
if (batchStarting.value || batchDeleting.value) return
batchStarting.value = true
stopReplyQueueSummaryPolling()
stopBatchStatusPolling()
const submitGeneration = batchStatusGeneration
message.loading({ content: '正在提交账号启动队列...', key: 'batch_start', duration: 0 })
@@ -1270,6 +1447,7 @@ const runBatchStart = async ({ accountIds = [], allAccounts = false }) => {
if (submitGeneration !== batchStatusGeneration) return
batchStarting.value = false
activeStartBatchId.value = null
startReplyQueueSummaryPolling()
message.error({
content: error.response?.data?.detail || error.message || '提交批量启动失败',
key: 'batch_start',
@@ -1281,7 +1459,7 @@ const runBatchStart = async ({ accountIds = [], allAccounts = false }) => {
const accountLabel = (acc) => acc?.username || `账号 #${acc?.id}`
const batchStartRpa = async () => {
if (batchStarting.value || startingAll.value) return
if (batchStarting.value || batchDeleting.value || startingAll.value) return
const targets = startableAccounts.value
.filter((a) => selectedIds.value.includes(a.id))
.map((a) => ({ id: a.id, label: accountLabel(a) }))
@@ -1296,7 +1474,7 @@ const batchStartRpa = async () => {
const startingAll = ref(false)
const startAllRpa = async () => {
if (batchStarting.value || startingAll.value) return
if (batchStarting.value || batchDeleting.value || startingAll.value) return
startingAll.value = true
try {
await runBatchStart({ allAccounts: true })
@@ -1496,7 +1674,7 @@ const formatCookieTime = (value) => {
return date.toLocaleString('zh-CN')
}
const applyCookieResponse = (data) => {
const applyCookieResponse = (data, { preserveRuntimeCredential = false } = {}) => {
editForm.value.cookie_updated_at = data.cookie_updated_at
editForm.value.cookie_count = data.cookie_count
editForm.value.cookie_valid = data.cookie_valid
@@ -1506,10 +1684,12 @@ const applyCookieResponse = (data) => {
editForm.value.has_sessionid = !!data.has_sessionid
editForm.value.sessionid = data.sessionid || ''
editForm.value.sessionid_ss = data.sessionid_ss || ''
editForm.value.im_ready = !!data.im_ready
editForm.value.im_status = data.im_status || ''
editForm.value.can_skip_browser = !!data.can_skip_browser
editForm.value.should_reset = !!data.should_reset
if (!preserveRuntimeCredential) {
editForm.value.im_ready = !!data.im_ready
editForm.value.im_status = data.im_status || ''
editForm.value.can_skip_browser = !!data.can_skip_browser
editForm.value.should_reset = !!data.should_reset
}
}
const openEditModal = async (acc) => {
@@ -1544,14 +1724,21 @@ const openEditModal = async (acc) => {
follow_welcome_content: acc.follow_welcome_content || '',
user_agent_profile: 'chrome_win120',
user_agent_custom: '',
egress_public_ip: acc.egress_public_ip || '',
egress_auto_attempts: Math.max(1, Number(acc.egress_auto_attempts) || 1),
}
initUserAgentFields(acc)
if (auth.canUpdateAccounts) {
fetchEgressChannels(false)
}
try {
const res = await api.get(`/accounts/${acc.id}/cookie?purpose=management`)
editForm.value.cookie_data = res.data.cookie_data
? JSON.stringify(JSON.parse(res.data.cookie_data), null, 2)
: ''
applyCookieResponse(res.data)
if (auth.canManageCookies) {
const res = await api.get(`/accounts/${acc.id}/cookie?purpose=management`)
editForm.value.cookie_data = res.data.cookie_data
? JSON.stringify(JSON.parse(res.data.cookie_data), null, 2)
: ''
applyCookieResponse(res.data)
}
} catch (error) {
message.error('加载 Cookie 失败')
} finally {
@@ -1578,7 +1765,7 @@ const refreshCredential = async () => {
}
const cookieRes = await api.get(`/accounts/${editForm.value.id}/cookie?purpose=management`)
applyCookieResponse(cookieRes.data)
applyCookieResponse(cookieRes.data, { preserveRuntimeCredential: true })
message.success('凭证检测完成')
} catch (error) {
message.error(error.response?.data?.detail || '凭证检测失败')
@@ -1631,8 +1818,10 @@ const saveAccountInfo = async () => {
follow_welcome_enabled: !!editForm.value.follow_welcome_enabled,
follow_welcome_content: (editForm.value.follow_welcome_content || '').trim() || null,
user_agent: resolveUserAgentToSave() || null,
egress_public_ip: (editForm.value.egress_public_ip || '').trim() || null,
egress_auto_attempts: Math.max(1, Math.min(8, Number(editForm.value.egress_auto_attempts) || 1)),
})
message.success('账号信息已保存(设备头将在下次启动托管时生效)')
message.success('账号信息已保存(公网发送通道立即生效,设备头下次启动生效)')
fetchAccounts()
} catch (error) {
message.error('保存账号信息失败')
@@ -1705,13 +1894,21 @@ const clearCookie = async () => {
}
}
onMounted(() => {
auth.fetchMe()
fetchPaymentConfig()
fetchDeviceProfiles()
fetchAccounts()
fetchRules()
onMounted(async () => {
await Promise.all([
auth.fetchMe(),
fetchPaymentConfig(),
fetchDeviceProfiles(),
fetchAccounts(),
fetchRules(),
])
startReplyQueueSummaryPolling()
if (route.query.action === 'add') {
openAddModal()
const nextQuery = { ...route.query }
delete nextQuery.action
router.replace({ path: route.path, query: nextQuery })
}
// //
//
})
@@ -1738,40 +1935,70 @@ onUnmounted(() => {
</p>
</div>
<div class="header-actions">
<div v-if="startableAccounts.length > 0" class="batch-toolbar">
<div
v-if="selectableAccounts.length > 0 && (auth.canStartAccounts || auth.canDeleteAccounts)"
class="batch-toolbar"
>
<a-checkbox
:indeterminate="selectedStartableCount > 0 && selectedStartableCount < startableAccounts.length"
:checked="startableAccounts.length > 0 && selectedStartableCount === startableAccounts.length"
:indeterminate="selectedAccountCount > 0 && selectedAccountCount < selectableAccounts.length"
:checked="selectableAccounts.length > 0 && selectedAccountCount === selectableAccounts.length"
:disabled="batchStarting || batchDeleting"
@change="onSelectAllChange"
>
全选可启动 ({{ startableAccounts.length }})
{{ auth.canDeleteAccounts ? '全选当前页' : '全选可启动' }} ({{ selectableAccounts.length }})
</a-checkbox>
<a-button
v-if="auth.canStartAccounts && startableAccounts.length > 0"
type="primary"
ghost
class="batch-start-btn"
:disabled="selectedStartableCount === 0"
:disabled="selectedStartableCount === 0 || batchDeleting"
:loading="batchStarting"
@click="batchStartRpa"
>
<template #icon><PlayCircleOutlined /></template>
批量启动{{ selectedStartableCount ? ` (${selectedStartableCount})` : '' }}
</a-button>
<a-button v-if="selectedStartableCount > 0" class="batch-clear-btn" @click="clearSelection">
<a-popconfirm
v-if="auth.canDeleteAccounts"
:title="`确认删除选中的 ${selectedAccountCount} 个账号?运行中的托管会先停止,关联的自动回复规则和消息日志也会被清除。`"
ok-text="确认删除"
cancel-text="取消"
placement="bottomRight"
@confirm="handleBatchDeleteAccounts"
>
<a-button
danger
class="batch-delete-btn"
:disabled="selectedAccountCount === 0 || batchStarting"
:loading="batchDeleting"
>
<template #icon><DeleteOutlined /></template>
批量删除{{ selectedAccountCount ? ` (${selectedAccountCount})` : '' }}
</a-button>
</a-popconfirm>
<a-button
v-if="selectedAccountCount > 0"
class="batch-clear-btn"
:disabled="batchStarting || batchDeleting"
@click="clearSelection"
>
取消选择
</a-button>
</div>
<a-space wrap>
<a-popconfirm
v-if="auth.canStartAccounts"
title="将启动所有未启动的账号(含其他分页),确认继续?"
ok-text="全部启动"
cancel-text="取消"
@confirm="startAllRpa"
>
<a-button
type="primary"
ghost
:loading="startingAll || batchStarting"
type="primary"
ghost
:loading="startingAll || batchStarting"
:disabled="batchDeleting"
>
<template #icon><ThunderboltOutlined /></template>
一键启动全部
@@ -1797,6 +2024,7 @@ onUnmounted(() => {
购买额度
</a-button>
<a-button
v-if="auth.canCreateAccounts"
type="primary"
class="gradient-btn"
:disabled="!canAddAccount && !canPurchaseSlots"
@@ -1864,9 +2092,10 @@ onUnmounted(() => {
:class="{ 'account-card-selected': isAccountSelected(acc.id) }"
>
<a-checkbox
v-if="!acc.quota_disabled && (acc.status === 'offline' || acc.status === 'error')"
v-if="auth.canDeleteAccounts || (auth.canStartAccounts && !acc.quota_disabled && (acc.status === 'offline' || acc.status === 'error'))"
class="account-select-checkbox"
:checked="isAccountSelected(acc.id)"
:disabled="batchStarting || batchDeleting"
@change="toggleAccountSelect(acc.id)"
/>
<!-- 账号顶部信息 -->
@@ -1989,7 +2218,13 @@ onUnmounted(() => {
<template #icon><MessageOutlined /></template>
自动回复
</a-button>
<a-button type="text" size="small" class="action-edit-btn" @click="openEditModal(acc)">
<a-button
v-if="auth.canUpdateAccounts"
type="text"
size="small"
class="action-edit-btn"
@click="openEditModal(acc)"
>
<template #icon><EditOutlined /></template>
编辑
</a-button>
@@ -2000,7 +2235,7 @@ onUnmounted(() => {
</div>
<div class="account-actions-primary">
<a-button
v-if="!acc.quota_disabled && (acc.status === 'offline' || acc.status === 'error' || acc.status === 'starting')"
v-if="auth.canStartAccounts && !acc.quota_disabled && (acc.status === 'offline' || acc.status === 'error' || acc.status === 'starting')"
type="primary"
ghost
size="small"
@@ -2012,7 +2247,7 @@ onUnmounted(() => {
启动托管
</a-button>
<a-button
v-if="!acc.quota_disabled && acc.status === 'logging_in'"
v-if="auth.canStartAccounts && !acc.quota_disabled && acc.status === 'logging_in'"
type="primary"
size="small"
class="action-btn-warn"
@@ -2022,7 +2257,7 @@ onUnmounted(() => {
扫码登录
</a-button>
<a-button
v-if="!acc.quota_disabled && acc.status === 'online'"
v-if="auth.canStopAccounts && !acc.quota_disabled && acc.status === 'online'"
danger
ghost
size="small"
@@ -2032,12 +2267,19 @@ onUnmounted(() => {
停止托管
</a-button>
<a-popconfirm
v-if="auth.canDeleteAccounts"
title="确认删除该账号?删除后其所有的自动回复规则和消息日志也将被清除。"
ok-text="确认"
cancel-text="取消"
@confirm="handleDeleteAccount(acc.id)"
>
<a-button type="text" danger size="small" class="action-delete-btn">
<a-button
type="text"
danger
size="small"
class="action-delete-btn"
:disabled="batchDeleting"
>
<template #icon><DeleteOutlined /></template>
</a-button>
</a-popconfirm>
@@ -2052,7 +2294,7 @@ onUnmounted(() => {
<UserOutlined style="font-size: 4rem; color: var(--text-muted); margin-bottom: 16px;" />
<h3>暂无托管账号</h3>
<p style="color: var(--text-secondary); margin-bottom: 20px;">添加一个抖音账号开始自动化回复工作吧</p>
<a-button type="primary" class="gradient-btn" @click="openAddModal">
<a-button v-if="auth.canCreateAccounts" type="primary" class="gradient-btn" @click="openAddModal">
<template #icon><PlusOutlined /></template>
立即添加
</a-button>
@@ -2107,7 +2349,10 @@ onUnmounted(() => {
<a-form-item label="手机号 (可选,方便记录备注)">
<a-input v-model:value="addForm.phone" placeholder="请输入绑定的手机号码" />
</a-form-item>
<a-form-item label="Cookie 数据 (可选,可直接导入登录态)">
<a-form-item
v-if="auth.canManageCookies"
label="Cookie 数据 (可选,可直接导入登录态)"
>
<a-textarea
v-model:value="addForm.cookie_data"
:rows="10"
@@ -2257,7 +2502,7 @@ onUnmounted(() => {
placeholder="0 或留空则继承系统默认"
/>
<div class="field-hint">
设置 N 秒后同一账号的待回复会话会依次排 1 条在 N 发送 2 条在 2N 后发送以此类推各账号队列互不影响0 或留空表示继承系统默认当前生效 {{ editForm.reply_delay_effective }} 0 表示不启用兜底排队立即回复
设置 N 秒后同一账号队列为空时首条回复等待 0 并立即发送后续待回复会话按 N 2N 秒依次排队各账号队列互不影响实际发送仍受全局带宽队列保护0 或留空表示继承系统默认当前生效 {{ editForm.reply_delay_effective }} 0 表示不启用兜底排队收到消息后立即回复
</div>
</a-form-item>
</a-col>
@@ -2293,6 +2538,53 @@ onUnmounted(() => {
</div>
</a-form-item>
</a-col>
<a-col :span="24">
<a-form-item label="公网发送通道">
<div class="egress-channel-row">
<a-select
v-model:value="editForm.egress_public_ip"
:options="egressChannelOptions"
:loading="egressChannelsLoading"
placeholder="自动选择服务器默认公网出口"
style="flex: 1; min-width: 0;"
/>
<a-button
:loading="egressChannelsLoading"
@click="fetchEgressChannels(true)"
>
重新检测
</a-button>
</div>
<div class="field-hint">
<template v-if="egressChannels.length > 1">
已检测到 {{ egressChannels.length }} 个不同公网 IP固定选择后该账号的 IM 请求将绑定到对应本地网卡地址
</template>
<template v-else-if="egressChannels.length === 1">
当前仅检测到一个公网出口 {{ egressChannels[0].public_ip }}仍可提前保存自动切换次数增加出口后重新检测即可
</template>
<template v-else>
系统会自动检测服务器网卡与公网 IP 的映射未检测到时保持默认路由
</template>
</div>
<div v-if="egressChannelsError" class="egress-channel-error">
{{ egressChannelsError }}
</div>
</a-form-item>
</a-col>
<a-col :xs="24" :sm="12">
<a-form-item label="发送最多尝试通道数 N">
<a-input-number
v-model:value="editForm.egress_auto_attempts"
:min="1"
:max="8"
:precision="0"
style="width: 100%;"
/>
<div class="field-hint">
包含首选通道只有明确收到通道/安全校验失败时才按顺序切换超时等结果不确定的请求不会重发避免重复消息
</div>
</a-form-item>
</a-col>
<a-col :span="24">
<a-form-item label="伪装设备头(User-Agent">
<a-select
@@ -2320,7 +2612,10 @@ onUnmounted(() => {
启动托管时浏览器登录与 IM 发送将使用所选设备头须与 a_bogus 签名一致修改后请重新启动托管
</p>
<a-form-item label="Cookie 数据(Playwright storage_state JSON">
<a-form-item
v-if="auth.canManageCookies"
label="Cookie 数据(Playwright storage_state JSON"
>
<a-textarea
v-model:value="editForm.cookie_data"
:rows="isMobile ? 8 : 12"
@@ -2331,6 +2626,7 @@ onUnmounted(() => {
<div class="edit-modal-footer">
<a-popconfirm
v-if="auth.canManageCookies"
title="确认清除该账号的 Cookie?清除后需重新扫码登录。"
ok-text="确认"
cancel-text="取消"
@@ -2338,8 +2634,20 @@ onUnmounted(() => {
>
<a-button danger :loading="editSaving">清除 Cookie</a-button>
</a-popconfirm>
<a-button :loading="editSaving" @click="saveAccountInfo">保存账号信息</a-button>
<a-button type="primary" class="gradient-btn" :loading="editSaving" @click="saveCookie">
<a-button
v-if="auth.canUpdateAccounts"
:loading="editSaving"
@click="saveAccountInfo"
>
保存账号信息
</a-button>
<a-button
v-if="auth.canManageCookies"
type="primary"
class="gradient-btn"
:loading="editSaving"
@click="saveCookie"
>
保存 Cookie
</a-button>
</div>
@@ -2378,17 +2686,30 @@ onUnmounted(() => {
关闭后该账号不再发送兜底回复状态与策略中心的是否启用开关同步
</span>
</div>
<a-switch v-model:checked="accountDefaultRuleForm.is_active" />
<a-switch
v-model:checked="accountDefaultRuleForm.is_active"
:disabled="!auth.canWriteRules"
/>
</div>
<ReplyRuleEditor v-model:replies="accountDefaultRuleForm.replies" />
<ReplyRuleEditor
v-model:replies="accountDefaultRuleForm.replies"
:readonly="!auth.canWriteRules"
:can-upload-cards="auth.canWriteLinkCards"
/>
<a-space style="width: 100%; justify-content: flex-end; margin-top: 8px;">
<a-button @click="goAccountRulesPage(rulesModalAccount.id)">
<template #icon><SettingOutlined /></template>
管理全部规则
</a-button>
<a-button type="primary" class="gradient-btn" :loading="rulesModalSaving" @click="saveAccountDefaultRule">
<a-button
v-if="auth.canWriteRules"
type="primary"
class="gradient-btn"
:loading="rulesModalSaving"
@click="saveAccountDefaultRule"
>
保存兜底回复
</a-button>
</a-space>
@@ -2430,10 +2751,10 @@ onUnmounted(() => {
</div>
<a-alert
v-if="auth.user?.role === 'viewer'"
v-if="!auth.canWriteMessages"
type="info"
show-icon
message="当前账号为只读权限,可查看队列详情,但不能执行立即发送。"
message="当前账号无消息发送权限,可查看队列详情,但不能执行立即发送。"
class="reply-queue-alert reply-queue-readonly-alert"
/>
@@ -3014,6 +3335,24 @@ onUnmounted(() => {
background: rgba(255, 255, 255, 0.02) !important;
}
.batch-toolbar :deep(.batch-delete-btn.ant-btn-dangerous) {
color: #fca5a5 !important;
border-color: rgba(248, 113, 113, 0.45) !important;
background: rgba(239, 68, 68, 0.08) !important;
}
.batch-toolbar :deep(.batch-delete-btn.ant-btn-dangerous:not(:disabled):hover) {
color: #fecaca !important;
border-color: rgba(252, 165, 165, 0.75) !important;
background: rgba(239, 68, 68, 0.16) !important;
}
.batch-toolbar :deep(.batch-delete-btn.ant-btn-dangerous:disabled) {
color: rgba(203, 213, 225, 0.45) !important;
border-color: rgba(255, 255, 255, 0.08) !important;
background: rgba(255, 255, 255, 0.02) !important;
}
.batch-toolbar :deep(.batch-clear-btn.ant-btn-default) {
color: #cbd5e1 !important;
border-color: rgba(255, 255, 255, 0.16) !important;
@@ -4256,6 +4595,18 @@ onUnmounted(() => {
line-height: 1.55;
}
.egress-channel-row {
display: flex;
align-items: center;
gap: 10px;
}
.egress-channel-error {
margin-top: 6px;
color: #fbbf24;
font-size: 0.78rem;
}
.im-credential-grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
+75 -14
View File
@@ -2,18 +2,25 @@
import { ref, onMounted, onUnmounted } from 'vue'
import api from '../api'
import MessageBubble from '../components/MessageBubble.vue'
import { useAuthStore } from '../stores/auth'
import {
UserOutlined,
MessageOutlined,
CheckCircleOutlined,
ClockCircleOutlined,
ArrowRightOutlined,
ThunderboltOutlined
ThunderboltOutlined,
SettingOutlined,
PlusOutlined
} from '@ant-design/icons-vue'
const auth = useAuthStore()
const stats = ref({
totalAccounts: 0,
activeAccounts: 0,
myAccounts: 0,
myActiveAccounts: 0,
totalMessages: 0,
repliedMessages: 0,
replyRate: '0%'
@@ -24,14 +31,15 @@ const loading = ref(true)
const fetchStats = async () => {
try {
const [accountsRes, statsRes] = await Promise.all([
api.get(`/accounts`),
const [accountStatsRes, statsRes] = await Promise.all([
api.get(`/dashboard/account-stats`),
api.get(`/logs/stats`)
])
const accounts = accountsRes.data
stats.value.totalAccounts = accounts.length
stats.value.activeAccounts = accounts.filter(a => a.status === 'online').length
stats.value.totalAccounts = accountStatsRes.data.total_accounts || 0
stats.value.activeAccounts = accountStatsRes.data.online_accounts || 0
stats.value.myAccounts = accountStatsRes.data.my_accounts || 0
stats.value.myActiveAccounts = accountStatsRes.data.my_online_accounts || 0
//
stats.value.totalMessages = statsRes.data.total || 0
@@ -61,16 +69,21 @@ const fetchRecentLogs = async () => {
}
let statsInterval = null
const refreshVisibleStats = () => {
if (document.visibilityState === 'visible') fetchStats()
}
onMounted(() => {
fetchStats()
fetchRecentLogs()
// 10
statsInterval = setInterval(fetchStats, 10000)
//
statsInterval = setInterval(refreshVisibleStats, 60000)
document.addEventListener('visibilitychange', refreshVisibleStats)
})
onUnmounted(() => {
if (statsInterval) clearInterval(statsInterval)
document.removeEventListener('visibilitychange', refreshVisibleStats)
})
</script>
@@ -84,8 +97,27 @@ onUnmounted(() => {
多账户自动回复RPA后台支持快捷扫码登录状态持久化保存以及自定义关键字规则精准答复
</p>
</div>
<div class="banner-icon">
<ThunderboltOutlined style="font-size: 4rem; color: #c084fc; opacity: 0.3;" />
<div class="banner-side">
<div v-if="auth.hasPermission('menu.accounts')" class="banner-actions">
<router-link to="/accounts">
<a-button size="large">
<template #icon><UserOutlined /></template>
{{ auth.hasGlobalDataScope ? '账号管理' : `我的账号(${stats.myAccounts}` }}
</a-button>
</router-link>
<router-link
v-if="auth.canCreateAccounts"
:to="{ path: '/accounts', query: { action: 'add' } }"
>
<a-button type="primary" size="large" class="gradient-btn">
<template #icon><PlusOutlined /></template>
{{ auth.hasGlobalDataScope ? '添加账号' : '添加自己的账号' }}
</a-button>
</router-link>
</div>
<div class="banner-icon">
<ThunderboltOutlined style="font-size: 4rem; color: #c084fc; opacity: 0.3;" />
</div>
</div>
</div>
@@ -98,7 +130,7 @@ onUnmounted(() => {
<UserOutlined />
</div>
<div class="stat-info">
<span class="stat-label">托管账号</span>
<span class="stat-label">全平台托管账号</span>
<h2 class="stat-value">{{ stats.totalAccounts }}</h2>
</div>
</div>
@@ -111,7 +143,7 @@ onUnmounted(() => {
<CheckCircleOutlined />
</div>
<div class="stat-info">
<span class="stat-label">在线运行</span>
<span class="stat-label">全平台在线运行</span>
<h2 class="stat-value text-green">{{ stats.activeAccounts }}</h2>
</div>
</div>
@@ -199,8 +231,9 @@ onUnmounted(() => {
<div class="quick-actions-grid" style="margin-top: 20px;">
<router-link to="/accounts" class="quick-action-card">
<UserOutlined class="action-icon text-gradient" />
<span>账号配置</span>
<p>扫码登录并托管多个抖音账号</p>
<span>{{ auth.hasGlobalDataScope ? '账号管理' : '我的账号' }}</span>
<p v-if="auth.hasGlobalDataScope">管理全平台账号配置与运行状态</p>
<p v-else>仅查看和管理自己添加的账号当前 {{ stats.myAccounts }} </p>
</router-link>
<router-link to="/rules" class="quick-action-card">
@@ -225,6 +258,20 @@ onUnmounted(() => {
border-left: 4px solid var(--primary-color);
}
.banner-side {
display: flex;
align-items: center;
gap: 28px;
flex-shrink: 0;
}
.banner-actions {
display: flex;
gap: 12px;
flex-wrap: wrap;
justify-content: flex-end;
}
.stat-card {
display: flex;
align-items: center;
@@ -436,6 +483,20 @@ onUnmounted(() => {
display: none;
}
.banner-side,
.banner-actions {
width: 100%;
justify-content: flex-start;
}
.banner-actions > a {
flex: 1 1 180px;
}
.banner-actions :deep(.ant-btn) {
width: 100%;
}
.stat-card {
padding: 16px;
}
+30 -53
View File
@@ -18,6 +18,12 @@ import {
buildStickerPayload,
parseMessageContent
} from '../utils/messageContent'
import {
isGenericPeerName,
extractPeerUid,
matchPeerConversation,
resolveConversationId as resolvePeerConversationId
} from '../utils/peerMatch'
const route = useRoute()
const logs = ref([])
@@ -172,7 +178,7 @@ const onConvListScroll = (e) => {
const fetchAccounts = async () => {
try {
const res = await api.get('/accounts')
const res = await api.get('/account-options')
accounts.value = Array.isArray(res.data) ? res.data : res.data?.items || []
} catch (error) {
console.error(error)
@@ -241,26 +247,6 @@ const getAccountName = (accountId) => {
const getAccount = (accountId) =>
accounts.value.find((a) => Number(a.id) === Number(accountId)) || null
const isGenericPeerName = (name, peerUid = '') => {
const value = (name || '').trim()
if (!value) return true
if (peerUid && value === peerUid) return true
if (/^\d+$/.test(value)) return true
if (/^用户\d+$/.test(value)) return true
return false
}
const extractPeerUid = (conv) => {
const raw = String(conv?.sender_id || '').trim()
if (!raw) return ''
if (/^\d+$/.test(raw)) return raw
if (/^0:1:\d+:\d+$/.test(raw)) {
return raw.split(':')[3] || ''
}
const last = raw.split(':').pop()
return /^\d+$/.test(last || '') ? last : ''
}
const peerUidFromLog = (log) => {
const uid = extractPeerUid({ sender_id: log.sender_id })
if (uid) return uid
@@ -272,11 +258,18 @@ const peerUidFromLog = (log) => {
if (fromConv) return fromConv
}
}
if (log.sender_name && log.sender_name !== '[系统发送]') {
const byName = list.find(
(c) => c.sender_name === log.sender_name && extractPeerUid(c)
// 退 +
//
if (
log.sender_name
&& log.sender_name !== '[系统发送]'
&& !isGenericPeerName(log.sender_name)
) {
const byName = list.filter(
(c) => String(c.sender_name || '').trim() === String(log.sender_name).trim()
&& extractPeerUid(c)
)
if (byName) return extractPeerUid(byName)
if (byName.length === 1) return extractPeerUid(byName[0])
}
return ''
}
@@ -310,17 +303,8 @@ const convKey = (log, peerUid = undefined) => {
return `${log.account_id}::name:${log.sender_name || 'unknown'}`
}
const findPeerMeta = (conv) => {
const list = convListCache.value[conv.account_id] || []
const peerUid = extractPeerUid(conv)
return list.find(
(c) =>
c.sender_name === conv.sender_name ||
c.conversation_id === conv.sender_id ||
(peerUid && (c.sender_id === peerUid || extractPeerUid(c) === peerUid)) ||
(conv.sender_id && c.conversation_id?.endsWith(`:${conv.sender_id}`))
) || null
}
const findPeerMeta = (conv) =>
matchPeerConversation(convListCache.value[conv.account_id] || [], conv)
const enrichConversation = (conv) => {
const peer = findPeerMeta(conv)
@@ -333,8 +317,6 @@ const enrichConversation = (conv) => {
}
}
const isConvId = (id) => /^0:1:\d+:\d+$/.test(String(id || '').trim())
const fetchConvList = async (accountId) => {
if (!accountId) return []
if (convListCache.value[accountId]) {
@@ -349,20 +331,8 @@ const fetchConvList = async (accountId) => {
}
}
const resolveConversationId = (conv) => {
const peerUid = extractPeerUid(conv)
const list = convListCache.value[conv.account_id] || []
const match = list.find(
(c) =>
(peerUid && (c.sender_id === peerUid || extractPeerUid(c) === peerUid)) ||
c.sender_name === conv.sender_name ||
c.conversation_id === conv.sender_id ||
(conv.sender_id && c.conversation_id?.endsWith(`:${conv.sender_id}`))
)
if (match?.conversation_id) return match.conversation_id
if (isConvId(conv.sender_id)) return conv.sender_id
return conv.sender_id || ''
}
const resolveConversationId = (conv) =>
resolvePeerConversationId(convListCache.value[conv.account_id] || [], conv)
const isAccountOnline = (accountId) => {
const acc = accounts.value.find(a => a.id === accountId)
@@ -531,13 +501,20 @@ const sendMessage = async () => {
}
const conversationId = resolveConversationId(conv)
if (!conversationId) {
message.error('无法解析会话 ID,请刷新日志或到私信收发页重试')
message.error(
'无法确认这条会话对应的抖音用户(常见于昵称重复或对方资料未解析),'
+ '已阻止发送以免发错人;请到「私信收发」页选中该用户后再发'
)
return
}
sending.value = true
try {
const body = { conversation_id: conversationId, content }
//
//
const expectedPeerUid = extractPeerUid(conv)
if (expectedPeerUid) body.peer_uid = expectedPeerUid
const parsed = parseMessageContent(content)
if (parsed.type === 'sticker') {
body.message_type = 'sticker'

Some files were not shown because too many files have changed in this diff Show More