Repository navigation
[Feature] Log driver memory at RL train step boundaries - #2316
Open
PengchengShi00 wants to merge 1 commit into
Open
PengchengShi00 wants to merge 1 commit into
PengchengShi00 wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
本次改动把 RL 训练的 driver 内存观测统一到既有模块
xtuner/v1/utils/memory_watch.py:log_memory_event(tag, marker),按训练事件输出一行[MemReport]...(进程内存拆分),不受周期采样开关控制、默认始终输出。produce_utils.take_train_batch返回前打点(共卡/非共卡模式统一覆盖),以及BaseRLTrainer._log_step末尾打点(每个训练步)。main()用monitor_rank_memory(cfg.trainer.work_dir)包住build()+fit(),复用既有周期 JSONL 采样(需XTUNER_MEMORY_WATCH_ENABLED=1)。xtuner/v1/utils/mem_report.py已并入memory_watch.py并删除。改动规模:代码 5 个文件 +108/−3;最终提交
ea0e9eb2共 6 个文件 +274/−3(含 skill 文档.claude/skills/rl-memory-diagnosis/SKILL.md)。Review 意见处理状态
XTUNER_MEMORY_WATCH_ENABLED控制produce_utils.take_train_batchevent与MemoryEvent枚举同名不同义marker,docstring 说明二者无关_read_status/get_loggerlog_memory_event单测TestLogMemoryEvent4 个用例IndexError)_read_process_sections一致,/proc格式稳定)marker内含空格ea0e9eb2与代码同一 commit 提交;若仓库要求拆分可后续拆出Main Flowchart before this PR
flowchart TD subgraph SFT["xtuner/v1/train/cli/sft.py :: main()"] S1["monitor_rank_memory(trainer_cfg.work_dir)"] S2["Trainer.from_config(...) / Trainer.fit()"] end subgraph RLC["xtuner/v1/train/cli/rl.py :: main()"] R1["ray.init(address='auto')"] R2["Config.fromfile(config) + work_dir 覆盖"] R3["cfg.trainer.build()"] R4["BaseRLTrainer.fit()"] R1 --> R2 --> R3 --> R4 end subgraph TRAIN["xtuner/v1/train/rl_trainer.py :: BaseRLTrainer"] T1["fit() 每步循环"] T2["AgentLoopManager.produce_batch(...) / DisaggAgentLoopManager.get_batch(...)"] T3["BaseRLTrainer._log_step(...)"] T1 --> T2 T1 --> T3 end subgraph MW["xtuner/v1/utils/memory_watch.py"] W1["monitor_rank_memory(work_dir)"] W2["_MemoryWatch.__enter__ / _sample / __exit__"] W3["MemoryEvent: START / PERIODIC / STOP"] W1 --> W2 --> W3 end S1 --> W1 S2 -. 在 monitor 上下文内执行 .-> S1 R4 -. 未接入 .-x W1 T2 -. 无内存观测 .-> T2 T3 -. 无内存观测 .-> T3改动前,
memory_watch只服务 SFT 入口;RL 路径没有任何进程级内存观测。Main Flowchart after this PR
flowchart TD subgraph RLC["xtuner/v1/train/cli/rl.py :: main()"] R1["ray.init(address='auto')"] R2["Config.fromfile(config) + work_dir 覆盖"] R3["cfg.trainer.build()"] R4["BaseRLTrainer.fit()"] R5["with monitor_rank_memory(cfg.trainer.work_dir)"] R1 --> R2 --> R5 --> R3 --> R4 end subgraph TRAIN["xtuner/v1/train/rl_trainer.py / agent_loop_manager/produce_utils.py"] T1["fit() 每步循环"] T2["produce_utils.take_train_batch(...)"] T3["BaseRLTrainer._log_step(...)"] T1 --> T2 T1 --> T3 end subgraph MW["xtuner/v1/utils/memory_watch.py"] G0{"XTUNER_MEMORY_WATCH_ENABLED == 1 ?"} W1["monitor_rank_memory(work_dir)"] W2["_MemoryWatch 周期采样"] W3["log_memory_event(tag, marker) 始终执行"] W4["_read_status(Path('/proc/self/status'))"] W5["格式化 + get_logger().info(line)"] W1 --> G0 G0 -- "yes" --> W2 W3 --> W4 --> W5 end R5 --> W1 T2 -- "take_train_batch done" --> W3 T3 -- "train_step 结束" --> W3 W2 --> JSONL[("<work_dir>/memory/rank_*.jsonl")] W6 --> DRIVERLOG[("driver.log / stderr")] classDef changed fill:#ffe082,stroke:#f57f17,stroke-width:2px class R5,T2,W3,W4,W5,W6,G0 changed文字说明(高亮部分为本次修改):
xtuner/v1/train/cli/rl.py:build()+fit()移入with monitor_rank_memory(...),与 SFT 入口写法一致;开关关闭时直接透传。xtuner/v1/rl/agent_loop_manager/produce_utils.py::take_train_batch:在组装并返回ProduceBatchResult前打点一次。该函数是共卡AgentLoopManager.produce_batch与DisaggAgentLoopManager.get_batch的共同消费路径,一处接入覆盖两种模式;AgentLoopManager类本身已无改动。xtuner/v1/train/rl_trainer.py::_log_step:每个训练步结束时打点(三条训练路径共用该方法)。xtuner/v1/utils/memory_watch.py:log_memory_event单函数完成"读/proc/self/status→ 格式化 → 写日志";事件打点不受任何开关控制,周期采样仍由XTUNER_MEMORY_WATCH_ENABLED=1控制。核心原理
/proc/self/status是内核按需合成的虚拟文件,每次read返回读取时刻的进程状态;VmRSS/RssAnon/RssShmem/VmHWM来自 mm 的 per-CPU 计数,读取只需汇总,不遍历 VMA/页表,成本为微秒级,可安全放在每步路径上。smaps_rollup/numa_maps(O(VMA),被限流到 30 s 一次)相比,事件打点只取便宜字段,二者互补。_read_status返回字节数;log_memory_event按_GIB换算为 GiB,rss带GiB后缀,其余字段沿用原mem_report输出习惯,[MemReport]前缀保持与既有诊断命令兼容。XTUNER_MEMORY_WATCH_ENABLED=1控制:默认即可在训练日志中按 step 观察 driver 堆;需要 JSONL 曲线时再打开开关。核心实现与对应的单测
核心实现:
xtuner/v1/utils/memory_watch.py:501log_memory_event:单函数完成读/proc/self/status→ 按固定字段顺序格式化为 GiB →get_logger().info;缺失字段与非内存字段自动跳过;读取失败(OSError/ValueError)静默降级,无环境开关。xtuner/v1/rl/agent_loop_manager/produce_utils.py:660(take_train_batch返回前,共卡/非共卡共用);xtuner/v1/train/rl_trainer.py:1242(_log_step末尾)。对应单测:
tests/utils/test_memory_watch.py::TestLogMemoryEventtest_line_converts_bytes_to_gibrss单位后缀test_missing_and_non_memory_keys_are_skippedtest_logs_without_periodic_switchXTUNER_MEMORY_WATCH_ENABLED=0时仍输出一行test_unreadable_status_is_silent未运行任何代码或测试(本地无 GPU 环境);以上仅为静态检查(ruff check / format 通过)。
抽象与信息隐藏评估
_read_status,没有复制/proc解析逻辑;周期采样、cgroup/NUMA 等复杂细节仍封装在_MemoryWatch内。log_memory_event与MemoryEvent的语义区分已写入 docstring(marker 为自由文本,MemoryEvent仅用于 JSONL 记录)。_read_status与get_logger注入合成数据,不依赖真实/proc,测试成本可接受。剩余可选优化(非阻塞):
(name, key, suffix)三元组、空字符串后缀表达"仅 rss 带 GiB",略隐晦;若后续要扩展字段可改为显式格式表。新增功能守则
produce_utils.take_train_batch(函数)增加 1 次调用;BaseRLTrainer._log_step增加 1 次调用;rl.py::main增加 1 个with。AgentLoopManager/DisaggAgentLoopManager均零改动,无新增字段。memory_watch;调用点只有tag/marker两个标签。改动符合"既有类零污染"。
公开接口的线性业务流程评估
log_memory_event:读状态 → 格式化 → 打日志,三步同层级,符合 Composed Method;未再拆分纯函数(用户决定),避免无价值转发 wrapper。rl.py::main的接入是 2 行with,不引入额外私有函数。单测建议(已落实)
新增的 4 个用例已覆盖输出格式、开关独立性与异常降级。后续如需更完整:
take_train_batch增加轻量测试,断言其返回前调用了log_memory_event(monkeypatch 该函数即可,无需真实 replay buffer 之外的重依赖);_log_step的打点属于集成路径,建议依赖现有 RL 端到端用例覆盖,不单独构造。其他 issues
XTUNER_MEMORY_WATCH_ENABLED=1只控制周期 JSONL 采样。若后续担心主干日志噪声,可再评估是否加独立开关(本次明确不加)。marker内含空格(take_train_batch_done train_step=3)被包进[...]。 grep 不受影响;若希望机器解析更容易,可用:连接。_read_status中value.split()[0]可能抛IndexError,调用处只捕(OSError, ValueError);与_MemoryWatch._read_process_sections一致,实际风险很低,维持现状。work_dir。 未修改原有 docstring(本次不改变既有功能语义),共享存储场景由使用者自行设置较大的XTUNER_MEMORY_WATCH_INTERVAL;smaps_rollup/numa_maps每 30 s 的开销为已知取舍。.claude/skills/rl-memory-diagnosis/SKILL.md已随ea0e9eb2与代码同一 commit 提交。 文档与代码无耦合,若仓库偏好[SKILL]独立 commit(已有先例),可后续拆分。monitor_rank_memory的原有注释保持原样(未改动既有功能语义);仅新增的log_memory_event带 docstring,说明事件打点与周期采样开关相互独立。