Skip to content

[Fix] Align GLM-5.3 Flash clamped SwiGLU rounding with LMDeploy - #2304

Open
qescccczmr wants to merge 1 commit into
InternLM:agent-sandboox-rl-previewfrom
qescccczmr:fix/glm53-lmdeploy-swiglu-rounding
Open

qescccczmr wants to merge 1 commit into
InternLM:agent-sandboox-rl-previewfrom
qescccczmr:fix/glm53-lmdeploy-swiglu-rounding

Conversation

@qescccczmr

@qescccczmr qescccczmr commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

BF16 clamped SwiGLU currently rounds the SiLU result before multiplying by the up projection. LMDeploy's precise-multiply path retains that intermediate in FP32, introducing a training/rollout rounding difference even when both sides use BF16 activations.

This adds an opt-in XTUNER_GLM53_LMDEPLOY_SWIGLU=1 path for GLM-5.3 Flash clamped SwiGLU. It reuses LMDeploy's CUDA forward kernel and registers a mathematical backward in FP32 with the same clamp boundaries. Both fused gate/up and separate gate/up callers are covered. The custom operator includes fake-tensor registration for full-graph compilation.

Set the variable before importing XTuner in every training worker. The enabled CUDA path requires LMDeploy with silu_and_mul(..., precise_mul=True) support; LMDeploy is imported lazily. The default path and CPU/NPU fallback retain their existing behavior. Separate gate/up callers concatenate their inputs for the serving kernel.

Validation: 15 focused tests passed on H200 with PyTorch 2.9.1 / CUDA 12.8, covering exact serving-forward parity (BF16/FP32, empty and non-last split dimensions), VJPs against an FP64 oracle including clamp boundaries, CPU fallback, AOT full-graph backward, and a nonzero SGD update. Ruff lint/format and git diff --check passed.

This addresses the activation rounding boundary only. Full-model PPO KL improvement has not yet been measured. KPool, FP8 quantization, KDA, and DeepEP changes are outside this PR.

@qescccczmr qescccczmr changed the title [Fix] Align GLM-5.3 clamped SwiGLU rounding with LMDeploy [Fix] Align GLM-5.3 Flash clamped SwiGLU rounding with LMDeploy Nov 12, 2026
@qescccczmr
qescccczmr changed the base branch from main to agent-sandboox-rl-preview November 12, 2026 11:34
@qescccczmr
qescccczmr force-pushed the fix/glm53-lmdeploy-swiglu-rounding branch from 2bef3aa to 7f738b4 Compare November 12, 2026 12:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant