Skip to content

Fix pooled text conditioning dropout in SDXL InstructPix2Pix training - #15247

Open
mzeynali wants to merge 1 commit into
huggingface:mainfrom
mzeynali:fix/sdxl-instruct-pix2pix-conditioning-dropout
Open

mzeynali wants to merge 1 commit into
huggingface:mainfrom
mzeynali:fix/sdxl-instruct-pix2pix-conditioning-dropout

Conversation

@mzeynali

Copy link
Copy Markdown

What does this PR do?

Fixes text conditioning leakage in SDXL InstructPix2Pix training. When a prompt was dropped, encoder_hidden_states was replaced with empty-prompt conditioning, but add_text_embeds still contained the original prompt's pooled embedding.

This change also replaces the pooled embedding with the second text encoder's empty-prompt output, using the same dropout mask.

Validation

Added a regression test using tiny, real SDXL components and local synthetic data:

  • With full text dropout, changing only the prompt produces identical trained UNet weights.
  • Without dropout, changing the prompt affects the trained weights.

Both cases pass with the fix. The full-dropout case fails on the original code.

Lint, formatting, and check_ai.py passed. make style and make fix-copies passed in an isolated snapshot. GPU training was not tested.

AI-assisted self-review

Reviewed against the repository guidelines. No blocking findings or dead code were identified, and no findings were intentionally left unresolved.

@github-actions github-actions Bot added examples size/M PR with diff < 200 LOC labels Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples size/M PR with diff < 200 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants