SDXL-Based Diagram Generation System
Unlike traditional image generation focused on natural photos, this system specializes in structured scientific/technical diagrams where text slots, nodes, and edges must be preserved.
Building a diagram generation system using Stable Diffusion XL enhanced with ControlNets and LoRA fine-tuning. Documenting progress, experiments, and technical developments.
Progress, experiments, and findings as the research evolves.
Unlike traditional image generation focused on natural photos, this system specializes in structured scientific/technical diagrams where text slots, nodes, and edges must be preserved.
More updates coming soon
Follow along as the research progresses
stabilityai/stable-diffusion-xl-base-1.0
Paired with custom VAE: madebyollin/sdxl-vae-fp16-fix
Two ControlNet branches:
Enforces nodes + edges alignment
Text mask region placement
Low-rank adapters (rank=16) applied to UNet attention layers for efficient training.
Each sample includes:
• LoRA weights merged into UNet
• Dual ControlNet conditioning
• StableDiffusionXLControlNetPipeline