Advanced Engineering Informatics · 2026

DesignLoRA

Interactive Fine-Tuning of Text-to-Image Models with Designer-in-the-Loop Feedback

Seung Won Lee1,2   Yejin Yun1,2   Kyung Hoon Hyun1,2,*

1Department of Interior Architecture Design, Hanyang University 2Human-Centered AI Design Institute, Hanyang University *Corresponding author

A short demonstration of DesignLoRA's iterative workflow, from designer feedback to model adaptation and candidate refinement.

DesignLoRA Conceptual Overview

Overview of DesignLoRA: an image generator with LoRA is refined over rounds; the VLM evaluates candidates and the designer selects and asks questions, progressively steering results.
At each round, the designer asks a question and selects an image. A vision-language model evaluates the candidates using the question and selection history, and DesignLoRA updates the image generator for the next round.

Abstract

Although integration of text-to-image models into the design process requires context-sensitive, personalized control, current tools offer limited support. This misalignment forces prompt–regeneration loops, disrupting design ideation flow and wasting computational resources. However, most low-rank adaptation (LoRA) methods are trained offline with fixed reward signals and do not reflect context-specific preferences. We present DesignLoRA, a designer-in-the-loop method combining VLM-guided gradient distillation with preferential Bayesian optimization. DesignLoRA addresses a limitation of visual question answering-based fine-tuning: its reliance on explicitly stated preferences. By combining preference- and question-conditioned VLM probability signals, it enables rapid fine-tuning aligned with context-specific preferences. A comparative study with 18 designers showed DesignLoRA achieves higher preference alignment with fewer iterations and lower perceived effort. Qualitative evaluation with professional designers revealed advantages in exploratory workflows, where it complements newer AI models by capturing aesthetic preferences beyond explicit prompts. These findings point to a path toward near-real-time personalized control for evolving AI models.

How it works

DesignLoRA pipeline: input preparation, LoRA-based generation, BTL preference-context construction, dual-prompt VLM evaluation, LoRA adaptation, EI-based candidate sampling, and next-panel construction.
One interaction round, from image generation and VLM evaluation to LoRA adaptation and candidate selection.
01

Collect feedback

The designer asks a question and selects one image from the current set.

02

Evaluate images

A VLM scores each candidate against the question and the accumulated selection history.

03

Update LoRA

The two VLM losses are combined to update the LoRA while the base generator remains frozen.

04

Select candidates

Expected Improvement selects the images shown in the next round.

Experimental interface

DesignLoRA interface: a four-image candidate panel, prompt box, adjustable condition attributes, and a natural-language question box.
The interface shows four candidates alongside the input prompt, design conditions, and a question field. The designer selects an image and enters a question before the next round.

Results with other models

DesignLoRA applied to FLUX.2 and Qwen-Image generators, showing progressive refinement across rounds.
Examples using FLUX.2 and Qwen-Image. See Appendix B for details.

BibTeX

@article{lee2026designlora,
  title     = {Design {LoRA}: Interactive fine-tuning of text-to-image
               models with designer-in-the-loop feedback},
  author    = {Lee, Seung Won and Yun, Yejin and Hyun, Kyung Hoon},
  journal   = {Advanced Engineering Informatics},
  volume    = {76},
  pages     = {105053},
  year      = {2026},
  month     = nov,
  issn      = {1474-0346},
  doi       = {10.1016/j.aei.2026.105053},
  url       = {https://doi.org/10.1016/j.aei.2026.105053},
  publisher = {Elsevier BV}
}