Contribution

1. Dynamic fine-control prompt editing framework

2. Effective results
• improve image aesthetics
• ensure semantic consistency between prompts and generated images
• align more closely with human preferences

3. Insightful findings
• artist names and texture-related modifiers enhance the artistic quality of generated images
• it is more effective to introduce these terms in the latter half of the diffusion process
• assigning a lower weight to complex terms promotes a more balanced image generation

Task

• a pre-trained text-to-image generative model

• user input text

• a modified prompt with fine-grained control, so that the generated image, , exhibits enhanced visual effects while remaining faithful to the semantics of the initial prompt .

Method

Dynamic Fine-control Prompt, DF-Prompt

• weights the token embeddings for controlling the overall influences of token during generation.

• is the normalized range that delineates the start and end steps during the iterative denoising process.

Overview

Stage 1: Plain Prompt Refinement,

• plain input prompt

• suffix modifiers one by one, until the model outputs the stop sign.

• i.e. construct
where .

Data Selection (Construct Dataset)

• start with a given prompt from publicly available prompt

• is split at a division point (the first comma in ).

• obtain the short prompt .

• remaining tokens form the modifier set .

• measures the image-text relevance by using pre-trained CLIP model

• returns the aesthetic score

• : tolerance constant

Fine-tuning

Stage 2: DF-Prompt Generation,

• initial text

• DF-Prompt

Method: online reinforcement learning

• initial state: initial text .

• action space: tripartite
• word space
• discrete time range space
• discrete weight space

• policy: policy model
At each step of online exploration, the model selects an action , in accordance with the policy model .

• reward funstion

Training