# Images

> Text to image with Stable Diffusion 1.5, SDXL, FLUX.1 and Qwen-Image, on the Mac's GPU.

Image generation runs on the GPU engine, so it needs a Mac. The models are
written out in this repository: CLIP, T5, the UNet, the MMDiT, the VAEs and
both schedulers. Only convolution and matrix multiplication come from a
framework.

```bash
kvad pull stabilityai/stable-diffusion-xl-base-1.0
kvad images make "a red fox in fresh snow" \
    --model stabilityai/stable-diffusion-xl-base-1.0
```

The picture is written to `image-ID.png`, or to `--out FILE`. It is also kept
on the server with the settings that made it, where the web UI's Images page
and `kvad images` list it.

## Models

| Model | In memory | A step, on an M5 Pro | Steps |
|---|---|---|---|
| Stable Diffusion 1.5 | 1.9 GB | 0.75 s at 512² | 25 |
| SDXL | 6.4 GB | 3.1 s at 1024² | 30 |
| FLUX.1-schnell, q8 | 18.3 GB | about 7 s at 1024² | 4 |
| FLUX.1-dev, q8 | 17 GB | about 7 s at 1024² | 28 |
| Qwen-Image, q8 | about 29 GB | 28 s at 1024² | 20 |

SDXL and Qwen-Image run the denoiser twice per step while guidance is on, and
those figures include both passes. FLUX.1-schnell needs no guidance.
FLUX.1-dev takes a guidance scale, 3.5 unless said, as a number the model
reads: one pass a step, and no negative prompt. Its repository is gated and
its licence non-commercial, so it needs a Hugging Face token for an account
that has accepted the licence.

### Fine-tunes

SDXL and Stable Diffusion 1.5 fine-tunes load in any of the three layouts
people publish: a diffusers folder, a single checkpoint file in Stability's
own layout, or a file on your disk.

```bash
kvad pull Lykon/dreamshaper-8
kvad images make "a lighthouse at dusk" --model Lykon/dreamshaper-8
kvad images make "a lighthouse at dusk" --model ~/Downloads/some-sdxl-finetune.safetensors
```

A repository with several checkpoints is named as `repo:file.safetensors`.

`stabilityai/sdxl-turbo` draws in one step with no guidance, at 512², and
those are what it is given when a request does not say: about 0.3 s of
denoising on an M5 Pro, and 3 s to decode. Four steps are a little sharper.

```bash
kvad images make "a red fox in fresh snow" --model stabilityai/sdxl-turbo
```

### GGUF

Community GGUF files of Qwen-Image's and FLUX.1-schnell's transformers load by
the repository and the quantisation together. The rest of the model comes from
the `base_model` on the model card.

```bash
kvad pull city96/Qwen-Image-gguf:Q4_K_S
kvad images make "a red fox in fresh snow" --model city96/Qwen-Image-gguf:Q4_K_S
```

A Q4_K_S holds 9.1 GB less than q8 for Qwen-Image, and 5.5 GB less for FLUX,
for a step that is 5 to 18% longer.

## Options

```text
kvad images make PROMPT [--out FILE] [--model MODEL] [--size WxH] [--steps N]
                        [--guidance F] [--negative TEXT] [--seed N]
                        [--lora NAME[:SCALE]]...
```

Anything left out is the model's own default. `--lora` applies a
[LoRA](https://kvad.eu/docs/lora/) for this picture only.

```bash
kvad images            # the gallery, as a list
kvad images rm 12
```

## From a picture

`kvad images edit` makes a picture from one you give it. The picture is noised
part of the way and drawn over from there, so the prompt says what the result
should be, not what to change: "a green apple on a wooden table", not "make
the apple green". SDXL and Stable Diffusion 1.5 do this, and their fine-tunes.

```bash
kvad images edit "a green apple on a wooden table" --image apples.jpg --strength 0.5
kvad images edit "a blue ceramic bowl" --image apples.jpg --mask right-half.png
```

`--strength` is how far the picture is taken towards noise, above 0 and at
most 1, and 0.75 unless said. Near 0 it comes back as it was; at 1 nothing of
it is left. An edit runs that share of its steps.

With `--mask` only part of the picture is drawn anew, at strength 1 unless
said, and the rest comes back exactly as it went in. The mask is a picture the
same size: transparent where the model should draw, as OpenAI's masks are, or
white on black if it has no transparency. The web UI's Images page lets you
paint one on the picture.

Without `--size`, the result has the picture's own shape, at about as many
pixels as the model's own size. The server reads the picture with `ffmpeg`,
so it needs one installed.

These models were not trained to inpaint, so a mask's edge can show where what
is drawn does not meet what was kept. Models trained to follow an editing
instruction (FLUX.1 Kontext, Qwen-Image-Edit) are not here yet.

## From the API

`POST /v1/images/generations` is OpenAI's images endpoint, with the settings
OpenAI has no field for beside its own.

```bash
curl http://127.0.0.1:5823/v1/images/generations \
  -H 'content-type: application/json' \
  -d '{
    "model": "stabilityai/stable-diffusion-xl-base-1.0",
    "prompt": "a red fox in fresh snow",
    "size": "1024x1024",
    "steps": 30,
    "guidance_scale": 5,
    "seed": 5,
    "response_format": "url"
  }'
```

With `"stream": true` the server sends an event per denoising step, a rough
preview beside each when `partial_images` asks for one, and a completed event
per image. See [OpenAI-compatible API](https://kvad.eu/docs/api/#images).

`POST /v1/images/edits` is OpenAI's edits endpoint: the same fields, with an
`image`, an optional `mask` and `strength`, as a multipart form or as JSON
with each file a `data:` URL.

```bash
curl http://127.0.0.1:5823/v1/images/edits \
  -F image=@apples.jpg \
  -F prompt="a green apple on a wooden table" \
  -F strength=0.5 \
  -F response_format=url
```

## Memory

The VAE decode is the most memory an image asks for: it is the only stage at
full resolution. If a picture fails with a message about GPU memory, another
model is probably loaded. `kvad ps` shows what is, and `kvad unload` frees it.

## The pictures on the front page

The four pictures on [the front page](https://kvad.eu/) were made with the commands below,
on an M5 Pro, with SDXL's defaults: 1024², 30 steps, guidance 5. Each took 82
to 86 seconds to denoise and 9 to decode. The same seed draws the same
picture.

```bash
M=stabilityai/stable-diffusion-xl-base-1.0
kvad images make "a lighthouse on a rocky Norwegian coast at dusk, calm sea, warm light in the lamp room, long exposure photograph" --model $M --seed 11
kvad images make "a red fox in fresh snow at the edge of a birch forest, early morning light, wildlife photograph" --model $M --seed 5
kvad images make "a wooden stave church in a green valley under low clouds, watercolour" --model $M --seed 3
kvad images make "an open book on a wooden desk by a window, a single candle, oil painting, Dutch golden age" --model $M --seed 21
```