AI Generates Images as Editable Transparent Layers

A research method called LayerDiffuse lets diffusion models generate transparent images directly, along with separate foreground and background layers that can be edited independently. The work was published in the paper “Transparent Image Layer Diffusion using Latent Transparency” by Lvmin Zhang and Maneesh Agrawala (https://arxiv.org/abs/2402.17113), and code is distributed under the Apache-2.0 license.

Most image generators produce opaque pictures, so anyone wanting a cutout subject has to run a separate background-removal or matting step afterward. LayerDiffuse instead trains the model to output an alpha channel as part of generation, which matters for content that simple masking handles poorly, such as glass, semi-transparent glowing effects, fur, hair, and whiskers.

Latent Transparency

The central idea in the paper is a technique the authors call “latent transparency.” Rather than adding alpha as an extra channel that would require retraining the variational autoencoder, the method encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. Transparency is added as a latent offset designed to cause minimal disruption to the model’s original latent distribution. Because the change is small, the authors describe the approach as one that can finetune existing latent diffusion generators into transparent image generators rather than requiring a model built from scratch.

A dedicated VAE pipeline handles the encoding and decoding. According to the Forge extension documentation, the transparent encoder extracts a latent offset that is added to images to assist transparency diffusion, and the decoder takes the standard VAE output plus the latent image and produces the final PNG.

Layers and Conditional Control

Beyond single transparent images, the method supports multiple transparent layers and can be adapted to various conditional control systems. The paper lists applications including foreground-conditioned and background-conditioned layer generation, joint layer generation, and structural control of layer contents.

The reference Forge integration (github.com/layerdiffusion/sd-forge-layerdiffuse) exposes these capabilities concretely. It supports both SDXL and SD1.5, and includes directional models such as foreground-to-background and background-to-foreground, plus joint models that generate foreground, background, and a blended composite together. Joint generation uses an attention sharing module so the separate diffusion processes stay coordinated. The extension also allows assigning independent prompts to the foreground, background, and blended layers, which keeps a single global prompt from corrupting individual layers.

The documentation notes practical constraints tied to how many images each mode emits. Generating everything together produces three images, so the batch size has to be divisible by three; background-to-foreground produces two, so its batch size must be divisible by two. Using the wrong batch size yields noise rather than usable output.

Reported Results

The paper reports a user study in which participants preferred the natively generated transparent content over a generate-then-matte baseline in 97 percent of cases. Participants also rated the quality as comparable to commercial transparent assets such as those from Adobe Stock. The training data consisted of 1 million transparent image layer pairs gathered with a human-in-the-loop collection scheme. The authors state the technique can be applied to different open-source image generators, which is reflected in the extension’s support for more than one base model family.