eB's Stable Diffusion Inpainting Tutorial

Installation

WebUI

You will need to install a WebUI to interact with Stable Diffusion, I will be showing off everything through the Automatic1111 webui:
https://github.com/AUTOMATIC1111/stable-diffusion-webui

Alternatively you can use the vladmandic fork, which is what I have been using:
https://github.com/vladmandic/automatic

Models

You will need models to actually run on. There are sooo many out there with many pros and cons, so its about trying some out and seeing how they work for you.

As for me, the best I have found to date is epicrealism_pureEvolutionV2

Models get placed in models/Stable-diffusion

Controlnet

Install the controlnet extension (the vlad webui comes with it), google is your friend for installing extensions

Along with the extension you will need to download some models (.safetensors) and configs (.yaml) from here:
https://civitai.com/models/38784?modelVersionId=44811

I use openpose and depth, you are free to experiment with others

These get placed in your extensions/sd-webui-controlnet/models folder (extensions-builtin for vlad webui)

IMPORTANT: if you plan on using multiple controlnets, head over to Settings and search controlnet and there should be a slider for Multi ControlNet: Max models amount (requires restart), set this to 3 (or more) and restaret the webui

Custom Script

I run a custom script to make my life easier for this. To "install" the script, make a advanced_inpaint.py file in your scripts folder in the webui installation and paste the following into the contents. I will explain what this script does later.

import modules.scripts as scripts
import gradio as gr
from modules import processing
from modules.processing import Processed
from modules.shared import state
from PIL import Image
import math
import cv2
import numpy as np

class Script(scripts.Script):
    def title(self):
        return "Advanced Inpaint"

    def show(self, is_img2img):
        return is_img2img

    def ui(self, is_img2img):
        with gr.Row():
            pixel_count = gr.Slider(minimum=100, maximum=1000, step=10, label="Pixel Count (k)", value=500)
            mask_dilate = gr.Slider(minimum=0, maximum=64, step=1, label="Mask Dilate", value=16)
        with gr.Row():
            do_pass_1 = gr.Checkbox(label="Do Pass 1", value=True)
            do_pass_2 = gr.Checkbox(label="Do Pass 2", value=True)
            do_pass_3 = gr.Checkbox(label="Do Pass 3", value=True)
        with gr.Row():
            pass_1_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 1 Denoise", value=1.00)
            pass_2_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 2 Denoise", value=0.35)
            pass_3_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 3 Denoise", value=0.20)
        with gr.Row():
            pass_1_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 1 Steps", value=20)
            pass_2_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 2 Steps", value=40)
            pass_3_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 3 Steps", value=80)
        return [
            pixel_count, mask_dilate,
            do_pass_1, pass_1_denoise, pass_1_steps,
            do_pass_2, pass_2_denoise, pass_2_steps,
            do_pass_3, pass_3_denoise, pass_3_steps,
        ]

    def run(self, p, pixel_count, mask_dilate,
        do_pass_1, pass_1_denoise, pass_1_steps,
        do_pass_2, pass_2_denoise, pass_2_steps,
        do_pass_3, pass_3_denoise, pass_3_steps,
    ):
        if p.init_images[0] is None or p.image_mask is None:
            return Processed(p, [], p.seed, "Empty image or mask")

        width, height = p.init_images[0].size
        resize = (width,height,)

        pixel_count *= 1000
        if (width * height) > pixel_count:
            n = width / height
            height = int(math.sqrt(pixel_count / n))
            width = int(pixel_count / height)
            resize = (width,height,)
            p.init_images = [ p.init_images[0].resize(resize) ]
            p.image_mask = p.image_mask.resize(resize)
            p.height = height
            p.width = width

        state.begin()

        # PASS 1
        p.denoising_strength = float(pass_1_denoise)
        p.steps = int(pass_1_steps)
        processed = processing.process_images(p)
        if len(processed.images) == 0:
            print("FAILURE in first pass, got back 0 images")
            return Processed(p, [], p.seed, "Empty image or mask")

        # dilate the mask
        if mask_dilate > 0:
            kernel = np.zeros((mask_dilate*2+1, mask_dilate*2+1), np.float32)
            kernel[mask_dilate, mask_dilate] = 1
            kernel = cv2.GaussianBlur(kernel, kernel.shape, mask_dilate / 4)
            img = np.asarray(p.image_mask)
            img = cv2.dilate(img, kernel)
            p.image_mask = Image.fromarray(img)

        # PASS 2
        if do_pass_2:
            p.init_images = [ processed.images[0].resize(resize) ]
            p.mask_blur = mask_dilate
            p.denoising_strength = float(pass_2_denoise)
            p.steps = int(pass_2_steps)
            processed = processing.process_images(p)
            if len(processed.images) == 0:
                print("FAILURE in 2nd pass, got back 0 images")
                return Processed(p, [], p.seed, "Empty image or mask")

        # PASS 3
        if do_pass_3:
            p.init_images = [ processed.images[0].resize(resize) ]
            p.mask_blur = mask_dilate
            p.denoising_strength = float(pass_3_denoise)
            p.steps = int(pass_3_steps)
            processed = processing.process_images(p)
            if len(processed.images) == 0:
                print("FAILURE in 3rd pass, got back 0 images")
                return Processed(p, [], p.seed, "Empty image or mask")

        state.end()

        return Processed(p, [processed.images[0]], p.seed, "Rescaled Image")

Running Things

Prompt

Once everything is installed properly, restart your webui and head over to the From Image > Inpaint area.

My positive prompt is something around:

Highest quality, masterpiece, photorealistic, medium shot, RAW photo, of sexy girl, flashing, revealing tits, breasts out of shirt, exposing boobs, (highly detailed skin), skin texture

This can be heavily edited or customized, just the starting point for me and what I like to do

Same goes for negative prompt, which I leave empty unless I need something specific out of a run

Controlnets

Scrolling down you will see a ControlNet vX.X.X section that you can expand

For each controlnet you use you will need to enable (and probably enable Low VRAM)

I use the following setup:

  1. Depth at 0.5 Control Weight
  2. OpenPose at 1.0 Control Weight
  3. Reference at 1.0 Control Weight

Script

At the very bottom is a section for scripts, select Advanced Inpaint that we created in the setup

The script will do the following:

  1. Downscale images (maintaining aspect ratio) to a desired pixel count. For exampe if I feed in a 2000x1000 images, that is 2 million pixels. A Pixel Count of 500 (thousand) will downscale the image to 1000x500
  2. Run an inpaint job (with controlnets) at Pass 1 Denoise strength and Pass 1 Steps count
  3. Dilate your inpainting mask for Pass 2, aka growing your mask to make the inpaint be able to blend things better, controlled by Mask Dilate
  4. Using the new mask run an inpaint job (using original controlnets) at Pass 2 Denoise strength and Pass 2 Steps count

Using the script will allow for you to more easily get clean replacements of sections that blend well into their surroundings.

Feel free of course to tinker with running things manually and trying different techniques.

Hope you enjoy,
eB

Edit

Pub: 05 Aug 2023 02:52 UTC

Edit: 07 Aug 2023 00:02 UTC

Views: 1255