eB's Stable Diffusion Inpainting Tutorial
Installation
WebUI
You will need to install a WebUI to interact with Stable Diffusion, I will be showing off everything through the Automatic1111 webui:
https://github.com/AUTOMATIC1111/stable-diffusion-webui
Alternatively you can use the vladmandic fork, which is what I have been using:
https://github.com/vladmandic/automatic
Models
You will need models to actually run on. There are sooo many out there with many pros and cons, so its about trying some out and seeing how they work for you.
As for me, the best I have found to date is epicrealism_pureEvolutionV2
Models get placed in models/Stable-diffusion
Controlnet
Install the controlnet extension (the vlad webui comes with it), google is your friend for installing extensions
Along with the extension you will need to download some models (.safetensors) and configs (.yaml) from here:
https://civitai.com/models/38784?modelVersionId=44811
I use openpose and depth, you are free to experiment with others
These get placed in your extensions/sd-webui-controlnet/models folder (extensions-builtin for vlad webui)
IMPORTANT: if you plan on using multiple controlnets, head over to Settings and search controlnet and there should be a slider for Multi ControlNet: Max models amount (requires restart), set this to 3 (or more) and restaret the webui
Custom Script
I run a custom script to make my life easier for this. To "install" the script, make a advanced_inpaint.py file in your scripts folder in the webui installation and paste the following into the contents. I will explain what this script does later.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | import modules.scripts as scripts
import gradio as gr
from modules import processing
from modules.processing import Processed
from modules.shared import state
from PIL import Image
import math
import cv2
import numpy as np
class Script(scripts.Script):
def title(self):
return "Advanced Inpaint"
def show(self, is_img2img):
return is_img2img
def ui(self, is_img2img):
with gr.Row():
pixel_count = gr.Slider(minimum=100, maximum=1000, step=10, label="Pixel Count (k)", value=500)
mask_dilate = gr.Slider(minimum=0, maximum=64, step=1, label="Mask Dilate", value=16)
with gr.Row():
do_pass_1 = gr.Checkbox(label="Do Pass 1", value=True)
do_pass_2 = gr.Checkbox(label="Do Pass 2", value=True)
do_pass_3 = gr.Checkbox(label="Do Pass 3", value=True)
with gr.Row():
pass_1_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 1 Denoise", value=1.00)
pass_2_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 2 Denoise", value=0.35)
pass_3_denoise = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Pass 3 Denoise", value=0.20)
with gr.Row():
pass_1_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 1 Steps", value=20)
pass_2_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 2 Steps", value=40)
pass_3_steps = gr.Slider(minimum=10, maximum=100, step=1, label="Pass 3 Steps", value=80)
return [
pixel_count, mask_dilate,
do_pass_1, pass_1_denoise, pass_1_steps,
do_pass_2, pass_2_denoise, pass_2_steps,
do_pass_3, pass_3_denoise, pass_3_steps,
]
def run(self, p, pixel_count, mask_dilate,
do_pass_1, pass_1_denoise, pass_1_steps,
do_pass_2, pass_2_denoise, pass_2_steps,
do_pass_3, pass_3_denoise, pass_3_steps,
):
if p.init_images[0] is None or p.image_mask is None:
return Processed(p, [], p.seed, "Empty image or mask")
width, height = p.init_images[0].size
resize = (width,height,)
pixel_count *= 1000
if (width * height) > pixel_count:
n = width / height
height = int(math.sqrt(pixel_count / n))
width = int(pixel_count / height)
resize = (width,height,)
p.init_images = [ p.init_images[0].resize(resize) ]
p.image_mask = p.image_mask.resize(resize)
p.height = height
p.width = width
state.begin()
# PASS 1
p.denoising_strength = float(pass_1_denoise)
p.steps = int(pass_1_steps)
processed = processing.process_images(p)
if len(processed.images) == 0:
print("FAILURE in first pass, got back 0 images")
return Processed(p, [], p.seed, "Empty image or mask")
# dilate the mask
if mask_dilate > 0:
kernel = np.zeros((mask_dilate*2+1, mask_dilate*2+1), np.float32)
kernel[mask_dilate, mask_dilate] = 1
kernel = cv2.GaussianBlur(kernel, kernel.shape, mask_dilate / 4)
img = np.asarray(p.image_mask)
img = cv2.dilate(img, kernel)
p.image_mask = Image.fromarray(img)
# PASS 2
if do_pass_2:
p.init_images = [ processed.images[0].resize(resize) ]
p.mask_blur = mask_dilate
p.denoising_strength = float(pass_2_denoise)
p.steps = int(pass_2_steps)
processed = processing.process_images(p)
if len(processed.images) == 0:
print("FAILURE in 2nd pass, got back 0 images")
return Processed(p, [], p.seed, "Empty image or mask")
# PASS 3
if do_pass_3:
p.init_images = [ processed.images[0].resize(resize) ]
p.mask_blur = mask_dilate
p.denoising_strength = float(pass_3_denoise)
p.steps = int(pass_3_steps)
processed = processing.process_images(p)
if len(processed.images) == 0:
print("FAILURE in 3rd pass, got back 0 images")
return Processed(p, [], p.seed, "Empty image or mask")
state.end()
return Processed(p, [processed.images[0]], p.seed, "Rescaled Image")
|
Running Things
Prompt
Once everything is installed properly, restart your webui and head over to the From Image > Inpaint area.
My positive prompt is something around:
This can be heavily edited or customized, just the starting point for me and what I like to do
Same goes for negative prompt, which I leave empty unless I need something specific out of a run
Controlnets
Scrolling down you will see a ControlNet vX.X.X section that you can expand
For each controlnet you use you will need to enable (and probably enable Low VRAM)
I use the following setup:
- Depth at 0.5 Control Weight
- OpenPose at 1.0 Control Weight
- Reference at 1.0 Control Weight
Script
At the very bottom is a section for scripts, select Advanced Inpaint that we created in the setup
The script will do the following:
- Downscale images (maintaining aspect ratio) to a desired pixel count. For exampe if I feed in a 2000x1000 images, that is 2 million pixels. A Pixel Count of 500 (thousand) will downscale the image to 1000x500
- Run an inpaint job (with controlnets) at
Pass 1 Denoisestrength andPass 1 Stepscount - Dilate your inpainting mask for Pass 2, aka growing your mask to make the inpaint be able to blend things better, controlled by
Mask Dilate - Using the new mask run an inpaint job (using original controlnets) at
Pass 2 Denoisestrength andPass 2 Stepscount
Using the script will allow for you to more easily get clean replacements of sections that blend well into their surroundings.
Feel free of course to tinker with running things manually and trying different techniques.
Hope you enjoy,
eB