LyCORIS / LORA / DreamBooth tutorial

Foreword

This is the written part of the tutorial that describes my process of creating DreamBooth models and their further extractions into LORA and LyCORIS models.

This document covers basic info regarding my DreamBooth installation, all the scripts I use and will provide links to all the needed tools and external sources of information (i.e. detailed DreamBooth installation guide).

The detailed process of training data set preparation is covered (and shown) in the video part of the tutorial.

Video available here: https://www.veed.io/view/58216d46-9d45-4e7f-8e93-5115f27b695c?sharingWidget=true&panel=share
And also here: https://civitai.com/models/45539

DreamBooth

I use InB4DevOps version of DreamBooth (which is a tweak of ShivamShrirao) - it can be installed on linux and windows under wsl2 (not tried to do it in native windows but maybe it is now possible). There are other dreambooths (and allegedly the A1111 Dreambooth uses the version that has the same improvement that InB4DevOps applied, but I have not tested).

Please see NerdyRodent's installation guide of ShivamShrirao (you can find it in the links section). Once you have it installed - you can then replace the code with InB4DevOps repo (that is how I did, since I started on ShivamShrirao; you could probably start with InB4DevOps directly but I have not tested installing it that way :P)

If you are doing DreamBooth using different scripts - then you may have different quality. But if my image selection tips are helpful to you and your DreamBooth quality is acceptable, then you can use the scripts listed below to convert it to LORA/LyCORIS just fine!

Scripts

concepts_list_morgan.json:

1
2
3
4
5
6
7
8

[
    {
        "instance_prompt":      "photo of sks person",
        "class_prompt":         "photo of a person",
        "instance_data_dir":    "../../../data/morgan",
        "class_data_dir":       "../../../data/person"
    }
]

Note: for a male I use "person" and for a female I use "woman".
As in tutorials for Dreambooth, I use regularization class images, they are in data/person and data/woman respectively, these are not special, just generated 500 images using sd1.5 with basic prompts of photo of sks person and photo of sks woman respectively

Note2: instead of sks you could use other token, or some even say that it is bad to use sks because it is also a weapon. In the past, using 1.4 it could indeed make a person hold a rifle occasionally. It does not really happen anymore with custom models. An example of a good token is zwx

train_morgan.sh:

1
2
3
4
5
6
7

export MODEL_NAME="/home/malcolmrey/models/realistic2"
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
export CLASS_DIR="/home/malcolmrey/dreambooth/data/person"
export INSTANCE_DIR="/home/malcolmrey/github/data/morgan"
export OUTPUT_DIR="/home/malcolmrey/dreambooth/models/morgan"

accelerate launch train_dreambooth.py  --pretrained_model_name_or_path=$MODEL_NAME  --pretrained_vae_name_or_path="stabilityai/sd-vae-ft-mse"  --instance_data_dir=$INSTANCE_DIR  --output_dir=$OUTPUT_DIR  --with_prior_preservation --prior_loss_weight=1.0  --seed=3434554  --resolution=512  --train_text_encoder  --train_batch_size=1  --shuffle_after_epoch  --use_8bit_adam  --gradient_checkpointing  --gradient_accumulation_steps=1  --learning_rate=1e-6  --lr_scheduler="constant"  --lr_warmup_steps=0  --num_class_images=500  --max_train_steps=3000  --concepts_list="concepts_list_morgan.json"

Note: the learning rate differs between plain Shivam and the improved version, I'm using the one provided by the improved version. The max training steps currently is at 3000 but for quite a while I was using 2500. I believe it should be tweaked to your liking based on the base model (for realistic 2.0 it seems that 3000 is better than 2500). The shuffle_after_epoch is important and comes from the improved version. It will make the process slower, but will also increase the quality. This is set up to run under 11 GB VRAM, if you have over - you can remove the 8bit_adam. If you have 8 GB, you can try the other params that are listed in the repository and it may work (I have not tested it as I did not need it).

Note2: MODEL_NAME is the path to custom model saved locally, if you for whatever reason want to train using one of the base models then this is what you need to have: export MODEL_NAME="CompVis/stable-diffusion-v1-4" or export MODEL_NAME="runwayml/stable-diffusion-v1-5"

convert ckpt to diffusers:

1

python .\convert.py --checkpoint_path .\realisticVisionV20_v20.ckpt --dump_path ./output

convert diffusers to ckpt:

1

python convert_to_ckpt.py --half --model_path /home/malcolmrey/dreambooth/models/morgan/3000 --checkpoint_path checkpoints/model_morgan_v1_realistic2_3000.ckpt

convert dreambooth to LORA (from within Kohya):

1

.\venv\Scripts\python.exe "networks\extract_lora_from_models.py" --save_precision fp16 --save_to "D:/Development/StableDiffusion/Models/Lora/lora_morgan.pt" --model_org "D:/Development/StableDiffusion/Models/realisticVisionV20_v20.ckpt" --model_tuned "C:/Development/StableDiffusion/stable-diffusion-webui/models/Stable-diffusion/model_morgan_v1_realistic2_3000.ckpt" --dim 320

convert dreambooth to LyCORIS (from within KohakuBlueleaf/LyCORIS):

1

python extract_locon.py --safetensors --device=cuda --mode="fixed" --linear_dim=64 --conv_dim=32 C:/Development/StableDiffusion/stable-diffusion-webui/models/Stable-diffusion/realisticVisionV20_v20.ckpt F:/Development/StableDiffusion/SDModels/Models/2023-04-17/model_morgan_v1_realistic2_3000.ckpt C:/Development/StableDiffusion/stable-diffusion-webui/models/Stable-diffusion/locon/locon_morgan_v1_from_v1_64_32.safetensors

Important - how to use LyCORIS in A1111 WebUI

A1111 should be in rather recent version (at least from March 2023) and you should have the following extension installed: https://github.com/KohakuBlueleaf/a1111-sd-webui-locon
If you encounter some problems, I suggest to ask about it on the guide page at civitai where the comment section is available: https://civitai.com/models/45539/dreambooth-lycoris-lora-guide

Links

• https://www.youtube.com/watch?v=w6PTviOCYQY # Nerdy Rodent's ShivamShrirao Dreambooth installation tutorial
• https://github.com/ShivamShrirao/diffusers/tree/main/examples/dreambooth # ShivamShrirao Repo (Dreambooth)
• https://github.com/InB4DevOps/diffusers/tree/main/examples/dreambooth # Upgraded Repo (Dreambooth)
• https://github.com/KohakuBlueleaf/a1111-sd-webui-locon # A1111 LyCORIS plugin
• https://github.com/bmaltais/kohya_ss # Kohya SS
• https://github.com/KohakuBlueleaf/LyCORIS -> https://github.com/KohakuBlueleaf/LyCORIS/blob/main/tools/extract_locon.py # lycoris extractor
• https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py # ckpt to diffusers converter
• https://github.com/lkwq007/stablediffusion-infinity/blob/master/convert_checkpoint.py # another ckpt to diffusers converter
• https://gist.github.com/Christopher-Hayes/636ba25e0ae2e7020722d5386ac2571b # convert diffusers to ckpt
• https://www.ghisler.com/ # Total Commander (can be used freely as shareware)
• https://www.irfanview.com/ # Irfan View (free)
• https://bulkimagedownloader.com/ # Bulk Image Downloader (paid, but should have free trial period)
• https://www.captureone.com/en # CaptureOne (paid, but has free trial period)

Closure

If you feel like this tutorial was useful to you or perhaps you just like the work I do and wanted to support the things I do, here is a way: my buymeacoffee page: https://www.buymeacoffee.com/malcolmrey (the ko-fi page is no longer working, but this one is)

PS. I am currently working on SDXL LoRA guide so stay tuned :)

Thanks in advance and good luck with your trainings!

I've recently made an SDXL LoRA guide, it is available here: https://rentry.co/sdxl-lora-training