TRAINING A STABLE DIFFUSION LORA FOR RETARDS
Prerequisites:
This guide assumes you have a local copy of an SDXL model that you want to train a LORA on.
BUILDING A LORA, PART 1: PREPARING THE DATASET
To create a character/style LORA you must first gather a suitable number of images, cropped for the dimensions that you're going to be training on. SDXL primarily works with 1024x1024 images, so cropping your dataset to these dimensions is a safe bet. After that, you must tag these images in text files that share the filename of the image files you're tagging. For automatic tagging, you can use a program like Kohya_SS GUI (Also trains LORAs) from the following URL: https://github.com/bmaltais/kohya_ss
Install this, double click the gui file in the install directory, navigate to the "Utilities" tab, then the Captioning submenu (This is the default submenu for now). From there, I recommend using the WD14 tab for captioning. For characters or artists, you can add their name to the Prefix box. Feel free to fiddle around with tags you want to exclude, but default settings are mostly fine here. Hit Caption Images and wait for it to finish. You now have a tagged dataset for training.
QUICK NOTE ON DATASET IMAGES. "A suitable number" can vary, but generally you want to aim for 50 or more for characters/styles, and 100 or more for concepts. More is better, but means training takes longer which can be rough on weaker GPUs since training will absolutely consume your GPU while it's running. Training images should also clearly depict the subject (If a character) or the artist's style (If a style). You also get out what you put in, so try to avoid low quality, black and white, or otherwise poor examples of your character/style.
BUILDING A LORA, PART 2: ANNOYING FOLDER INFRASTRUCTURE
Now that you have a tagged dataset, it needs to be in a specific folder structure. Kohya_ss GUI also helps with this. Under the LoRA tab, navigate to the "Dataset Preparation" subtab. Select your captioned image folder, then set where you want this structure to take place. Make sure it's a location you can find, because this will be what the LORA trains on. Instance/Class Prompt can be whatever so just make sure it's descriptive. Repeats dictates how often the images should repeat so if you want lower repeats, set it to 5 or 10. Click prepare training data.
BUILDING A LORA, PART 3: WHAT THE FUCK ARE ALL THESE SETTINGS
Now that you've prepared your dataset, you need to determine a few things about the LORA you're going to make. Again, assuming you're using Kohya_ss GUI, go to the LoRA tab. In Source Model, click Model Quick Pick then select custom from the menu. Select the source model of your choice from your drive. For most local users, this will be the PonyXL file you use to generate images. Be sure to click the SDXL Model checkbox if you're using Pony or any other SDXL model. By default it saves as safetensors and this is good. From there, go to the Folders tab and select the img folder from the file structure you prepared in part 2. Then select the output folder of your choice, this is where the finished LORA will be saved. Change the model output name to whatever you want, and add any comments for other users. Then navigate to the Parameters tab.
Disclaimer: A lot of what's used in the Parameter's tab is a mixture of personal preference, GPU ability, and voodoo magic. Experiment freely, but be prepared to waste some time and GPU cycles doing so. Thankfully there are a number of presets in Kohya_ss to help ease the burden. For characters, the SDXL Lora AI Characters Standard 1.1 preset is fine if you have a >=12gb VRAM GPU. You can lower the Epoch amount to around ~15 as well, 50 is pretty overkill. Don't let the name fool you, these settings will work fine for both characters and styles. After you've picked your settings, you can click start training. Assuming everything was set up correctly you can expect a new LORA to pop up sometime later. This may be VERY MUCH later if you have a weaker GPU, and SDXL training on less than 12gb of VRAM requires some special tweaking which I'm still researching for now. But this should give you a starting point, just remember to post your LORAs ITT when you finish, we especially need more SDXL ones.
PARAMETER DESCRIPTIONS IN BRIEF:
Batch Size - How many images are trained at once. Higher batches means faster training, but requires a lot more VRAM.
Epoch - Every epoch represents 1 of each image multiplied by repeats being given as input to your LORA. More epochs just repeat this process. In my experience 5-10 epochs is plenty for training characters
LR Scheduler/Optimizer - Process that guides learning rates and the subprocess that optimizes it. Only really worth changing if you want to try to optimize your training. Constant or Linear and AdamW8bit save VRAM, but seem to have negligible impact on quality.
Learning Rate/Text Encoder Learning Rate - How fast your LORA learns. Generally you want to keep this between 0.0001 and 0.0005, lower rates may lead to taking longer to train, higher rates may fry your model
LR Warmup - How many steps per epoch are spent "warming up" the network instead of actually training it. I don't find that this does much beyond wasting time, so I tend to set it to zero. Some people like to keep it at 5% though, YMMV
Max Resolution - What resolution your network trains at. By default it's 512x512, but SDXL was trained at 1024x1024, so you'll want to aim for that resolution. Lower resolutions save VRAM and make training quicker. It is possible to train SDXL LORAs on lower resolutions, but quality may suffer.
Enable Buckets/Minimum Bucket Resolution/Maximum Bucket Resolution - Specifies the minimum and maximum resolutions for training images. Upper bounds should be the tallest/fattest image in your training dataset, lower bounds should be the shortest/skinniest. Definitely enable this if you have training images with different dimensions.
Network Rank (Dimension) - How complex your network is. Higher network rank produces LORAs that are larger on the disc, but allows them to understand more complicated character designs or artist styles. 16-32 works for 99% of all characters, 32 works for 99% of all styles and all but the most complex concepts. If you're unsure and want to be safe, like you're training a LoRA on John Carpenter's The Thing, increase this to 64. Network Alpha should be some multiple of this, typically 0.5x
All other Parameters not mentioned were considered too complex, obscure, or not worth changing to go into detail. Like I said, a lot of this stuff feels like voodoo.
TRAINING AT 8GB OF VRAM AKA HOW I LEARNED TO STOP WORRYING AND LOVE FP8
If you're a VRAMlet but you still want to train LORAs, make sure you have at least version 22.6.0 or newer of the Kohya_ss GUI. This update brought with it a tickbox in the "Advanced" subtab of the LORA Training Parameters tab called "fp8 base training (experimental)". Click to enable this feature, and be amazed as what once took days or even weeks now takes mere hours. What it does is convert the base model's precision levels into a more manageable size, allowing those with lower amounts of VRAM store the massive SDXL models without spilling over into slower memory spaces. While this sounds like it might ruin the quality of your LORAs, it only (temporarily) affects the model that you're training on, such as PonyXL or AutismMix. LORAs created are still saved with full precision. Further, this loss of precision only results in a roughly ~3% difference versus if the option wasn't enabled.
tl;dr, for those of us training at 8gb (Or possibly less) this is a great option that doesn't significantly impact LORA quality and is therefore a must-have.
FAQs and Troubleshooting
Q: I tried to train a LORA but it came out looking almost nothing like the character/style!
A: Your LORA is underbaked, either because you didn't train it for enough steps, the learning rate was too low, or you messed something up and the training dataset was untagged/not organized into the proper structure and the program didn't read it into the network. Try training it for more steps/epochs, and if that doesn't work double check your dataset/file structure.
Q: My LORA looks like the character, but has a ton of saturated colors or wonky details, she looks like she's melting!
A: Your LORA is overtrained. Try using an earlier epoch or if that fails, lowering your learning rate/total steps/epochs. Having too many repeats of the same image may also cause this
Q: Can I still train a LORA if I only have 6gb VRAM/4gb VRAM/2gb VRAM/an old thinkpad I stole from my IT temp job?
A: Yes if you enable fp8 training and gradient accumulation/yes but even with fp8 training it'll take a long time/probably not/definitely not. Sadly, training an SDXL LORA takes a lot of computing muscle. If you have bad hardware you can look into training on Google Colab or a similar cloud computing service for a small fee.