Dataset and tagging workflow guide for LoRa training!

Last update: 2/12/2023

This guide and the accompanying scripts do not cover training. This guide focuses on sourcing your images and generating/scraping/cleaning up tags.

I recommend you read the guide at least once before starting, that way you can get a rough idea how how you want your workflow to look.

This guide is unfinished.



misc/setup

Installing:
check the github for up to date install instructions

Folder structure

The following step-by-step folder structure is used:

  • 0 - raw - raw images from the internet / screenshots
  • 1 - cropped - cropped images
  • 2 - sorted - images grouped by quality / topic / etc
  • 3 - tagged - generated or scraped tags + downscaling of images to uniform resolution
  • 4 - fixed - inbetween step, can be skipped, useful if you want to manually edit your images.
  • 5 - out - final images. point your training script here

Sanity check

You can make sure everything is ok by refreshing the webui. It will give you a count of how many images, categories and tags you have in each step!

0. LoRa type & getting your images

The first step is getting your training images. These unprocessed images will go into the 0 - raw folder.
A good amount of images is anywhere between 50-150. People report working character models with as low as 20 images.

Manually curating the images seems to have the biggest effect! Don't just automatically dump all images from the booru.

Character/Screenshot type

A screenshot type LoRa is intended to mimic the consistent style of one or more characters as they appear in the actual source material. You can think of it as a style+character LoRa, since all generated images will have both your character and the artstyle of the show.

Your source material should be the best quality release for the show you want to train on. Don't just take screenshots of a streaming site, get/find a BluRay/DVD/Webrip instead.

After you got your files locally, you want to make sure to pick a media player that makes taking screenshots and seeking the video as painless as possible.

TBA: it should be possible to train for multiple characters at the same time.
TBA: someone was working on an automated script for this?

MPC-HC

Make sure to get the modern release. Search engines tend to link to the old website.

Your default screenshot key is F5. It saves into the last-used directory, so using file=>save image save a single screenshot and save it to your output folder to set your default save path. (You can also just save to downloads and move the images later.)

Make the following changes to the player settings (some might already be set to these)

  • Under Options=>Player=>Keys, you want to assign the following keys (click the key field and press the required combination of keys)
    • Save Image (auto) = F5
    • Frame-step = Ctrl + Right Arrow
    • Frame-step-back = Ctrl + Left Arrow
    • Jump Forward (small) = Shift + Right Arrow
    • Jump Backward (small) = Shift + Left Arrow
    • Jump Forward = Right Arrow
    • Jump Backward = Left Arrow
  • Under Options=>Tweaks, set something reasonable for the small and medium jump distances. I found 1000 (1sec) works well for small and 3000 (3sec) for medium. You will use this to quickly skip individual scenes. Don't bother with the large skip, you'll miss entire scenes.
  • Under the same tab (Options=>Tweaks), enable "Show preview on seek bar". This allows you to quickly find scenes that have your character, without having to skip around manually. Optionally, disabling "Fast seek (on keyframes)" will make the preview images more accurate at the cost of more CPU/disk usage.

MPV

Same settings as above, so read the reasoning for them there. If you use MPV, you know where to put these:

input.conf

1
2
3
4
5
6
7
F5 screenshot
Ctrl+Right frame-step
Ctrl+Left frame-back-step
Shift+Right seek 1
Shift+LEFT seek -1
Right seek 3
LEFT seek -3

mpv.conf

screenshot-directory /home/(you)/dataset/0-raw

Seekbar previews:

https://github.com/TheAMM/mpv_thumbnail_script

When to take screenshots

In short: whenever your character appears

  • You want to take a screenshots every time your character appears in a scene, even if they are in the background.
  • If it's cel animation, try to look for the individual cels. They will often reuse them so you might get duplicates.

Character/Fanart type

Go to your booru of choice (danbooru or gelbooru) and start saving images.
Tips (for Gelbooru):

  • Try to limit tags by excluding certain terms from your search (eg. -lowres)
  • Put sort:score after your search to see higher rated images first.
  • Either enable Settings=>Options=>General Only Listing or add -rating:explicit to your search string
  • Either enable Always Show Original in the settings or just make sure you're always saving the full resolution image and not the preview.

Outfit

If you want to train a specific outfit, you have two choices:

  1. only save images where your character is wearing the desired outfit
    • only possible if there's an outfit tag and enough images
    • your trained LoRa won't be able to generate anything else
  2. save images into separate subfolders (eg 2_default and 2_alternate_costume) recommended
    • you can have a third folder for images where you can't tell (1_closeups)

Style type

This is about copying the style of a given artist. I've only tested one style LoRa so far, but I found that you want to save the following to a separate folder (or even skip them altogether):

  • very early art
  • WIPs and sketches
  • Commissions that look nothing like their usual stuff

TBA: Scraping using gallery-dl

1. Cropping images

Tip: If you want to do editing, start with your better images! If you do all of your image in one go you'll probably want to die by the time you're done.

Use one of the following tools to crop your images:

Aspect ratio

You absolutely want your images have a 1:1 aspect ratio (square image). The resolution doesn't matter, as you can automatically scale it up/down to fit your final training resolution.

Having to fit your character in a square severely restricts what you can do. The easiest thing you can do is crop just the face, but doing so too many times will result in your model having a hard-time generating good full body pictures. Another alternative is painting over or filling in the extra image area around your character with a color similar to the original background. I recommend you save both a closeup and a full version if you have relatively few images in your dataset.

I've been told that cropping to exact aspect ratios is no longer necessary since new training scripts can detect and group ("bucket") your images by resolution/aspect ratio. This is only partially true. It does save time, but cropping images manually can still benefit the overall model quality.

Editing

Anything in your image (unless tagged) will be part of what the model thinks makes up your character. Having parts of other characters, extra limbs, etc in the image has a chance to make your model considerably worse.
The simplest way is to paint over the problem areas with a color that is similar to the background behind them.
You can do this after you're done with the cropping. That way you don't have to load every single image into your image editor.

TBA: anything that isn't GIMP.

GIMP

I used GIMP for the cropping step so I can do editing at the same time. If you hate yourself too, you may follow along.

Change the following settings:

  • under Windows, make sure "Single-Window mode" is enabled
  • under Edit=>Preferences=>Image Windows=>Snapping enable "Snap to Canvas Edges" and up the snapping distance to 20
  • under Edit=>Keyboard Shortcuts set the following keys:
    • Crop to Selection = F2
    • Overwrite = F3
  • In the tool settings for the rectangle select tool (left side), set the drop down to "fixed - aspect ratio" and set "1:1" in the box under it.

Since you're overwriting your images, copy all your sorted images from 0 - raw to 1 - cropped before starting. Make sure you're working with the right folder.

Caveats:

  • When you open GIMP, make sure no images are open since otherwise it just loads every subsequent picture as a new layer.
  • Convert your images to PNG beforehand. JPEG files will give you a popup to select quality settings each time you hit overwrite.

You can open all your images by selecting all of them (ctrl+a with the folder open) and dragging them onto the main GIMP window.
If you're also doing cropping, then you will use the rectangle select tool (default key R) to select the area of the image containing your character. Hitting F2 will remove everything outside your selection.

If you end up with unwanted content in your selection (parts of other characters):

  • Use the Pencil Tool (N) and Ctrl+Left Click on a part of the background to select a color that matches what would be behind the part you want to erase, then
  • Use the Healing Tool (H) and Ctrl+Left Click in a similar manner to fix the "rough" edges/textures the pencil tool left.
  • Change the brush and brush size in the left window to make this easier.

You can save your work with F3. you will have to do this for each image
Before closing GIMP, make sure that all your images actually got saved by opening your work folder (2 - cropped). You can click discard changes when closing GIMP, as saving here would create a project file for each of your images.

2. Sorting images

TBA: create-sorting-groups.py is currently broken and the webui version isn't implemented yet. Create folders manually.

It's important to sort your images to emphasize better images. If you run create-sorting-groups.py, it will create two folders.

  • 3_good => This folder should contain your main set of images
    • character clearly visible / in foreground
    • detailed closeup
    • scenes where you have multiple characters but you can easily crop around your character
    • popular images/modern stuff by the artist
  • 1_medium => This should contain any misc/random images that might be worse quality
    • character looking away from camera
    • random closeups / "stylized" closeups
    • scenes where cropping your character is impossible/parts of another character would always be in frame
    • smear frames
    • early art/WIPs by the artist

The current easiest way to sort images is to have all 3 folders open (your category ones and the original 1 - cropped folder) and just drag-and-drop the images into the right window. This will be implemented in the webui eventually.

TBA: webui sorting.

in-detail about grouping

The number in front of the underscore is how many times your image will be repeated back to the AI. More repeats means your image is more important/training will focus on it, but it can also overfit for those images if you have this number too high (it will always try to reproduce those same images instead of generating more varied stuff).

You can change the number/make your own groupings - the rest of the scripts will handle it just fine. If you know more about training you can tune these for your usecase. The "name" after the underscore can be anything. Nothing ever uses it, not even for training afaik.

It's important to have a folder like 1_medium with slightly-worse images that depict your character in different poses / from different angles.
eg: If you don't have a single image of your character looking away, how would the AI know what the back of their head looks like.

3. Tagging

Autotagging

If you're using the stable-diffusion-webui, you should have the option to do the tagging and scaling of images (2 - sorted -> 3 - tagged) with the webui.

  • change the Interrogate: deepbooru score threshold option in the main settings tab to around 0.7 or 0.8 to get less random tags.
  • Under the Train tab, select Preprocess images.
    • Set your resolution to the required size
    • Make sure "Use deepbooru for caption" is checked.
      You can check the Rentry for more details on image acquisition.
      TBA: fix scrape-booru-tags.py

Better autotagging.

I highly recommend you use the Waifu Diffusion 1.4 Tagger Extension instead, as it seems to add less incorrect tags. Workflow is very similar to the step above but you get better quality steps. You can still use the preprocessing tab to resize your images.

Scraping

TBA: fix scrape-booru-tags.py instead of having to rely on autogenerated ones.

4. Tag pruning

TBA: Detailed explanation
https://github.com/city96/dataset-webui

Edit
Pub: 12 Feb 2023 21:04 UTC
Edit: 28 Feb 2023 19:27 UTC
Views: 1578