- What is this about?
- What are sprites/expression images?
- 6 vs 28 emotion model
- Img2Img vs Inpaint
- Example sprite creation (6 sprites and inpaint):
In this guide, I attempt to teach you how to generate expression images as efficiently as possible. I'm still learning, so feel free to send me any corrections or additions for this Guide 😀
Sprites can be used in Silly Tavern to allow character's to express emotions through images of themselves. Look at the extras documentation and the discord for more info.
There are two supported models listed in the extension API documentation. The default has 6 expression images and another version can be selected to increase the number to 28. More sprites mean that the expressions will change more often and there is a higher chance that an appropriate sprite will be selected. There are some problems with the 28 sprite version, though. An obvious downside is that you have to create almost 5x as many sprites. On top of that, many of the sprites will probably end up looking very similar. The SD models I've tried so far weren't really good at making subtle changes to facial details like the eyebrow positions or how much the mouth is opened. Even so, using 28 provides a better experience in my opinion and with the method I will teach you, generating them should be relatively quick and painless.
As far as I'm aware, there are two main ways people generate expressions. The first is image to image, which partly destroys the whole base image and then regenerates it based on a new prompt. This leads to images that look visually different from each other, which adds a nice touch of variety. The downsides are that small details on your character might change, which bothers some people.
The other Method is using inpaint. With this method, you can specifically target the face of a character by painting over it. Then the SD will only destroy and rebuild the part which has been painted over. There will be no variety outside the face, which might make the character look stiff and robotic. In exchange, you get consistent outputs and the option to automatically remove backgrounds. I will show how that works in the following example:
For my method of creating sprites, you will need two prompt lists. The first is a basic description of your character. You probably already know that part, so I won't elaborate.
The second list contains the emotion's that will be used as variables to automatically generate all expressions in one go. This can be used to generate 6 and 28 sprite sets relatively fast. This list consist of descriptions of the emotions separated by commas. You can't use commas for the individual emotions because the comma separates the variables.
Generate a batch of base images with a high contrast background. After generating the images, we will use the background removal extension in webui to remove all of their backgrounds and pick an image that works well. The extension is pretty hit or miss, so it's important that you check that your image is compatible before generating all the sprites.
In this example, I've already put the first emotion at the beginning of the prompt. The emotion you pick first will have a great impact on the general vibe of the following images. It will later be replaced with the other emotions when the time comes to generate a batch of all of them. Make sure to always describe the emotion by mentioning the state of different facial features. This has proven to give much better results than simply mentioning the emotion. You can use Ai to generate a list if you want to.
Emotions for 6 sprite version:
Anger, Fear, Joy, Love, Sadness, Surprise
Emotions for 28 sprite version:
Admiration, Amusement, Anger, Annoyance, Approval, Caring, Confusion, Curiosity, Desire, Disappointment, Disapproval, Disgust, Embarrassment, Excitement, Fear, Gratitude, Grief, Joy, Love, Nervousness, Neutral, Optimism, Pride, Realization, Relief, Remorse, Sadness, Surprise
Make sure you have the background removal plugin for Webui. Switch to Extras > Batch process and upload all the images you just uploaded. Upscaling is not needed yet so turn it to 1x and select a background removal model (u2netp and u2net work the best for me). This is my result:
Make sure you have the additional networks plugin installed. Pick one of the images that work well, or regenerate if you don't like any of them. Then import the base image into PNG Info and send it to inpaint. Paint over the face of your character. Make sure to avoid painting over the hair and outline of the face.
After that, scroll down to Additional Networks and select X/Y/Z plot. Change X type to Prompt S/R and leave Y and Z at none. Insert your list of emotions into X values, and make sure that the emotion you put at the beginning of your prompt is also at the beginning of the list inside X values. This will generate an image of each item in the list:
Repeat step 3 with the newly generated images, and this time you can upscale the image to a higher resolution. Make sure to select the correct images if you have other files in the output folder, they get sorted by name by default and not by creation date. Here is my result:
For the 6 sprite version, it works well enough to just import the images over the Silly Tavern UI. If you plan on generating 28 sprite versions, i suggest you use a simple script that renames the images to their corresponding emotion based on creation date. (ChatGPT is pretty good at helping with stuff like that)
As you can see, there are still some visible white edges. You can attempt to fix this by adjusting the sliders on the bg removal plugin in webui. Alternatively, if you know your way around Gimp or Photoshop, you can use a mask to remove them manually since they should all have the same outline or just leave the background in the sprite.
Feel free to ask questions or provide further input in the Sprites-Discussion channel on the Silly Tavern discord.
Good luck and remember to stay Frosty