My awful guide to kobold 8-bit on windows.
Follow this:
Ignore the part about lowram and nobreakmodel as well as disabling lazyload, just don't touch the splitting slider from all on gpu and it will work.

Use only the bitsandbytes portion of this and follow the install part for that:

Help, 13B Erebus takes all of my ram:
Split it with this script: (I went with 25 shards)
Make sure to copy any of the files from the first to the second that don't exist except for pytorch_model.bin.

Pub: 22 Nov 2022 10:20 UTC
Edit: 25 Nov 2022 12:41 UTC
Views: 1209