This guide is incomplete and does not appear to be working for 8GB cards. (Confirmed working for 10GB+ cards though.) For support go to https://github.com/oobabooga/text-generation-webui/issues/147
for an updated guide see https://rentry.org/llama-tard-v2
Get the updated HF Converted model weights here:
Run LLaMA on a potato card (8GB VRAM Edition)
So you don't have a beefy card but you want to run the latest and greatest LLaMA? Well look no further.
All you need to do is install https://github.com/oobabooga/text-generation-webui.
This repo defaults to the most efficient settings, to load the most model into the smallest card.
(It also gives you a nice webUI with a chat interface too.)
Acquiring the CORRECT "HF" Model Weights
But wait, there's one more thing. You need the MODEL WEIGHTS. But you don't need just any LLaMA model weights.
The original leaked weights won't work. You need the "HF" (HuggingFace) converted model weights.
You can get them by torrenting this magnet link:
Specifically you will need the 7B model. You don't need 13B, 30B or 65B-- but just go ahead and grab them (and seed!) to share the love.
How to tell if you have the HF Converted Weights
If you already have some weights and are not sure if they're the right ones, here's how you can tell.
The WRONG original leaked weights have filenames that look like:
The CORRECT "HF Converted" weights have filenames that look like:
So you got the right weights, now what?
Put them in
Now, from a command prompt in the text-generation-webui directory, run:
conda activate textgen
python server.py --model LLaMA-7B --load-in-8bit
You now have the latest and greatest running on your dinky gaming card!
Help I got an OOM error (or something)
If you run into trouble, ask for help at https://github.com/oobabooga/text-generation-webui/issues/147
BONUS: KoboldAI Support
KoboldAI also requires the HF converted model weights in the torrent above.
Simply place the weights in
Until KoboldAI merges the patch to support these weights you'll have to patch it yourself.
(KobaldAI will probably be patched soon, so you could also just wait if you're not feeling brave.)
How to patch KoboldAI for LLaMA support
Coming Soon Watch this space for updates