Ayumi's LLM Role Play & ERP Ranking Archive 2 (Results from 2023-10-04)

This Ranking employs currently three important metrics to rank the models: The ALC-IQ (Ayumi LLM Character IQ), the ERP Score and the ERP Variety Score. Keep in mind though: this is just an automated benchmark employing rather primitive metrics, and it can't cover rating the quality of the generated output. It can only cover how seemingly well a Large Language Model (LLM) can understand character cards (see the ALC-IQ) and secondly if it can be used in to generate lewd responses (see the ERP Score and ERP Variance Score). The ERP benchmark is currently only based on a single character ('Ayumi') and a single fixed erotic setting, which is eventually subject to change. A few details about the testing procedure can be found further down.

The benchmark for the ALC-IQ works by letting the character answer how much they agree with a statement about their personality in a role playing chat log prompt. The character has to answer by writing a number between 1 and 5 (1 - disagree, 2 - slightly disagree, 3 - neutral, 4 - slightly agree, 5 - agree) to a statement they were presented with. The result will then be compared with the expected answer and the deviation from that is recorded. For more details refer to the section Ayumi LLM Character IQ - ALC-IQ.

The ERP Score is the average ratio of lewd words the model generates in a response, which is limited to 100 tokens. For more details refer to the section ERP Score

The third and rather new metric is the ERP Variety Score, this score measures the range of lewd words the model showed to generate in the responses for the ERP Score. This means, the models not only need to generate responses with many lewd words, but also with many different lewd words.

Emoji Key

ALC-IQ Emoji	Meaning	ERP Emoji	Meaning
⭐🧠	Best of High ALC-IQ Class, shows excellent understanding of the character cards in a role play chat.	🌶🌶	Very spicy model, capable of generating lots of lewd words
🧠	High ALC-IQ Class, shows excellent understanding of character cards in a role play chat.	🌶	Spicy model, capable of generating many lewd words
⭐📖	Best of Good ALC-IQ Class, shows good understanding of character cards in a role play chat.	👌	Likely not censored model, but generates probably short answers or fewer lewd words
📖	Good ALC-IQ Class, still gets details of the character cards in a role play chat.	🧊	Very possibly censored/SFW aligned model
⭐🤔	Best of Lower ALC-IQ Class, has it's challenges with details of the character card in a role play chat.
🤔	Lower ALC-IQ Class, certainly challenged with the character card in a role play chat.	❄	The ERP word variety of this model is great, it shows creative variety of lewd word usage.
⭐🤪	Best of Dumb ALC-IQ Class, very very challenged to get the character card in a role play chat.	✳	This model still shows knowledge of various lewd words, but there are better ones.
🤪	Dumb ALC-IQ Class, seems to be completely confused or has other issues getting the character card in a role play chat.	♻	This model has limitations in knowledge and usage of lewd words. It likely repeatedly uses the same words across regenerations.

Rank Symbol	Meaning
🥇 🥈 🥉	These medals are assigned broadly to the top ranked models. This is partially to give an impression how well these might work for you and partially also signal that there is no single definitive best model.
🎓	Top ALC-IQ ranks get this one.
🍆	Top ERP ranks get this one.

3B-7B Models

2023-11-01 Benchmark Re-Run V3: I currently run a completely new benchmark. Until I get around to update this page, you may find the most recent results here: http://ayumi.m8geil.de/ayumi_bench_v3_results.html

Rank	ALC-IQ Rank	ERP Rank	ALC-IQ	ERP Score	ERP Var Score	Model
🥇 1	🎓 1	🍆 3	⭐🧠 95.33	🌶🌶 30.25	❄ 131	🥇🎓🍆 Mistral Claude Chat 7B Q5_K_M
🥇 2	🎓 4	🍆 13	⭐🧠 91.88	🌶 21.54	❄ 152	🥇🎓🍆 Mistral ClaudeLimaRP v3 7B Q5_K_M
🥇 3	🎓 5	🍆 17	⭐🧠 90.73	🌶🌶 24.09	❄ 130	🥇🎓🍆 Mistral RP 0.1 7B Q5_K_M
🥇 4	🎓 6	🍆 16	⭐🧠 87.04	🌶🌶 25.56	❄ 128	🥇🎓🍆 Synthia v1.3 7B Q5_K_M
🥇 5	🎓 11	🍆 12	⭐🧠 84.33	🌶🌶 24.13	❄ 134	🥇🎓🍆 Samantha Mistral 7B Q5_K_M
🥇 6	🎓 2	🍆 27	⭐🧠 92.68	🌶 22.40	❄ 130	🥇🎓🍆 Mistral v0.1 7B Q5_K_M
🥇 7	🎓 9	🍆 21	⭐🧠 84.79	🌶🌶 27.76	✳ 120	🥇🎓🍆 Kuchiki 7B Q5_K_M
🥇 8	🎓 7	🍆 26	⭐🧠 86.75	🌶🌶 23.93	❄ 128	🥇🎓🍆 PetrolLM 7B Q5_K_M
🥇 9	29	🍆 11	⭐🧠 81.51	🌶🌶 25.50	❄ 132	🥇🍆 Zaraxls 7B Q5_K_M
🥇 10	🎓 14	🍆 31	⭐🧠 83.47	🌶 19.64	❄ 134	🥇🎓🍆 Zarafusionex 1.2 7B Q5_K_M
🥇 11	🎓 13	36	⭐🧠 83.53	🌶 22.97	❄ 122	🥇🎓 Hermes Limarp 7B Q5_K_M
🥇 12	🎓 17	35	⭐🧠 82.95	🌶🌶 27.63	✳ 114	🥇🎓 Zarablend 7B Q5_K_M
🥇 13	46	🍆 5	📖 79.55	🌶🌶 24.99	❄ 149	🥇🍆 MistRP v1.1 7B Q8_0
🥇 14	33	🍆 22	🧠 81.22	🌶 21.21	❄ 141	🥇🍆 Zarafusionex 7B Q5_K_M
🥇 15	🎓 10	50	⭐🧠 84.39	🌶🌶 30.30	102	🥇🎓 Zarablend 1.1 7B Q5_K_M
🥇 16	32	🍆 24	🧠 81.28	🌶 22.55	❄ 134	🥇🍆 Zarablendex VQ 7B (link broken) Q5_K_M
🥇 17	🎓 3	62	⭐🧠 92.45	🌶 19.75	✳ 117	🥇🎓 Kimiko Mistral 7B Q5_K_M
🥈 18	52	🍆 4	📖 78.80	🌶🌶 26.43	❄ 140	🥈🍆 Mistral Instruct v0.1 7B Q5_K_M
🥈 19	19	46	⭐🧠 82.43	🌶 19.14	❄ 130	🥈 Zarafusionex 1.1 7B Q5_K_M
🥈 20	🎓 8	67	⭐🧠 84.91	🌶 21.01	✳ 113	🥈🎓 Hermes LimaRP 7B Q5_K_M
🥈 21	20	54	⭐🧠 82.37	🌶 20.15	✳ 120	🥈 Zarafusionix 7B Q5_K_M
🥈 22	24	56	⭐🧠 81.91	🌶 19.25	✳ 120	🥈 Krakowiak 7B Q4_K_M
🥈 23	28	55	⭐🧠 81.74	🌶🌶 26.60	104	🥈 Zarablend M 7B Q5_K_M
🥈 24	34	48	🧠 80.82	🌶 20.38	❄ 122	🥈 Vigogne 2 7B Q5_K_M
🥈 25	22	65	⭐🧠 82.20	🌶🌶 26.39	101	🥈 Kuchiki 1.1 7B Q5_K_M
🥈 26	44	41	🧠 79.72	🌶🌶 27.21	✳ 111	🥈 Zarablend MX 7B Q5_K_M
🥈 27	59	🍆 29	📖 77.36	🌶 20.83	❄ 136	🥈🍆 Zaramix 7B Q5_K_M
🥈 28	63	🍆 25	📖 77.07	🌶 22.57	❄ 133	🥈🍆 LLaMA-2 Guanaco 7B Q5_1
🥈 29	55	37	📖 78.23	🌶 21.98	❄ 123	🥈 AstraMix 7B Q5_K_M
🥈 30	🎓 12	93	⭐🧠 83.64	🧊 13.36	✳ 120	🥈🎓 LLaMA 2 Monika V0.3B 7B Q5_1
🥈 31	41	60	🧠 79.84	🌶 22.22	✳ 112	🥈 Medusa 1.1 7B Q5_K_M
🥈 32	40	63	🧠 80.13	🌶 22.31	✳ 112	🥈 Hermes Kimiko 7B Q5_K_M
🥈 33	45	66	📖 79.67	🌶 19.01	✳ 119	🥈 Typly Pigeon 7B Q4_K_M
🥈 34	🎓 15	107	⭐🧠 83.12	👌 15.63	✳ 109	🥈🎓 LLaMA-2 7B Q8_0
🥈 35	37	83	🧠 80.24	🌶🌶 23.30	100	🥈 Zaraxe 7B Q5_K_M
🥈 36	48	70	📖 79.32	🌶 19.22	✳ 118	🥈 Nous Hermes 7B Q5_K_M
🥈 37	25	98	⭐🧠 81.80	👌 16.05	✳ 113	🥈 Dugong 7B Q5_1
🥈 38	68	49	📖 75.63	🌶 19.27	❄ 123	🥈 LLaMA-2 Coder 7B Q5_K_M
🥈 39	94	🍆 18	🤔 68.78	🌶🌶 25.34	❄ 128	🥈🍆 Hermesboros Limarp 7B Q5_K_M
🥈 40	99	🍆 14	🤔 67.40	🌶 22.80	❄ 142	🥈🍆 Vicuna 1.3 7B Q8_0
🥈 41	102	🍆 15	🤔 66.30	🌶🌶 26.00	❄ 128	🥈🍆 Airoboros GPT4 1.4.1 7B Q5_K_M
🥈 42	109	🍆 7	🤔 63.59	🌶🌶 25.59	❄ 137	🥈🍆 Samantha Mistral Instruct 7B Q5_K_M
🥉 43	54	73	📖 78.63	🌶 19.61	✳ 114	🥉 LosslessMegaCoder Mini 7B Q5_K_M
🥉 44	110	🍆 6	🤔 63.48	🌶🌶 28.11	❄ 131	🥉🍆 Airoboros GPT4 1.2 7B Q4_K_M
🥉 45	67	58	📖 75.69	🌶 21.05	✳ 116	🥉 Airoboros 2.1 7B Q5_K_M
🥉 46	30	104	⭐🧠 81.51	👌 16.64	✳ 109	🥉 LLaMA 2 7B Q5_1
🥉 47	31	103	🧠 81.34	🧊 11.74	✳ 118	🥉 Tsukasa Limarp 7B Q5_K_M
🥉 48	82	42	📖 72.24	🌶 20.90	❄ 126	🥉 Orca Mini v3 7B Q5_K_M
🥉 49	91	🍆 32	🤔 69.30	🌶🌶 23.61	❄ 122	🥉🍆 Wizard Vicuna Uncensored 7B Q5_K_M
🥉 50	80	53	📖 72.58	🌶 21.15	✳ 118	🥉 Marcoroni 7B Q5_K_M
🥉 51	18	129	⭐🧠 82.78	👌 13.67	99	🥉 LLaMA-2 PeanutButter v19 R8 7B Q5_K_M
🥉 52	103	🍆 28	🤔 65.73	🌶🌶 28.20	✳ 117	🥉🍆 Frank Uncensored 7B Q5_K_M
🥉 53	111	🍆 20	🤔 63.31	🌶 20.40	❄ 152	🥉🍆 OpenBuddy OpenLLaMA v5 7B Q3_K
🥉 54	60	82	📖 77.25	🌶 21.40	106	🥉 Airoboros 2.2 7B Q5_K_M
🥉 55	49	96	📖 79.15	👌 17.32	✳ 109	🥉 LlongOrca 16K 7B Q5_K_M (ext. context maybe broken)
🥉 56	121	🍆 10	🤔 61.52	🌶🌶 30.29	❄ 127	🥉🍆 Airoboros GPT4 7B Q4_K_M
🥉 57	23	128	⭐🧠 82.14	👌 15.95	93	🥉 Befenghuang Vigogne 2 Chat 7B Q5_K_S
🥉 58	105	🍆 30	🤔 65.26	🌶🌶 25.24	✳ 120	🥉🍆 Airoboros GPT4 1.4.1 Limarp 7B Q5_K_M
🥉 59	81	59	📖 72.52	🌶🌶 23.94	✳ 109	🥉 Spicyboros 2.2 7B Q5_K_M
60	88	51	🤔 71.77	🌶 19.62	✳ 121	Ganchengguang Yoko Japanse v0 7B Q5_K_S
61	42	109	🧠 79.72	🌶 18.09	96	Airoboros L2 2.2.1 7B Q5_K_M
62	118	🍆 19	🤔 62.21	🌶 23.23	❄ 132	🍆 Guanaco 7B Q5_K_M
63	101	40	🤔 66.71	🌶🌶 23.73	✳ 118	WizardLM V1.0 Uncensored 7B Q5_K_M
64	26	130	⭐🧠 81.80	👌 14.08	96	Jindo Instruct Pre-Alpha 7B Q5_K_M
65	87	57	📖 71.83	👌 16.87	❄ 130	LLongMA-2 Storysummarizer 7B Q5_K_M (ext. context maybe broken)
66	43	117	🧠 79.72	👌 17.65	95	Saiga 2 7B Q5_K
67	93	61	🤔 68.95	🌶🌶 28.27	95	Xwin LM V0.1 7B Q5_K_M
68	69	90	📖 75.58	🌶 18.86	✳ 109	MythoChizuru Mini 7B Q4_K_M
69	117	🍆 33	🤔 62.38	🌶🌶 28.04	✳ 115	🍆 Airoboros GPT4 1.3 7B Q4_K_M
70	83	74	📖 72.06	🌶 20.98	✳ 111	Saiga 7B Q5_1
71	🎓 16	155	⭐🧠 83.06	🧊 5.05	♻ 73	🎓 MedLLama 7B Q5_K_M
72	89	68	🤔 71.66	🌶🌶 24.71	104	Luna AI LLaMA-2 Uncensored 7B Q5_K_M
73	145	🍆 1	🤪 53.80	🌶🌶 28.09	❄ 146	🍆 Marx 3B Q5_1
74	35	134	🧠 80.47	👌 15.28	89	LLaMA-2 LoRA Assemble 7B Q5_K_M
75	21	151	⭐🧠 82.26	🧊 5.96	♻ 76	LLaMA 2 Delphi v0.2e 7B (link broken) Q5_1
76	146	🍆 2	🤪 53.80	🌶🌶 28.09	❄ 146	🍆 EverythingLM 3B Q5_1
77	57	110	📖 77.94	👌 13.90	✳ 110	Beluga Limarp 7B Q5_K_M
78	47	123	📖 79.38	👌 15.13	100	Kimiko 7B Q5_K_M
79	84	79	📖 72.00	👌 16.93	✳ 120	Pygmalion 7B Q5_1
80	66	102	📖 76.15	👌 14.98	✳ 112	LLaMA-2 Instruct 32K 7B Q5_K_M (ext. context maybe broken)
81	38	136	🧠 80.13	👌 13.84	93	LLaMA-2 Mistral 7B Q5_K_M
82	79	88	📖 72.81	🌶🌶 25.14	93	WizardMath V1.0 7B Q5_K_M
83	86	80	📖 71.83	🌶🌶 29.93	♻ 81	Airoboros GPT4 2.0 LLaMA-2 7B Q5_K_M
84	148	🍆 9	🤪 53.63	🌶🌶 25.02	❄ 139	🍆 Open LLaMA Open Instruct 7B Q8_0
85	77	95	📖 73.56	🌶 18.64	107	MythoLogic Mini 7B Q5_K_M
86	73	100	📖 74.65	👌 17.27	107	Pygmalion 2 7B Q5_K_M
87	27	156	⭐🧠 81.80	🧊 6.10	♻ 68	LLaMA-2 Chat 7B Q5_1
88	122	44	🤔 61.46	👌 17.13	❄ 139	Nous Yarn 128K 7B Q5_K_M (ext. context maybe broken)
89	78	97	📖 72.87	🌶 23.16	89	Luna AI 7B Q8_0
90	50	131	📖 79.03	👌 16.06	89	ELYZA Jp LLaMA-2 7B Q5_K_M
91	64	115	📖 76.90	👌 17.85	93	Medusa 1.3 7B Q5_K_M
92	96	78	🤔 68.32	👌 17.45	✳ 118	LLaMA 7B Q8_0
93	56	126	📖 78.11	🧊 11.82	105	Tulpar Limarp 7B Q5_K_M
94	97	77	🤔 68.03	🌶 19.27	✳ 114	Pygmalion Vicuna 7B Q5_K_M
95	130	39	🤪 60.08	👌 17.07	❄ 142	OpenLLaMA v2 7B Q5_K_M
96	61	125	📖 77.19	👌 17.68	♻ 82	ELYZA Jp LLaMA-2 Instruct 7B Q5_K_M
97	72	112	📖 74.83	👌 15.80	105	StableBeluga 7B Q5_K_M
98	39	154	🧠 80.13	🧊 6.42	♻ 70	Photolens LLaMA 2 Langchain Chat 7B Q5_1
99	36	159	🧠 80.36	🧊 5.14	♻ 65	LLaMA-2 Chat Code Cherry Pop 7B Q5_K_M
100	162	🍆 8	🤪 52.07	🌶🌶 23.99	❄ 148	🍆 OpenLLaMA Open Instruct v2 7B Q8_0
101	126	52	🤔 60.66	🌶🌶 25.23	✳ 110	Airoboros GPT4 1.4 7B Q5_K_M
102	132	45	🤪 59.91	🌶 20.93	❄ 123	CodeLLaMA 7B Q5_K_M
103	151	🍆 23	🤪 53.40	🌶🌶 23.31	❄ 131	🍆 Puma 3B Q5_1
104	139	38	🤪 57.32	👌 17.20	❄ 146	AlpacaCielo 2 8K 7B Q5_K_M (ext. context maybe broken)
105	100	85	🤔 67.22	👌 15.73	✳ 120	Nous Yarn 64K 7B Q5_K_M
106	144	🍆 34	🤪 54.26	🌶 19.70	❄ 132	🍆 Deacon 3B Q5_0
107	75	122	📖 74.31	👌 16.95	96	GOAT Community 7B Q5_1
108	107	84	🤔 65.21	🌶🌶 27.30	♻ 86	Lunaboros 7B Q4_K_M
109	58	144	📖 77.48	🧊 11.91	♻ 80	LLaMA-2 32K 7B Q5_K_M (ext. context maybe broken)
110	106	87	🤔 65.21	🌶🌶 26.56	♻ 88	Lunaboros LimaRP 7B Q4_K_M
111	143	43	🤪 54.78	🌶 21.74	✳ 121	OpenLLaMA 7B Q5_K_M
112	98	99	🤔 67.80	🌶🌶 27.62	♻ 66	Airoboros GPT4 2.0 7B Q5_K_M
113	71	133	📖 75.52	👌 15.09	93	Tulpar v0 7B Q4_0
114	62	145	📖 77.13	🧊 10.80	♻ 84	Tsukasa 7B Q5_K_M
115	108	91	🤔 64.86	🌶🌶 24.63	♻ 88	Chinese Alpaca 2 7B Q5_K_S
116	51	160	📖 79.03	🧊 4.15	♻ 60	MedLLaMA-2 Chat 7B Q5_K_S
117	76	132	📖 73.56	👌 15.32	92	Guanaco Uncensored 7B Q5_K_M
118	92	113	🤔 69.12	👌 16.32	104	Metharme 7B Q5_1
119	53	161	📖 78.74	🧊 5.81	♻ 46	Trurl 2 Polish 7B Q5_1
120	65	152	📖 76.56	🧊 6.02	♻ 76	Merak v2 7B Q5_K_M
121	153	47	🤪 53.17	🌶 18.76	❄ 129	Mamba GPT v4 3B Q5_1
122	133	72	🤪 59.10	👌 16.39	❄ 123	Hermes LLongMA 2 8K 7B Q5_1 (ext. context maybe broken)
123	136	69	🤪 58.41	🌶 18.47	✳ 119	Leo Hessianai Chat 7B Q5_K_M
124	70	150	📖 75.52	🧊 9.05	♻ 72	Vicuna v1.5 16K 7B Q5_K_M (ext. context maybe broken)
125	112	101	🤔 63.19	🌶🌶 25.41	♻ 69	Airoboros GPT4 m2.0 7B Q5_K_M
126	119	94	🤔 62.21	🌶🌶 24.39	♻ 85	Airoboros GPT4 m2.0 LLaMA-2 7B Q5_K_M
127	135	76	🤪 58.76	🌶🌶 26.48	93	WizardLM Uncensored 7B Q5_K_M
128	104	114	🤔 65.44	👌 15.84	104	ALMA Pretrain 7B Q5_K_M
129	85	141	📖 71.95	👌 14.51	♻ 75	Chinese LLaMA-2 7B Q5_K
130	131	92	🤪 60.02	👌 17.83	✳ 109	Vicuna CoT 7B Q5_K_M
131	74	162	📖 74.31	🧊 4.48	♻ 49	LLaMA-2 Silverlin. Verilog 7B Q4_K_M
132	95	137	🤔 68.49	🧊 13.02	94	LLaMA-2 Galleon 7B Q5_K_M
133	147	75	🤪 53.69	👌 14.68	❄ 128	Marx V2 3B Q4_1
134	90	147	🤔 71.54	🧊 8.45	♻ 78	StableBeluga Samantha V3 7B Q4_0
135	155	71	🤪 53.00	🌶 19.39	✳ 116	OpenLLaMA 3B Q5_1
136	149	81	🤪 53.57	🧊 11.93	❄ 130	OpenLLaMA v2 3B Q5_0
137	125	111	🤔 60.89	🌶 22.61	♻ 72	MAmmoTH 7B Q5_K_M
138	165	64	🤪 51.15	👌 17.84	✳ 121	OpenBuddy OpenLLaMA v10 3B Q5_0
139	120	118	🤔 61.69	👌 17.11	97	Tulu 7B Q5_K_M
140	113	127	🤔 62.96	👌 13.98	98	WizardCoder Python V1.0 7B Q5_K_M
141	150	89	🤪 53.46	👌 17.04	✳ 117	Griffin 3B (link broken) Q4_1
142	124	124	🤔 61.12	🧊 11.78	107	CodeLLaMA Instruct 7B Q5_K_M
143	140	105	🤪 57.09	👌 17.44	103	CodeLLaMA Python 7B Q5_K_M
144	129	120	🤪 60.25	🧊 13.56	108	Gorilla 7B Q5_K_M
145	160	86	🤪 52.30	🌶 21.09	106	WizardVicuna Uncens Instr PL 3B Q5_1
146	134	121	🤪 58.99	🧊 11.40	✳ 110	LLaMA-2 KO Chat 7B Q5_1
147	114	149	🤔 62.79	🧊 6.70	♻ 74	Pandalyst V1.1 7B Q5_K_M
148	142	119	🤪 56.22	🧊 12.27	✳ 110	Mamba GPT v2 3B Q5_1
149	116	153	🤔 62.44	🧊 6.43	♻ 72	LLaMA-2 KO 7B Q5_1
150	157	106	🤪 52.94	👌 18.05	98	Open LLaMA 7B Q5_1
151	115	158	🤔 62.67	🧊 5.93	♻ 65	Based 7B Q5_K_M
152	137	135	🤪 58.12	🧊 10.29	103	PMC LLaMA 7B Q4_0
153	161	108	🤪 52.19	🧊 11.77	✳ 115	Alpachino Baichuan Instruction 7B Q5_0
154	138	140	🤪 57.66	🧊 13.01	♻ 85	LMSYS Vicuna 1.5 7B Q5_1
155	141	138	🤪 56.85	👌 15.00	♻ 88	Vicuna v1.5 7B Q5_K_M
156	128	157	🤔 60.43	🧊 6.49	♻ 58	Dolphin LLaMA-2 7B Q5_K_M
157	123	163	🤔 61.18	🧊 2.81	♻ 52	Scarlett 7B Q5_K_M
158	166	116	🤪 50.81	🧊 10.78	✳ 114	Baichuan 7B Q5_1
159	127	165	🤔 60.60	🧊 3.90	♻ 45	Tulu Uncensored TV Alpaca 7B (link broken) Q5_1
160	152	143	🤪 53.34	👌 13.97	♻ 75	Orca Mini 3B Q5_1
161	154	142	🤪 53.11	🧊 13.15	♻ 78	Komt LLaMA-2 Chat 7B Q5_K_M
162	156	146	🤪 52.94	🧊 8.64	♻ 80	OpenLLaMA Odia 3B Q5_1
163	164	139	🤪 51.50	🧊 12.99	89	LLaMA Deus v3 7B Q4_0
164	158	148	🤪 52.88	🧊 8.69	♻ 73	Open Cabrita 3B Q5_1
165	163	164	🤪 51.50	🧊 1.97	♻ 47	WizardLM 7B Q5_K_M
166	159	170	🤪 52.42	🧊 0.00	♻ 0	LLongMA 2 7B Q5_1 (ext. context maybe broken)
167	167	167	🤪 47.58	🧊 1.14	♻ 10	TinyLLaMA Chat v0.2 1B Q5_K_M
168	168	168	🤪 47.58	🧊 0.00	♻ 0	PY007 TinyLLaMA Chat v0.2 1B Q8_0
169	171	166	🤪 42.28	🧊 1.64	♻ 16	ToolLLaMA 7B Q5_1
170	169	169	🤪 47.58	🧊 0.00	♻ 0	LongChat v1.5 32K 7B Q5_K_M (ext. context maybe broken)
171	170	171	🤪 47.58	🧊 0.00	♻ 0	LMSYS LongChat 1.5 32k 7B Q5_1 (ext. context maybe broken)

13B Models

Rank	ALC-IQ Rank	ERP Rank	ALC-IQ	ERP Score	ERP Var Score	Model
🥇 1	🎓 2	🍆 13	⭐🧠 93.20	🌶🌶 26.59	❄ 147	🥇🎓🍆 MLewdBoros LRPSGPT 2Char 13B Q5_K_M
🥇 2	🎓 1	🍆 20	⭐🧠 93.43	🌶🌶 27.08	❄ 140	🥇🎓🍆 Athena v1 13B Q5_K_M
🥇 3	🎓 16	🍆 5	⭐🧠 91.88	🌶🌶 27.82	❄ 149	🥇🎓🍆 MLewdBoros 13B Q5_K_M
🥇 4	🎓 5	🍆 27	⭐🧠 92.97	🌶🌶 26.10	❄ 141	🥇🎓🍆 Airoboros 2.1 13B Q5_K_M
🥇 5	🎓 20	🍆 17	⭐🧠 91.36	🌶🌶 29.75	❄ 136	🥇🎓🍆 Pygmalion 2 SuperCOT 13B Q5_K_M
🥇 6	🎓 14	🍆 32	⭐🧠 92.22	🌶 25.59	❄ 145	🥇🎓🍆 ReMM Mistral 13B Q5_K_M
🥇 7	38	🍆 4	⭐🧠 89.98	🌶🌶 28.91	❄ 145	🥇🍆 Slerpeno 13B Q5_K_M
🥇 8	🎓 12	🍆 39	⭐🧠 92.51	🌶🌶 26.47	❄ 133	🥇🎓🍆 Amethyst 13B Q5_K_M
🥇 9	🎓 22	🍆 31	⭐🧠 91.07	🌶🌶 28.20	❄ 133	🥇🎓🍆 ReMM v2 13B Q5_K_M
🥇 10	🎓 13	🍆 43	⭐🧠 92.51	🌶🌶 26.96	✳ 129	🥇🎓🍆 Amethyst Mistral 13B Q4_K_S
🥇 11	🎓 4	55	⭐🧠 93.03	🌶 24.94	❄ 136	🥇🎓 MythoMix 13B Q5_K_M
🥇 12	32	🍆 25	⭐🧠 90.26	🌶🌶 29.09	❄ 134	🥇🍆 AppleSauce 13B Q5_K_M
🥇 13	🎓 18	🍆 46	⭐🧠 91.53	🌶🌶 26.83	✳ 127	🥇🎓🍆 MythoMakiseMerged 13B Q5_K_M
🥇 14	45	🍆 16	⭐🧠 89.52	🌶🌶 26.95	❄ 144	🥇🍆 MLewd V2-1 015 13B Q4_K_S
🥇 15	40	🍆 22	⭐🧠 89.92	🌶🌶 25.69	❄ 156	🥇🍆 Spicyboros 2.2_2 13B Q5_K_M
🥇 16	31	🍆 33	⭐🧠 90.32	🌶🌶 26.75	❄ 136	🥇🍆 Airoboros Creative lmoe 13B Q5_K_M
🥇 17	47	🍆 21	🧠 89.34	🌶🌶 27.02	❄ 139	🥇🍆 Athena v2 13B Q5_K_M
🥇 18	29	🍆 44	⭐🧠 90.38	🌶🌶 28.19	✳ 126	🥇🍆 ReMM v2.2 13B Q5_K_M
🥇 19	🎓 27	🍆 48	⭐🧠 90.44	🌶🌶 26.31	✳ 131	🥇🎓🍆 OpenRP 13B Q5_K_M
🥇 20	28	🍆 47	⭐🧠 90.44	🌶 23.88	❄ 145	🥇🍆 Redmond Puffin 13B Q5_1
🥇 21	65	🍆 6	🧠 88.65	🌶🌶 28.26	❄ 147	🥇🍆 MLewd v2-2 13B Q5_K_M
🥇 22	🎓 17	66	⭐🧠 91.65	🌶🌶 28.64	✳ 119	🥇🎓 ReMM 0.65 SLERP 13B Q5_K_M
🥇 23	🎓 24	58	⭐🧠 90.90	🌶🌶 27.80	✳ 124	🥇🎓 ReMM v2.1 13B Q5_K_M
🥇 24	🎓 6	80	⭐🧠 92.86	🌶🌶 26.11	✳ 122	🥇🎓 MythoMax Kimiko V2 13B Q5_K_M
🥇 25	33	🍆 52	⭐🧠 90.21	🌶🌶 27.40	✳ 125	🥇🍆 MLewdBoros SuperCOT 13B Q5_K_M
🥇 26	39	🍆 53	⭐🧠 89.92	🌶🌶 29.90	✳ 121	🥇🍆 BerrySauce 13B Q5_K_M
🥇 27	34	62	⭐🧠 90.21	🌶 23.34	❄ 139	🥇 Stheno 1.3 13B Q5_K_M
🥈 28	89	🍆 2	📖 87.33	🌶🌶 29.01	❄ 151	🥈🍆 MLewd V2-1 13B Q5_K_M
🥈 29	46	56	⭐🧠 89.46	🌶 25.36	❄ 134	🥈 MLewd Chat 13B Q5_K_M
🥈 30	86	🍆 10	📖 87.38	🌶🌶 26.75	❄ 147	🥈🍆 Unholy v1 10L 13B Q5_K_M
🥈 31	30	79	⭐🧠 90.38	🌶 23.05	❄ 136	🥈 Magpie 13B Q5_K_M
🥈 32	55	🍆 50	🧠 88.94	🌶🌶 26.38	✳ 129	🥈🍆 Pygmaltion 2 SuperCOT weighted 13B Q5_K_M
🥈 33	87	🍆 14	📖 87.38	🌶🌶 26.75	❄ 147	🥈🍆 Unholy v1 13B Q5_K_M
🥈 34	93	🍆 11	📖 87.10	🌶🌶 26.69	❄ 147	🥈🍆 Unholy v1 12L 13B Q5_K_M
🥈 35	🎓 19	102	⭐🧠 91.36	🌶 25.64	✳ 118	🥈🎓 ReMM v2 Kimiko v2 13B Q5_K_M
🥈 36	72	🍆 40	📖 88.25	🌶 23.56	❄ 157	🥈🍆 ZettaPi 13B Q5_K_M
🥈 37	42	76	⭐🧠 89.86	🌶 24.61	✳ 131	🥈 UndiMix v3 13B Q5_K_M
🥈 38	54	67	🧠 89.00	👌 22.64	❄ 142	🥈 Airoboros L2 2.2.1 13B Q5_K_M
🥈 39	64	57	🧠 88.65	🌶 25.53	❄ 133	🥈 Teknium OpenHermes 13B Q5_K_S
🥈 40	50	77	🧠 89.23	🌶🌶 26.10	✳ 123	🥈 ReMM v2 Variant 13B Q5_K_M
🥈 41	52	78	🧠 89.06	🌶 22.80	❄ 137	🥈 Airoboros 2.2 13B Q5_K_M
🥈 42	96	🍆 26	📖 86.87	🌶🌶 26.75	❄ 138	🥈🍆 ReMM 13B Q5_K_M
🥈 43	98	🍆 24	📖 86.69	🌶 25.45	❄ 157	🥈🍆 MLewd V2-1 050 13B Q4_K_S
🥈 44	35	100	⭐🧠 90.21	🌶 25.09	✳ 121	🥈 Chronos Beluga 13B Q5_K_M
🥈 45	113	🍆 8	📖 86.00	🌶🌶 26.33	❄ 163	🥈🍆 Stheno Inverted 1.2 13B Q5_K_M
🥈 46	101	🍆 23	📖 86.64	🌶🌶 27.03	❄ 138	🥈🍆 MLewd v2 13B Q5_K_M
🥈 47	🎓 3	141	⭐🧠 93.20	🌶 25.64	109	🥈🎓 MythoMaxKurisu 13B Q5_K_M
🥈 48	60	75	🧠 88.71	🌶 23.25	❄ 136	🥈 Spicyboros 2.2 13B Q4_K_M
🥈 49	58	82	🧠 88.82	🌶🌶 26.30	✳ 120	🥈 Chronolima Airo Grad 13B Q5_K_M
🥈 50	56	89	🧠 88.94	🌶 24.99	✳ 124	🥈 UndiMix v4 13B Q5_K_M
🥈 51	🎓 9	146	⭐🧠 92.57	🌶 24.20	111	🥈🎓 Huginn v1.2 13B Q5_K_M
🥈 52	🎓 15	142	⭐🧠 92.17	👌 18.04	✳ 129	🥈🎓 Huginn 13B Q5_K_M
🥈 53	🎓 10	148	⭐🧠 92.57	🌶 24.20	111	🥈🎓 ReMM SLERP 13B Q5_K_M
🥈 54	124	🍆 12	📖 85.54	🌶🌶 26.24	❄ 156	🥈🍆 Holomax 13B Q5_K_M
🥈 55	104	🍆 38	📖 86.58	🌶 25.55	❄ 139	🥈🍆 ReMM Lion 13B Q5_K_M
🥈 56	94	🍆 51	📖 87.04	🌶 24.89	❄ 137	🥈🍆 LLaMA-2 Chat Uncensored 13B Q5_1
🥈 57	🎓 11	151	⭐🧠 92.57	🌶 24.20	111	🥈🎓 MythoMax 13B Q5_K_M
🥈 58	102	🍆 45	📖 86.64	🌶 25.29	❄ 138	🥈🍆 Chronos Hermes 2 13B Q5_K_M
🥈 59	71	85	📖 88.31	🌶🌶 29.46	113	🥈 Blind Test Janus 13B Q5_1
🥈 60	🎓 23	143	⭐🧠 91.01	👌 22.52	✳ 119	🥈🎓 Emerhyst 13B Q5_K_M
🥈 61	81	74	📖 87.56	🌶🌶 28.50	✳ 117	🥈 Pygmalion 2 SuperCOT2 13B Q5_K_M
🥈 62	36	129	⭐🧠 90.09	🌶🌶 27.53	101	🥈 OpenRP SuperCOT 13B Q5_K_M
🥈 63	57	108	🧠 88.94	👌 22.06	❄ 133	🥈 Orca Mini v3 13B Q5_K_M
🥈 64	146	🍆 7	🤔 84.22	🌶🌶 29.48	❄ 140	🥈🍆 OpenAssistant LLaMA-2 8k Orca 13B Q5_K_M (ext. context maybe broken)
🥈 65	🎓 7	179	⭐🧠 92.86	🌶 23.29	105	🥈🎓 MythoMax Kimiko Mix 13B Q5_K_M
🥈 66	49	131	🧠 89.29	🌶 24.63	114	🥈 Airolima Chronos Grad 13B Q5_K_M
🥈 67	135	🍆 28	📖 84.85	🌶🌶 26.33	❄ 139	🥈🍆 qCammel L2 13B Q5_K_M
🥉 68	151	🍆 9	🤔 83.76	🌶🌶 27.86	❄ 142	🥉🍆 Athena v3 13B Q5_K_M
🥉 69	62	117	🧠 88.65	👌 17.56	❄ 136	🥉 Stheno Chat 13B Q5_K_M
🥉 70	59	122	🧠 88.71	🌶🌶 26.05	111	🥉 Unholy v1.1 13B Q5_K_M
🥉 71	48	136	🧠 89.34	👌 21.03	✳ 125	🥉 StableBeluga 13B Q5_K_M
🥉 72	109	69	📖 86.12	🌶🌶 25.86	✳ 126	🥉 Airoboros GPT4 1.4.1 13B Q5_K_M
🥉 73	91	92	📖 87.15	🌶 24.94	✳ 124	🥉 Mistral PetroLimaRP v3 12B Q5_K_M
🥉 74	168	🍆 1	🤔 80.82	🌶🌶 28.11	❄ 164	🥉🍆 Legerdemain 13B Q5_K_M
🥉 75	83	106	📖 87.44	👌 21.62	❄ 134	🥉 Pygmalion 2 13B Q5_K_M
🥉 76	🎓 25	176	⭐🧠 90.67	🧊 14.06	✳ 125	🥉🎓 Inkbot 4k 13B Q4_K_M
🥉 77	144	🍆 36	🤔 84.27	🌶🌶 25.85	❄ 138	🥉🍆 Stheno Inverted 13B Q5_K_M
🥉 78	122	63	📖 85.60	🌶 25.19	✳ 131	🥉 MegaMix S1 13B Q5_K_M
🥉 79	130	🍆 54	📖 85.14	🌶🌶 25.70	❄ 133	🥉🍆 ReMM PIPPA 13B Q5_K_M
🥉 80	125	60	📖 85.48	🌶🌶 26.64	✳ 125	🥉 ReMM v1 LRPSGPT 2Char 13B Q5_K_M
🥉 81	🎓 21	185	⭐🧠 91.07	👌 16.73	✳ 117	🥉🎓 LlongOrca 16K 13B Q5_K_M (ext. context maybe broken)
🥉 82	66	132	🧠 88.65	👌 22.06	✳ 124	🥉 Kimiko V2 13B Q5_K_M
🥉 83	153	🍆 29	🤔 83.24	🌶🌶 28.87	❄ 133	🥉🍆 ReMM S Kimiko v2 13B Q5_K_M
🥉 84	79	118	📖 87.90	👌 22.34	✳ 126	🥉 Kimiko 13B Q5_K_M
🥉 85	120	72	📖 85.77	🌶 24.29	❄ 132	🥉 GradientPutri MegaMix S1 13B Q5_K_S
🥉 86	51	155	🧠 89.23	🌶 23.27	114	🥉 Vigogne 2 13B Q5_K_M
🥉 87	76	126	📖 88.02	🌶🌶 30.14	♻ 94	🥉 Airochronos 13B Q5_K_M
🥉 88	171	🍆 15	🤔 80.36	🌶🌶 26.06	❄ 157	🥉🍆 Huginn v3 13B Q5_K_M
🥉 89	103	99	📖 86.58	🌶 24.99	✳ 122	🥉 Saiga 2 13B Q5_K
🥉 90	74	134	📖 88.13	🌶 25.43	111	🥉 MythoLogic 13B Q5_K_M
🥉 91	172	🍆 18	🤔 80.36	🌶🌶 26.06	❄ 157	🥉🍆 Huginn v4 13B Q5_K_M
🥉 92	119	83	📖 85.83	🌶 25.14	✳ 125	🥉 Mythalion 13B Q5_K_M
🥉 93	173	🍆 19	🤔 80.36	🌶🌶 26.06	❄ 157	🥉🍆 Huginn v4.5 13B Q5_K_M
🥉 94	44	175	⭐🧠 89.57	👌 19.85	✳ 118	🥉 Redmond Puffin v1.3 13B Q5_K_M
95	189	🍆 3	🤔 76.96	🌶🌶 29.33	❄ 146	🍆 Airoboros 2.1 YaRN 64K 13B Q5_K_M
96	134	71	📖 84.85	🌶 23.97	❄ 135	Guanaco Uncensored 13B Q5_K_M
97	115	95	📖 86.00	🌶 23.87	✳ 126	Firefly v1.2 13B Q5_K_M
98	121	90	📖 85.71	🌶🌶 26.04	✳ 120	Fireflx v1.2 13B Q5_K_M
99	82	137	📖 87.56	👌 21.44	✳ 124	Chronos Hermes v2 13B Q5_K_M
100	169	🍆 34	🤔 80.70	🌶🌶 25.78	❄ 142	🍆 MLewd v1 13B Q5_K_M
101	141	68	🤔 84.62	🌶 25.33	✳ 131	Camel Platypus 2 13B Q5_K_M
102	63	163	🧠 88.65	👌 22.40	115	MXLewdMini 13B Q5_K_M
103	77	149	📖 88.02	🌶🌶 31.72	♻ 78	Airoboros GPT4 2.0 13B Q5_K_M
104	99	123	📖 86.69	👌 21.37	✳ 127	h2oGPT 13B (link broken) Q5_K_M
105	163	🍆 49	🤔 81.51	🌶 23.18	❄ 150	🍆 Huginn v1.3 13B Q5_K_M
106	177	🍆 35	🤔 79.78	🌶🌶 26.07	❄ 136	🍆 MegaMix T1 13B Q5_K_M
107	69	165	🧠 88.42	👌 20.60	✳ 119	Stheno 1.8 13B Q5_K_M
108	176	🍆 37	🤔 79.84	🌶🌶 26.15	❄ 136	🍆 MLewd v1-7 TRY2 13B Q5_K_M
109	67	170	🧠 88.59	👌 18.80	✳ 120	Stable Platypus 2 13B Q5_K_M
110	108	121	📖 86.41	👌 19.82	❄ 132	Chronos 2 13B Q5_K_M
111	184	🍆 30	🤔 78.00	🌶🌶 28.05	❄ 135	🍆 AlpacaCielo 13B Q5_K_M
112	175	🍆 41	🤔 79.95	🌶 24.02	❄ 150	🍆 LLongMA-2 Storysummarizer 13B Q5_K_M (ext. context maybe broken)
113	75	162	📖 88.13	🌶 24.39	108	Chronoboros Grad 13B Q5_K_M
114	80	157	📖 87.62	🌶🌶 31.26	♻ 73	Airoboros GPT4 2.0 LLaMA-2 13B Q5_K_M
115	68	172	🧠 88.48	👌 18.52	✳ 120	UndiMix v2 13B Q5_K_M
116	145	81	🤔 84.22	🌶 23.08	❄ 136	Platypus 2 13B Q5_K_M
117	🎓 8	246	⭐🧠 92.80	🧊 12.13	♻ 81	🎓 LLaMA-2 Ensemble v6 13B Q5_K_M
118	110	124	📖 86.12	🌶🌶 32.04	♻ 92	Thorns 13B Q5_K_M
119	129	104	📖 85.20	🌶 22.98	✳ 129	StableBeluga Instruct PL Lora 13B Q5_1
120	170	65	🤔 80.53	🌶 23.01	❄ 141	Gywy Chinese v1 13B Q5_1
121	133	111	📖 84.97	🌶🌶 26.09	114	Hermes Kimiko 13B Q5_K_M
122	111	138	📖 86.06	👌 22.40	✳ 121	Chronohermes Grad 13B Q5_K_M
123	178	59	🤔 79.26	🌶 25.63	❄ 132	MLewd 13B Q5_K_M
124	70	192	📖 88.31	🧊 13.44	✳ 118	LLaMA-2 Chat AYT 13B Q5_K_M
125	157	93	🤔 82.95	🌶🌶 29.00	112	Crestfall FrankenMon 13B Q5_K_M
126	164	86	🤔 81.22	🌶 24.78	✳ 125	MegaMix A1 13B Q5_K_M
127	53	222	🧠 89.06	🧊 14.21	102	TerraMix 16K 13B Q5_K_M (ext. context maybe broken)
128	181	73	🤔 78.69	🌶🌶 28.48	✳ 117	Frank Uncensored 13B Q5_K_M
129	43	243	⭐🧠 89.69	🧊 11.99	♻ 89	WizardLM 1.2 PL 13B Q5_1
130	155	110	🤔 83.06	🌶 23.71	✳ 124	Frankensteins Monster 13B Q4_K_S
131	🎓 26	265	⭐🧠 90.61	🧊 5.88	♻ 70	🎓 PuddleJumper 13B Q5_K_M
132	167	96	🤔 80.93	🌶🌶 26.54	116	OniiChat Hermes Limarp 13B Q5_K_M
133	97	181	📖 86.75	👌 19.23	115	LLaMA-2 Mistral 13B Q5_K_M
134	37	253	⭐🧠 90.09	🧊 8.02	♻ 78	WizardLM v1.2 13B Q4_0
135	126	147	📖 85.37	🌶 23.61	113	Nous Hermes 13B Q5_K_M
136	78	208	📖 87.90	🧊 15.16	109	UndiMix v1 13B Q5_K_M
137	128	150	📖 85.20	👌 22.53	✳ 117	Nous Hermes LLaMA-2 13B Q5_K_M
138	174	98	🤔 80.18	🌶 24.93	✳ 123	Stheno 13B Q5_K_M
139	149	128	🤔 83.93	🌶 22.69	✳ 122	LLaMA-2 Guanaco 13B Q4_1
140	132	156	📖 85.02	👌 19.86	✳ 122	EverythingLM V3 16K 13B Q5_K_M (ext. context maybe broken)
141	41	267	⭐🧠 89.92	🧊 4.50	♻ 60	Speechless LLaMA-2 13B Q5_K_M
142	88	211	📖 87.33	🧊 16.72	105	UltraLM v2.0 13B Q5_K_M
143	84	216	📖 87.44	🧊 13.63	106	Spring Dragon 13B Q5_K_M
144	216	61	🤪 70.97	👌 22.40	❄ 153	Nous Yarn 128K 13B Q5_K_M (ext. context maybe broken)
145	73	234	📖 88.25	🧊 12.34	♻ 94	LLaMA-2 LoRA Assemble 13B Q5_K_M
146	199	84	🤔 74.83	👌 19.55	❄ 148	Dans RetroRodeo 13B Q5_K_M
147	197	88	🤔 75.75	🌶 24.38	✳ 126	Nous Hermes Writer 13B Q4_K_S
148	185	105	🤔 77.76	🌶 23.59	✳ 125	WizardMath V1.0 13B Q5_K_M
149	188	103	🤔 77.13	👌 22.62	✳ 131	Nous Yarn 64K 13B Q5_K_M
150	222	64	🤪 68.43	🌶 23.52	❄ 138	Chronos Hermes SuperHOT 8K 13B Q5_1 (ext. context maybe broken)
151	85	230	📖 87.38	🧊 11.67	102	Marcoroni 13B Q5_K_M
152	143	161	🤔 84.33	🌶 24.71	108	Hermes LimaRP 13B Q4_K_M
153	105	207	📖 86.58	👌 16.93	106	Mythical Destroyer V2 13B (link broken) Q5_K_M
154	158	144	🤔 82.72	🌶 23.49	115	Chronorctypus Limarobormes 13B Q5_K_M
155	112	203	📖 86.06	👌 18.32	109	OpenChat v3.2 13B Q5_K_M
156	127	186	📖 85.31	🧊 14.83	✳ 119	OpenOrcaxOpenChat Preview2 13B Q5_1
157	150	160	🤔 83.87	👌 22.34	✳ 117	Synthia 13B Q5_K_M
158	61	269	🧠 88.71	🧊 4.34	♻ 46	Iubaris V3 13B Q5_K_M
159	117	202	📖 85.89	👌 17.65	110	LosslessMegaCoder Mini 13B Q5_K_M
160	95	231	📖 86.92	🧊 12.34	♻ 99	LLaMA-2 Chat Limarp v2 13B Q5_K_M
161	191	116	🤔 76.27	🌶 25.12	✳ 117	Manticore SuperHOT 8K 13B Q5_K_M (ext. context maybe broken)
162	123	198	📖 85.60	🧊 16.16	112	OpenBuddy LLaMA-2 v11.1 13B Q5_K_M
163	183	127	🤔 78.17	🌶🌶 30.71	♻ 93	Airoboros GPT4 m2.0 13B Q5_K_M
164	182	135	🤔 78.51	👌 17.65	❄ 132	Holodeck 1 13B Q5_K
165	154	169	🤔 83.12	👌 21.42	116	ALMA Pretrain 13B Q5_K_M
166	260	🍆 42	🤪 61.52	🌶 24.76	❄ 143	🍆 Hermes LLongMA 2 8K 13B Q5_1 (ext. context maybe broken)
167	100	239	📖 86.64	🧊 12.88	♻ 91	OpenOrca STX 13B Q5_K_M
168	92	251	📖 87.15	🧊 7.45	♻ 86	Samantha 1.11 13B Q5_K_M
169	207	114	🤔 72.12	👌 19.75	❄ 136	Vicuna 1.3 PL 13B Q5_1
170	136	200	📖 84.74	👌 19.43	107	CalliopeDS 13B Q5_K_M
171	210	115	🤪 71.89	👌 19.80	❄ 135	MAmmoTH 13B Q5_K_M
172	90	260	📖 87.33	🧊 7.97	♻ 74	Speechless Hermes Orca Plat WizLM 13B Q5_K_M
173	137	204	📖 84.74	👌 16.80	109	LLaMA-2 Ensemble v5 13B Q5_K_M
174	235	87	🤪 66.24	🌶 24.15	✳ 127	LLaMA SuperCOT 13B Q5_K_M
175	165	173	🤔 81.16	🌶 22.88	111	Stheno 1.2 13B Q5_K_M
176	252	70	🤪 63.82	👌 22.56	❄ 142	Chronos Hermes 13B Q5_K_M
177	195	140	🤔 75.92	🌶🌶 28.25	♻ 93	Airoboros GPT4 m2.0 LLaMA-2 13B Q5_K_M
178	159	184	🤔 82.60	👌 17.85	116	Dans QuestionableCocktail 2 13B Q4_1
179	234	94	🤪 66.42	🌶🌶 28.52	113	Airoboros GPT4 1.3 13B Q5_1
180	131	219	📖 85.08	👌 18.80	♻ 90	Tsukasa Limarp 16K 13B Q5_K_M (ext. context maybe broken)
181	106	249	📖 86.52	🧊 9.47	♻ 83	Mythical Destroyer 13B Q5_K_M
182	107	250	📖 86.46	🧊 8.65	♻ 86	Athena-tmp 13B Q5_K_M
183	139	212	🤔 84.62	👌 16.80	103	OpenOrca Platypus 2 13B Q5_K_M
184	244	91	🤪 64.63	🌶🌶 26.10	✳ 119	MythoBoros 13B Q5_K_M
185	160	195	🤔 82.32	🧊 14.65	114	OpenOrcaxOpenChat 2 LangChain Chat 13B Q5_1
186	118	247	📖 85.83	🧊 11.03	♻ 84	ChatAYT Lora Assamble Marcoroni 13B Q5_K_M
187	161	199	🤔 82.20	👌 18.20	110	Vicuna v1.5 16K 13B Q5_K_M (ext. context maybe broken)
188	180	177	🤔 78.86	👌 21.18	113	YuLan Chat 2 13B Q5_K_M
189	116	254	📖 86.00	🧊 8.08	♻ 78	LLaMA-2 Chinese Chat 13B Q5_1
190	114	257	📖 86.00	🧊 6.74	♻ 78	LLaMA-2 13B Q5_K_M
191	142	225	🤔 84.56	🧊 12.67	103	LLaMA-2 LangChain Chat 13B Q5_K_S
192	156	209	🤔 83.01	👌 17.96	104	Sentdex WSB GPT 13B Q5_K_M
193	202	159	🤔 72.81	🌶 23.57	111	Manticore 13B Q5_K_M
194	242	112	🤪 65.15	🌶 23.25	✳ 125	Wizard Vicuna Uncensored SuperHOT 8k 13B Q5_K_S (ext. context maybe broken)
195	138	237	📖 84.74	🧊 8.75	♻ 98	LLaMA-2 Chat 13B Q5_1
196	248	107	🤪 64.23	👌 22.36	✳ 131	MyhtoLogic 13B Q5_K_M
197	247	109	🤪 64.29	🌶 22.80	✳ 127	Guanaco 13B Q5_K_M
198	254	101	🤪 63.54	👌 20.82	❄ 136	Chronos 13B Q5_K_M
199	179	193	🤔 79.03	🌶 23.06	♻ 94	Dans MythsteryModel 13B Q5_K_M
200	212	154	🤪 71.49	🌶 24.43	110	JanniesBasedLigma 13B Q5_K_M
201	213	153	🤪 71.43	👌 21.15	✳ 121	Tsukasa Limarp 13B Q5_K_M
202	204	164	🤔 72.47	👌 19.03	✳ 122	CodeLLaMA Oasst SFT V10 13B Q5_K_M
203	261	97	🤪 60.77	🌶 23.57	✳ 127	OpenLLaMA 13B Q5_K_M
204	243	119	🤪 64.75	👌 21.20	✳ 129	Chronos WizardLM UC SCOT ST 13B Q5_K_M
205	140	245	🤔 84.62	🧊 8.72	♻ 90	Luban 13B Q5_K_M
206	245	120	🤪 64.63	🌶 23.07	✳ 123	OpenBuddy OpenLLaMA v7 13B Q4_K
207	230	139	🤪 67.17	🌶 23.79	114	WizardLM V1.0 Uncensored 13B Q5_K_M
208	238	133	🤪 65.78	🌶 24.05	115	Chimera 13B Q5_K_M
209	186	197	🤔 77.59	👌 18.79	110	Barcenas 13B Q5_K_M
210	224	152	🤪 68.15	👌 21.75	✳ 120	Chronos SuperHOT 8K 13B Q5_K_M (ext. context maybe broken)
211	152	241	🤔 83.35	🧊 11.90	♻ 92	Trurl 2 Polish 13B Q5_1
212	206	178	🤔 72.29	👌 20.37	114	CAMEL Combined Data 13B Q5_K_M
213	218	167	🤪 69.99	🌶 23.13	111	Minotaur 13B Q5_K_M
214	201	189	🤔 73.10	👌 19.90	110	Tulu 13B Q5_K_M
215	265	113	🤪 57.89	🌶🌶 26.16	113	Petra Instruct 13B Q5_K_M
216	166	232	🤔 80.99	🧊 14.00	♻ 95	Trurl 2 Polish Instruct 13B Q5_1
217	147	256	🤔 84.10	🧊 6.80	♻ 78	Codeup Alpha 13B Q5_K_M
218	253	130	🤪 63.77	👌 22.02	✳ 125	Alpacino SuperCOT 13B Q4_0
219	223	168	🤪 68.20	🌶 24.15	107	Hypermantis 13B Q5_K_M
220	225	166	🤪 68.03	👌 21.14	✳ 119	MedAlpaca 13B Q5_1
221	208	187	🤪 72.00	👌 22.22	107	Heegyu LIMA2 13B Q5_1
222	148	259	🤔 84.10	🧊 6.80	♻ 78	h2oGPT Chat 13B (link broken) Q5_K_M
223	236	158	🤪 66.24	👌 22.27	✳ 118	Dans PersonalityEngine 13B Q5_1
224	264	125	🤪 59.39	👌 22.47	✳ 124	Nous-Hermes 13B Q4_0
225	198	205	🤔 75.06	👌 20.50	♻ 96	Vicuna 1.5 13B Q5_0
226	220	183	🤪 68.61	👌 22.27	109	WizardMega 13B Q5_K_M
227	194	215	🤔 75.92	🧊 16.27	101	Chinese Alpaca 2 13B Q5_K
228	231	171	🤪 66.71	👌 21.16	✳ 117	OpenBuddy LLaMA-2 v8.1 13B Q3_K
229	162	255	🤔 81.68	🧊 6.74	♻ 79	CodeUp LLaMA-2 Chat 13B Q4_K_M
230	232	174	🤪 66.59	🌶 24.15	105	HyperMantis 13B Q5_K_M
231	196	218	🤔 75.75	👌 19.48	♻ 89	WizardLM 1.0 Uncensored 13B Q5_K_M
232	200	214	🤔 74.71	👌 19.62	♻ 94	LLaMA-2 Instruct Uncensored 13B Q5_0
233	258	145	🤪 62.90	🌶 23.79	113	Carl 13B Q5_K_M
234	221	190	🤪 68.49	👌 18.52	113	LLaMA 13B Q5_K_M
235	190	228	🤔 76.61	🧊 14.21	♻ 98	Manticore Chat Pyg 13B Q5_K_M
236	217	196	🤪 70.56	🧊 15.06	113	Chinese LLaMA-2 13B Q5_K
237	192	229	🤔 76.15	🧊 14.96	♻ 94	Manticore Chat Pyg SuperHOT 8K 13B Q5_K_M (ext. context maybe broken)
238	187	236	🤔 77.59	🧊 16.13	♻ 82	Vicuna v1.5 13B Q5_K_M
239	227	194	🤪 67.68	👌 20.98	107	CAMEL Role Playing Data 13B Q5_K_M
240	215	210	🤪 71.08	👌 21.44	♻ 91	BlueMethod 13B Q5_K_M
241	209	220	🤪 72.00	🧊 14.58	101	OpenBuddy Atom v9 13B Q5_K
242	239	188	🤪 65.67	👌 22.30	104	Ouroboros 13B Q5_K_M
243	193	244	🤔 75.98	🧊 12.95	♻ 81	LoKuS 13B Q5_K_M
244	211	226	🤪 71.54	🧊 12.34	103	CodeLLaMA Instruct 13B Q5_K_M
245	251	182	🤪 63.94	👌 21.87	111	Saiga 13B Q5_1
246	203	242	🤔 72.58	🧊 14.85	♻ 78	Metharme 13B Q5_1
247	219	223	🤪 69.53	🧊 14.42	♻ 100	Pandalyst V1.0 13B Q5_K_M
248	255	180	🤪 63.36	🌶 23.35	103	WizardLM Uncensored 13B Q5_K_M
249	228	213	🤪 67.57	👌 17.74	101	WizardLM V1.1 13B Q5_K_M
250	250	191	🤪 63.94	🧊 16.40	114	CodeLLaMA Python 13B Q5_K_M
251	229	217	🤪 67.40	👌 18.80	♻ 91	Asclepius 13B Q5_K_M
252	205	248	🤔 72.41	🧊 11.35	♻ 82	Manticore Chat Pyg Guanaco 13B Q4_K_M
253	214	238	🤪 71.26	🧊 14.02	♻ 91	Vicuna 1.3 German 13B Q5_K_M
254	246	206	🤪 64.46	🧊 14.41	111	CodeLLaMA 13B Q5_K_M
255	237	221	🤪 66.24	🧊 15.54	♻ 96	Vicuna 1.3 13B Q5_1
256	259	201	🤪 62.62	👌 21.82	♻ 96	Wizard Vicuna Uncensored 13B Q5_K_M
257	240	224	🤪 65.38	🧊 15.67	♻ 94	WizardLM 1.0 13B Q5_K_M
258	241	233	🤪 65.21	🧊 16.20	♻ 88	Based 13B Q5_K_M
259	226	252	🤪 67.86	🧊 10.49	♻ 73	Nexus Raven 13B Q5_K_M
260	249	240	🤪 64.00	🧊 14.32	♻ 86	WizardLM WizardCoder Python V1.0 13B Q4_K_S
261	263	227	🤪 59.56	🧊 14.46	♻ 95	Wizard Vicuna 13B Q5_K_M
262	233	263	🤪 66.42	🧊 8.22	♻ 47	Dolphin LLaMA 13B Q5_K_M
263	262	235	🤪 60.37	🧊 14.37	♻ 90	Vicuna CoT 13B Q5_K_M
264	257	266	🤪 63.31	🧊 4.42	♻ 71	Scarlett 13B Q5_K_M
265	256	268	🤪 63.36	🧊 6.06	♻ 38	Pygmalion 13B Q5_1
266	266	258	🤪 57.14	🧊 10.84	♻ 50	Taiwan LLaMA V1.0 13B Q5_K_M
267	268	262	🤪 56.57	🧊 9.46	♻ 44	Taiwan LLaMA v1.0 13B Q5_K_M
268	267	264	🤪 56.91	🧊 7.65	♻ 60	BigTranslate 13B Q4_K_M
269	270	261	🤪 53.46	🧊 8.80	♻ 60	Komt LLaMA-2 13B Q5_K_M
270	269	271	🤪 53.92	🧊 1.27	♻ 11	LMSYS Vicuna 1.5 16k 13B Q5_1 (ext. context maybe broken)
271	274	270	🤪 50.12	🧊 1.79	♻ 45	Stable Vicuna 13B Q5_K_M
272	271	274	🤪 52.42	🧊 0.00	♻ 0	EverythingLM V2 16K 13B Q4_K_S (ext. context maybe broken)
273	275	272	🤪 47.70	🧊 0.62	♻ 7	Chatxu (L2?) 13B Q4_0
274	272	276	🤪 52.42	🧊 0.00	♻ 0	LLongMA 2 13B Q5_1 (ext. context maybe broken)
275	273	275	🤪 50.81	🧊 0.00	♻ 0	EverythingLM 16K 13B Q5_K_M (ext. context maybe broken)
276	276	273	🤪 47.58	🧊 0.00	♻ 0	Dans CreepingSenseOfDoom 13B Q5_K_M

20B to 33B Models

Rank	ALC-IQ Rank	ERP Rank	ALC-IQ	ERP Score	ERP Var Score	Model
🥇 1	🎓 1	🍆 4	⭐🧠 92.74	🌶🌶 30.23	❄ 144	🥇🎓🍆 MLewd ReMM Chat 20B Q5_K_M
🥇 2	🎓 5	🍆 3	⭐🧠 91.53	🌶🌶 29.62	❄ 148	🥇🎓🍆 MLewd ReMM Chat Inverted 20B Q5_K_M
🥇 3	🎓 4	13	⭐🧠 91.65	🌶🌶 27.81	✳ 132	🥇🎓 MXLewd 20B Q5_K_M
🥇 4	8	🍆 10	⭐🧠 90.44	🌶 25.27	❄ 148	🥇🍆 Emerhyst 20B Q5_K_M
🥇 5	🎓 3	19	⭐🧠 92.17	🌶🌶 27.77	✳ 127	🥇🎓 Airoboros 2.1 33B Q4_K_M
🥇 6	18	🍆 6	📖 88.54	🌶🌶 32.89	❄ 136	🥇🍆 MM ReMM 20B Q5_K_M
🥈 7	13	15	🧠 89.57	🌶 24.24	❄ 146	🥈 Huginn 5 Prototype 19B Q4_K_S
🥈 8	9	25	⭐🧠 90.32	🌶 24.50	✳ 134	🥈 Airoboros GPT4 1.4 33B Q4_K_M
🥈 9	21	🍆 11	📖 88.02	🌶🌶 27.55	❄ 141	🥈🍆 Enterredaas 33B Q4_1
🥈 10	16	17	🧠 88.71	🌶🌶 27.40	✳ 132	🥈 Airochronos 33B Q5_K_M
🥈 11	22	14	📖 85.94	🌶 24.43	❄ 146	🥈 LLaMA-2 BlockTri Frankenstein 22B Q4_K_M
🥈 12	24	🍆 12	📖 85.43	🌶 25.92	❄ 142	🥈🍆 Lazarus 30B Q4_K_M
🥈 13	14	26	🧠 89.17	🌶 23.94	✳ 134	🥈 LLaMA SuperCOT 30B Q4_K_M
🥈 14	23	16	📖 85.77	🌶 25.68	❄ 139	🥈 Chronoboros 33B Q5_K_M
🥈 15	🎓 2	44	⭐🧠 92.57	👌 22.62	115	🥈🎓 SuperPlatty 30B Q4_K_M
🥉 16	40	🍆 2	🤔 82.55	🌶🌶 35.79	❄ 153	🥉🍆 COTHuginn 4.5 19B Q5_K_M
🥉 17	7	42	⭐🧠 90.73	👌 22.27	121	🥉 Platypus 2 70B Q2_K
🥉 18	38	🍆 5	🤔 82.83	🌶🌶 27.42	❄ 147	🥉🍆 LLaMA 2 Ari03 28B (link broken) Q5_1
🥉 19	17	33	🧠 88.71	🌶 26.86	117	🥉 Airoboros GPT4 2.0 33B Q5_K_M
🥉 20	12	39	🧠 89.75	👌 22.10	✳ 122	🥉 GPlatty 30B Q4_K_M
🥉 21	29	21	📖 84.62	🌶🌶 28.12	✳ 123	🥉 Saiga 30B Q5_1
22	11	46	⭐🧠 89.92	🌶 25.07	♻ 105	Airoboros GPT4 m2.0 33B Q5_K_M
23	33	20	📖 83.47	🌶 26.87	✳ 130	Fin LLaMA 33B Q4_K_M
24	28	27	📖 84.62	🌶🌶 28.68	115	CAMEL Combined Data 33B Q4_K_M
25	26	31	📖 84.85	🌶🌶 27.18	117	Vigogne Instruct 33B Q4_K_M
26	27	32	📖 84.79	👌 23.34	✳ 128	LLaMA-2 Frankensteined 22B Q4_K_M
27	🎓 6	58	⭐🧠 90.84	🧊 17.17	♻ 96	🎓 Platypus 30B Q4_K_M
28	35	24	🤔 83.12	👌 22.96	❄ 143	Guanaco 33B Q4_K_M
29	10	54	⭐🧠 90.09	🧊 18.98	♻ 106	LLaMA 30B Q5_K_M
30	15	49	🧠 89.00	👌 22.41	114	VicUnlocked LoRA 30B Q4_K_M
31	41	18	🤔 82.55	🌶🌶 31.32	✳ 122	Carl 33B Q4_K_M
32	54	🍆 7	🤪 75.81	🌶 25.84	❄ 156	🍆 Bacchus (L2*) 22B Q4_0
33	60	🍆 1	🤪 73.44	🌶🌶 37.23	❄ 166	🍆 MythoMax 33B Q4_K_M
34	42	23	🤔 82.14	🌶🌶 29.65	119	Frank Uncensored 33B Q5_K_M
35	25	45	📖 85.14	👌 23.68	111	Lazarus Instruct PL 30B Q4_1
36	34	37	🤔 83.35	🌶 26.54	♻ 109	WizardLM Uncensored 30B Q5_K_M
37	47	22	🤔 79.49	👌 22.11	❄ 147	Spicyboros C 2.2 34B Q4_K_M
38	59	🍆 9	🤪 73.79	🌶🌶 29.12	❄ 136	🍆 Wizard Vicuna LLaMA-2 22B Q4_K_M
39	39	36	🤔 82.72	🌶 25.67	116	Vicuna v1.3 33B Q4_K_M
40	19	60	📖 88.48	🧊 9.54	♻ 77	Upstage LLaMA Instruct 30B Q5_K_M
41	45	29	🤔 80.07	👌 22.99	✳ 134	CodeLLaMA 34B Q4_K_M
42	63	🍆 8	🤪 72.47	🌶🌶 27.72	❄ 142	🍆 Daydreamer v3 22B Q5_K_M
43	20	61	📖 88.13	🧊 11.63	♻ 71	Hippogriff 30B Q4_K_M
44	32	47	📖 83.87	👌 23.12	111	Tulu 30B Q5_K_M
45	30	51	📖 84.27	🧊 18.75	112	Dans PersonalityEngine 30B Q4_1
46	50	28	🤪 78.92	👌 21.98	❄ 141	Huginn Prototype 22B Q4_K_M
47	43	38	🤔 81.16	🌶 25.16	114	WizardLM V1.0 Uncensored 33B Q4_K_M
48	31	55	📖 83.99	🧊 18.92	♻ 86	Based 30B Q5_K_M
49	52	30	🤪 78.34	🌶 24.40	✳ 128	Wizard Vicuna Uncensored 30B Q5_K_M
50	49	35	🤔 79.15	👌 21.51	✳ 131	CodeLLaMA Python 34B Q4_K_M
51	44	41	🤔 80.18	👌 20.68	✳ 126	Chronos 33B Q5_K_M
52	36	52	🤔 83.06	🧊 19.81	♻ 106	Epsilon 30B Q4_K_M
53	37	59	🤔 83.06	🧊 8.47	♻ 101	MindFlay 22B Q4_0
54	56	40	🤪 74.48	👌 20.11	✳ 126	Airoboros C 2.1 34B Q5_K_M
55	46	53	🤔 79.55	🧊 18.91	♻ 110	Airoboros C 2.2 34B Q4_K_M
56	62	34	🤪 73.16	🌶 25.26	119	LLaMA 2 DayDreamer V1 22B Q5_K_M
57	57	43	🤪 74.48	👌 20.11	✳ 126	Airoboros C 2.1b 34B Q5_K_M
58	53	50	🤪 76.04	🧊 17.46	121	CodeLLaMA Instruct 34B Q4_K_M
59	48	56	🤔 79.38	🧊 13.38	♻ 103	Synthia v1.2 34B Q4_K_M
60	61	48	🤪 73.21	👌 20.97	117	Phind CodeLLaMA v1 34B Q4_K_S
61	55	57	🤪 74.83	🧊 14.03	♻ 102	Airobors C 2.1 34B Q4_K_M
62	51	62	🤪 78.63	🧊 5.38	♻ 70	Scarlett 33B Q4_K_M
63	58	65	🤪 74.19	🧊 4.61	♻ 48	Samantha 1.11 CodeLLaMA 34B Q4_K_M
64	64	63	🤪 60.08	🧊 5.86	♻ 69	BrainToast 20B Q5_K_M
65	66	64	🤪 51.15	🧊 2.59	♻ 56	WizardLM 30B Q4_K_M
66	65	66	🤪 52.42	🧊 0.00	♻ 0	Airoboros GPT4 1.4 SuperHOT 8K 33B Q4_K_M (ext. context maybe broken)

About Extended Context (8K, 16K, 32K)

As you may have noticed, there are a few models currently (2023-08-09) that have a bad ALC-IQ and even worse ERP-Score. A few of these models are:

LLaMA-2 32K 7B
LMSYS LongChat 1.5 32k 7B
LLongMA 2 7B
Hermes LLongMA 2 8K (L2) 7B

And a few others. The reason for this is simple: The GGML file format is a mess. And even after the new GGUF file format arrived, people sometimes seem to fail to properly quantize the context extended models into a GGUF file. The benchmark does sometimes not have proper results for these models because:

The GGUF file creator messed up somehow (for instance: converted a GGML file to GGUF without the proper rope scaling settings).
For GGML Files:
- A special setting is required in llama.cpp to enable compatibility with these models. Called --rope-freq-base and --rope-freq-scale. These need to be set to the right magic values corresponding to the model at hand.
- Determining these magic ROPE values is not hard, if they were properly documented. But only few pages on huggingfaces that provide GGML file quantizations document these. TheBloke really tries hard, but sometimes even the original model uploaders don't provide any information about the right values.
- And most importantly: It would require carrying meta data out of band along with each file for me. I don't have the time figuring out the right values. And I believe most users won't ever bother either.
- There are also other important options which are not mentioned yet, but are crucial for some GGML files to work properly:
  - --gqa (grouped-query attention factor) is one of these, it is required to set to the magic value 8 for LLaMA 2 70B to work.
  - --rms-norm-eps is an epsilon value for inference of the models. This value is different bewettn LLaMA 1 (1e-6) and LLaMA 2 (1e-5). It makes a difference in how well either model works. The original default 1e-6 was actually replaced recently by 5e-6 which is half way between the both values, and suppsedly should work fine. But in my own tests I saw quite some variance in the performance of the quantized GGML models, which were kind of contradicting to what was stated on llama.cpp. But I decided to not dig further, because there is still too much sampling randomness involved in the ALC-IQ (beta). Which I will eventually fix.

Ranking Changelog

2023-11-01 Benchmark Re-Run V3
I currently run a completely new benchmark. Until I get around to update this page, you may find the most recent results here: http://ayumi.m8geil.de/ayumi_bench_v3_results.html
2023-10-04 V36
- Benchmark Results as CSV - Timestamp 20231004_193917

Size	Rank	Model
3B-7B	1 / 171	🥇🎓🍆(⭐🧠❄🌶🌶) Mistral Claude Chat 7B Q5_K_M
3B-7B	2 / 171	🥇🎓🍆(⭐🧠❄) Mistral ClaudeLimaRP v3 7B Q5_K_M
3B-7B	3 / 171	🥇🎓🍆(⭐🧠❄🌶🌶) Mistral RP 0.1 7B Q5_K_M
3B-7B	4 / 171	🥇🎓🍆(⭐🧠❄🌶🌶) Synthia v1.3 7B Q5_K_M
3B-7B	5 / 171	🥇🎓🍆(⭐🧠❄🌶🌶) Samantha Mistral 7B Q5_K_M
3B-7B	6 / 171	🥇🎓🍆(⭐🧠❄) Mistral v0.1 7B Q5_K_M
3B-7B	8 / 171	🥇🎓🍆(⭐🧠❄🌶🌶) PetrolLM 7B Q5_K_M
3B-7B	13 / 171	🥇🍆(❄🌶🌶) MistRP v1.1 7B Q8_0
3B-7B	17 / 171	🥇🎓(⭐🧠) Kimiko Mistral 7B Q5_K_M
3B-7B	18 / 171	🥈🍆(❄🌶🌶) Mistral Instruct v0.1 7B Q5_K_M
3B-7B	42 / 171	🥈🍆(❄🌶🌶) Samantha Mistral Instruct 7B Q5_K_M
3B-7B	81 / 171	LLaMA-2 Mistral 7B Q5_K_M
3B-7B	91 / 171	Medusa 1.3 7B Q5_K_M
3B-7B	106 / 171	🍆(❄) Deacon 3B Q5_0
3B-7B	123 / 171	Leo Hessianai Chat 7B Q5_K_M
3B-7B	147 / 171	Pandalyst V1.1 7B Q5_K_M
13B	6 / 276	🥇🎓🍆(⭐🧠❄) ReMM Mistral 13B Q5_K_M
13B	8 / 276	🥇🎓🍆(⭐🧠❄🌶🌶) Amethyst 13B Q5_K_M
13B	10 / 276	🥇🎓🍆(⭐🧠🌶🌶) Amethyst Mistral 13B Q4_K_S
13B	13 / 276	🥇🎓🍆(⭐🧠🌶🌶) MythoMakiseMerged 13B Q5_K_M
13B	60 / 276	🥈🎓(⭐🧠) Emerhyst 13B Q5_K_M
13B	68 / 276	🥉🍆(❄🌶🌶) Athena v3 13B Q5_K_M
13B	73 / 276	🥉 Mistral PetroLimaRP v3 12B Q5_K_M
13B	78 / 276	🥉 MegaMix S1 13B Q5_K_M
13B	85 / 276	🥉(❄) GradientPutri MegaMix S1 13B Q5_K_S
13B	106 / 276	🍆(❄🌶🌶) MegaMix T1 13B Q5_K_M
13B	107 / 276	Stheno 1.8 13B Q5_K_M
13B	126 / 276	MegaMix A1 13B Q5_K_M
13B	133 / 276	LLaMA-2 Mistral 13B Q5_K_M
13B	142 / 276	UltraLM v2.0 13B Q5_K_M
13B	199 / 276	Dans MythsteryModel 13B Q5_K_M
13B	247 / 276	Pandalyst V1.0 13B Q5_K_M
13B	259 / 276	Nexus Raven 13B Q5_K_M
20B-34B	4 / 66	🥇🍆(⭐🧠❄) Emerhyst 20B Q5_K_M

2023-09-25 V35
- Benchmark Results as CSV - Timestamp 20230925_091551

Size	Rank	Model
3B-7B	34 / 155	🥈🍆(❄🌶🌶) Wizard Vicuna Uncensored 7B Q5_K_M
3B-7B	36 / 155	🥈🍆(❄🌶🌶) Airoboros GPT4 1.4.1 7B Q5_K_M
3B-7B	42 / 155	🥉🍆(🌶🌶) Frank Uncensored 7B Q5_K_M
3B-7B	49 / 155	🥉🍆(🌶🌶) WizardLM V1.0 Uncensored 7B Q5_K_M
3B-7B	52 / 155	🥉 Airoboros L2 2.2.1 7B Q5_K_M
3B-7B	53 / 155	🍆(❄🌶🌶) Guanaco 7B Q5_K_M
3B-7B	66 / 155	(🌶🌶) Xwin LM V0.1 7B Q5_K_M
3B-7B	112 / 155	ALMA Pretrain 7B Q5_K_M
3B-7B	113 / 155	(🌶🌶) WizardLM Uncensored 7B Q5_K_M
3B-7B	115 / 155	Vicuna CoT 7B Q5_K_M
3B-7B	123 / 155	Tulu 7B Q5_K_M
3B-7B	126 / 155	MAmmoTH 7B Q5_K_M
3B-7B	129 / 155	Gorilla 7B Q5_K_M
3B-7B	134 / 155	Based 7B Q5_K_M
3B-7B	149 / 155	WizardLM 7B Q5_K_M
3B-7B	151 / 155	TinyLLaMA Chat v0.2 1B Q5_K_M
13B	13 / 259	🥇🎓🍆(⭐🧠🌶🌶) ReMM v2.2 13B Q5_K_M
13B	16 / 259	🥇🍆(⭐🧠❄🌶🌶) Athena v2 13B Q5_K_M
13B	33 / 259	🥈🍆(❄) ZettaPi 13B Q5_K_M
13B	34 / 259	🥈(❄) Airoboros L2 2.2.1 13B Q5_K_M
13B	65 / 259	🥉(❄) Stheno Chat 13B Q5_K_M
13B	66 / 259	🥉(🌶🌶) Airoboros GPT4 1.4.1 13B Q5_K_M
13B	68 / 259	🥉🎓(⭐🧠) Inkbot 4k 13B Q4_K_M
13B	93 / 259	MXLewdMini 13B Q5_K_M
13B	115 / 259	(🌶🌶) Frank Uncensored 13B Q5_K_M
13B	127 / 259	EverythingLM V3 16K 13B Q5_K_M
13B	134 / 259	(❄) Dans RetroRodeo 13B Q5_K_M
13B	150 / 259	ALMA Pretrain 13B Q5_K_M
13B	157 / 259	LLaMA SuperCOT 13B Q5_K_M
13B	158 / 259	(❄) MAmmoTH 13B Q5_K_M
13B	163 / 259	(❄) Chronos Hermes 13B Q5_K_M
13B	170 / 259	(🌶🌶) MythoBoros 13B Q5_K_M
13B	179 / 259	Guanaco 13B Q5_K_M
13B	181 / 259	Manticore 13B Q5_K_M
13B	183 / 259	MyhtoLogic 13B Q5_K_M
13B	185 / 259	Chronos WizardLM UC SCOT ST 13B Q5_K_M
13B	186 / 259	(❄) Chronos 13B Q5_K_M
13B	191 / 259	WizardLM V1.0 Uncensored 13B Q5_K_M
13B	193 / 259	Chimera 13B Q5_K_M
13B	197 / 259	CAMEL Combined Data 13B Q5_K_M
13B	198 / 259	Minotaur 13B Q5_K_M
13B	201 / 259	Tulu 13B Q5_K_M
13B	204 / 259	Hypermantis 13B Q5_K_M
13B	212 / 259	WizardMega 13B Q5_K_M
13B	215 / 259	Manticore Chat Pyg 13B Q5_K_M
13B	224 / 259	CAMEL Role Playing Data 13B Q5_K_M
13B	225 / 259	BlueMethod 13B Q5_K_M
13B	227 / 259	Ouroboros 13B Q5_K_M
13B	231 / 259	WizardLM V1.1 13B Q5_K_M
13B	233 / 259	WizardLM Uncensored 13B Q5_K_M
13B	240 / 259	WizardLM 1.0 13B Q5_K_M
13B	241 / 259	Wizard Vicuna Uncensored 13B Q5_K_M
13B	242 / 259	Based 13B Q5_K_M
13B	245 / 259	Wizard Vicuna 13B Q5_K_M
13B	246 / 259	Vicuna CoT 13B Q5_K_M
13B	254 / 259	Stable Vicuna 13B Q5_K_M
20B-34B	1 / 65	🥇🎓🍆(⭐🧠❄🌶🌶) MLewd ReMM Chat 20B Q5_K_M
20B-34B	2 / 65	🥇🎓🍆(⭐🧠❄🌶🌶) MLewd ReMM Chat Inverted 20B Q5_K_M
20B-34B	3 / 65	🥇🎓🍆(⭐🧠🌶🌶) MXLewd 20B Q5_K_M
20B-34B	5 / 65	🥇🍆(❄🌶🌶) MM ReMM 20B Q5_K_M
20B-34B	11 / 65	🥈🍆(❄) Lazarus 30B Q4_K_M
20B-34B	12 / 65	🥈 LLaMA SuperCOT 30B Q4_K_M
20B-34B	14 / 65	🥈🎓(⭐🧠) SuperPlatty 30B Q4_K_M
20B-34B	19 / 65	🥉(⭐🧠) GPlatty 30B Q4_K_M
20B-34B	22 / 65	Fin LLaMA 33B Q4_K_M
20B-34B	24 / 65	(🌶🌶) CAMEL Combined Data 33B Q4_K_M
20B-34B	26 / 65	(❄) Guanaco 33B Q4_K_M
20B-34B	28 / 65	🎓(⭐🧠) Platypus 30B Q4_K_M
20B-34B	29 / 65	VicUnlocked LoRA 30B Q4_K_M
20B-34B	33 / 65	(🌶🌶) Frank Uncensored 33B Q5_K_M
20B-34B	36 / 65	WizardLM Uncensored 30B Q5_K_M
20B-34B	40 / 65	Upstage LLaMA Instruct 30B Q5_K_M
20B-34B	42 / 65	Hippogriff 30B Q4_K_M
20B-34B	43 / 65	Tulu 30B Q5_K_M
20B-34B	46 / 65	WizardLM V1.0 Uncensored 33B Q4_K_M
20B-34B	47 / 65	Based 30B Q5_K_M
20B-34B	48 / 65	Wizard Vicuna Uncensored 30B Q5_K_M
20B-34B	51 / 65	Epsilon 30B Q4_K_M
20B-34B	63 / 65	BrainToast 20B Q5_K_M
20B-34B	64 / 65	WizardLM 30B Q4_K_M

2023-09-18 V34
- Benchmark Results as CSV - Timestamp 20230918_215030

Size	Rank	Model
3B-7B	16 / 143	🥈🎓(⭐🧠🌶🌶) Kuchiki 1.1 7B Q5_K_M
3B-7B	51 / 143	Saiga 2 7B Q5_K
3B-7B	117 / 143	WizardCoder Python V1.0 7B Q5_K_M
3B-7B	140 / 143	PY007 TinyLLaMA Chat v0.2 1B Q8_0
13B	23 / 230	🥇🎓(⭐🧠❄) Magpie 13B Q5_K_M
13B	25 / 230	🥈(⭐🧠❄) MLewd Chat 13B Q5_K_M
13B	26 / 230	🥈🍆(🌶🌶) Pygmaltion 2 SuperCOT weighted 13B Q5_K_M
13B	74 / 230	🥉 Saiga 2 13B Q5_K
13B	140 / 230	OpenOrca STX 13B Q5_K_M
13B	143 / 230	CalliopeDS 13B Q5_K_M
13B	158 / 230	ChatAYT Lora Assamble Marcoroni 13B Q5_K_M
13B	223 / 230	Taiwan LLaMA v1.0 13B Q5_K_M
20B-34B	1 / 47	🥇🎓🍆(⭐🧠❄🌶🌶) MLewd ReMM Chat 20B Q5_K_M
20B-34B	2 / 47	🥇🎓(⭐🧠🌶🌶) MLewd ReMM Chat Inverted 20B Q5_K_M
20B-34B	17 / 47	Vigogne Instruct 33B Q4_K_M
20B-34B	27 / 47	Vicuna v1.3 33B Q4_K_M
20B-34B	37 / 47	Airoboros C 2.2 34B Q4_K_M
20B-34B	42 / 47	Synthia v1.2 34B Q4_K_M

2023-09-15 V33

Size	Rank	Model
3B-7B	1 / 140	🥇🎓🍆(⭐🧠🌶🌶) Kuchiki 7B Q5_K_M
3B-7B	26 / 140	🥈(❄) LLaMA-2 Coder 7B Q5_K_M
3B-7B	53 / 140	LLaMA-2 LoRA Assemble 7B Q5_K_M
3B-7B	134 / 140	OpenLLaMA Odia 3B Q5_1
13B	1 / 225	🥇🎓🍆(⭐🧠❄🌶🌶) MLewdBoros LRPSGPT 2Char 13B Q5_K_M
13B	20 / 225	🥇🍆(⭐🧠🌶🌶) BerrySauce 13B Q5_K_M
13B	47 / 225	🥈(❄) MLewd Chat 13B Q5_K_M
13B	48 / 225	🥈(🌶🌶) Pygmalion 2 SuperCOT2 13B Q5_K_M
13B	62 / 225	🥉(🌶🌶) ReMM v1 LRPSGPT 2Char 13B Q5_K_M
13B	100 / 225	LLaMA-2 Chat AYT 13B Q5_K_M
13B	116 / 225	LLaMA-2 LoRA Assemble 13B Q5_K_M
13B	225 / 225	Dans CreepingSenseOfDoom 13B Q5_K_M
20B-34B	20 / 41	(❄) Spicyboros C 2.2 34B Q4_K_M

2023-09-13 V32
- Benchmark Results as CSV - Timestamp 20230913_162143

Size	Rank	Model
3B-7B	40 / 137	🥉 Airoboros 2.2 7B Q5_K_M
3B-7B	108 / 137	LLaMA-2 Silverlin. Verilog 7B Q4_K_M
13B	12 / 217	🥇🎓🍆(⭐🧠🌶🌶) OpenRP 13B Q5_K_M
13B	18 / 217	🥇🍆(⭐🧠🌶🌶) MLewdBoros SuperCOT 13B Q5_K_M
13B	23 / 217	🥈🎓(⭐🧠) ReMM v2 Kimiko v2 13B Q5_K_M
13B	32 / 217	🥈(❄) Airoboros 2.2 13B Q5_K_M
13B	37 / 217	🥈 UndiMix v4 13B Q5_K_M
13B	47 / 217	🥈(⭐🧠🌶🌶) OpenRP SuperCOT 13B Q5_K_M
13B	50 / 217	🥈(🌶🌶) Unholy v1.1 13B Q5_K_M
20B-34B	26 / 41	Spicyboros C 2.2 34B Q4_K_M

2023-09-12 V31
- Benchmark Results as CSV - Timestamp 20230912_211333

Size	Rank	Model
3B-7B	35 / 135	🥉 Marcoroni 7B Q5_K_M
3B-7B	104 / 135	Chinese LLaMA-2 7B Q5_K
13B	4 / 210	🥇🎓🍆(⭐🧠❄🌶🌶) Pygmalion 2 SuperCOT 13B Q5_K_M
13B	7 / 210	🥇🎓🍆(⭐🧠❄🌶🌶) AppleSauce 13B Q5_K_M
13B	14 / 210	🥇🎓(⭐🧠🌶🌶) ReMM v2.1 13B Q5_K_M
13B	19 / 210	🥇🍆(❄🌶🌶) Unholy v1 10L 13B Q5_K_M
13B	20 / 210	🥇🍆(❄🌶🌶) Unholy v1 13B Q5_K_M
13B	21 / 210	🥇🍆(❄🌶🌶) Unholy v1 12L 13B Q5_K_M
13B	35 / 210	🥈🎓(⭐🧠) Huginn v1.2 13B Q5_K_M
13B	55 / 210	🥉🎓(⭐🧠) LlongOrca 16K 13B Q5_K_M
13B	62 / 210	🥉🍆(❄🌶🌶) Huginn v3 13B Q5_K_M
13B	84 / 210	🎓(⭐🧠) LLaMA-2 Ensemble v6 13B Q5_K_M
13B	105 / 210	Marcoroni 13B Q5_K_M
13B	125 / 210	LLaMA-2 Ensemble v5 13B Q5_K_M
13B	132 / 210	OpenOrca Platypus 2 13B Q5_K_M
13B	154 / 210	JanniesBasedLigma 13B Q5_K_M
13B	155 / 210	Barcenas 13B Q5_K_M
13B	157 / 210	Tsukasa Limarp 13B Q5_K_M
13B	174 / 210	Chinese Alpaca 2 13B Q5_K
13B	183 / 210	Chinese LLaMA-2 13B Q5_K

2023-09-10 V30
- Benchmark Results as CSV - Timestamp 20230910_141602

Size	Rank	Model
3B-7B	20 / 134	🥈 Medusa 1.1 7B Q5_K_M
3B-7B	30 / 134	🥈 LosslessMegaCoder Mini 7B Q5_K_M
3B-7B	37 / 134	🥉🎓(⭐🧠) LLaMA-2 PeanutButter v19 R8 7B Q5_K_M
3B-7B	38 / 134	🥉(⭐🧠) Befenghuang Vigogne 2 Chat 7B Q5_K_S
3B-7B	41 / 134	🥉(❄) Ganchengguang Yoko Japanse v0 7B Q5_K_S
3B-7B	42 / 134	🥉 LlongOrca 16K 7B Q5_K_M
3B-7B	45 / 134	🥉(🌶🌶) Spicyboros 2.2 7B Q5_K_M
3B-7B	62 / 134	(🌶🌶) Airoboros GPT4 2.0 LLaMA-2 7B Q5_K_M
3B-7B	93 / 134	(🌶🌶) Chinese Alpaca 2 7B Q5_K_S
3B-7B	97 / 134	Guanaco Uncensored 7B Q5_K_M
3B-7B	98 / 134	(❄) Mamba GPT v4 3B Q5_1
3B-7B	102 / 134	(🌶🌶) Airoboros GPT4 m2.0 LLaMA-2 7B Q5_K_M
13B	2 / 195	🥇🎓🍆(⭐🧠❄🌶🌶) MLewdBoros 13B Q5_K_M
13B	5 / 195	🥇🍆(⭐🧠❄🌶🌶) Spicyboros 2.2_2 13B Q5_K_M
13B	6 / 195	🥇🎓🍆(⭐🧠❄🌶🌶) ReMM v2 13B Q5_K_M
13B	12 / 195	🥇🍆(❄🌶🌶) MLewd v2-2 13B Q5_K_M
13B	14 / 195	🥇🎓(⭐🧠🌶🌶) ReMM 0.65 SLERP 13B Q5_K_M
13B	15 / 195	🥇🎓(⭐🧠❄) Stheno 1.3 13B Q5_K_M
13B	18 / 195	🥇🍆(❄🌶🌶) Teknium OpenHermes 13B Q5_K_S
13B	19 / 195	🥇(⭐🧠🌶🌶) ReMM v2 Variant 13B Q5_K_M
13B	23 / 195	🥈(⭐🧠❄) Spicyboros 2.2 13B Q4_K_M
13B	24 / 195	🥈🍆(❄🌶🌶) Stheno Inverted 1.2 13B Q5_K_M
13B	30 / 195	🥈🍆(❄🌶🌶) Holomax 13B Q5_K_M
13B	57 / 195	🥉(❄) Guanaco Uncensored 13B Q5_K_M
13B	60 / 195	🥉 Chronos Hermes v2 13B Q5_K_M
13B	64 / 195	🥉🍆(❄🌶🌶) Airoboros 2.1 YaRN 64K 13B Q5_K_M
13B	72 / 195	(🌶🌶) Airoboros GPT4 2.0 LLaMA-2 13B Q5_K_M
13B	91 / 195	Nous Hermes LLaMA-2 13B Q5_K_M
13B	125 / 195	Stheno 1.2 13B Q5_K_M
13B	128 / 195	(🌶🌶) Airoboros GPT4 m2.0 LLaMA-2 13B Q5_K_M
13B	180 / 195	Based 13B Q5_K_M
13B	187 / 195	Taiwan LLaMA v1.0 13B Q5_K_M
20B-34B	9 / 40	🥈🍆(❄🌶🌶) COTHuginn 4.5 19B Q5_K_M
20B-34B	20 / 40	🍆(❄🌶🌶) MythoMax 33B Q4_K_M
20B-34B	28 / 40	Based 30B Q4_K_M

2023-09-08 V29

The ERP Scores (ERP Score and ERP Variety Score) were completely reworked: The count of lewd words is now related to the total number of lewd words in a response. And the ERP Score is now the average of these and not the median anymore. And the ERP Variety Score was added, which tries to catch the erotic creative lewd word knowledge of a model. The ERP Rank is computed by slightly biasing towards the new ERP Variety Score.
Separate ranks for The ALC-IQ and the ERP Scores were introduced. The resulting model rank is now determined by a weighted sum of the ALC-IQ Rank and ERP Rank. Slightly biased towards the ALC-IQ Rank.
GGUF results replace the GGML results of a model now. Please note, that sometimes this can result in a model gaining of loosing ranks in the table. This is sadly just the nature of the foating point quantization. It just shows how similar these models and fine tunes are in the core and how sensitive this benchmark is.
New Symbols were added to signal good ALC-IQ Ranks (🎓) and good ERP Ranks (🍆). The medals (🥇, 🥈 and 🥉) are assigned to multiple ranks, because this ranking can't tell you ultimately which model is actually the best for you. That is not just because there are many known flaws in this benchmark, but also because a large part of your role play experience will also depend on: your expectations, the character card, the prompt annotations and the sampler settings you use.
And the following models were added on top of that to the table:

Benchmark Results as CSV - Timestamp 20230908_203426

Size	Rank	Model
3B-7B	5 / 123	🥇🎓🍆(⭐🧠🌶🌶) Zarablend 7B Q5_K_M
3B-7B	9 / 123	🥇🎓(⭐🧠❄) Zarafusionex 1.1 7B Q5_K_M
3B-7B	10 / 123	🥇🎓(⭐🧠) Hermes LimaRP 7B Q5_K_M
3B-7B	12 / 123	🥇(⭐🧠) Krakowiak 7B Q4_K_M
3B-7B	17 / 123	🥈(🌶🌶) Zarablend MX 7B Q5_K_M
3B-7B	21 / 123	🥈 Typly Pigeon 7B Q4_K_M
3B-7B	46 / 123	Kimiko 7B Q5_K_M
3B-7B	51 / 123	(🌶🌶) Luna AI LLaMA-2 Uncensored 7B Q5_K_M
3B-7B	58 / 123	Pygmalion 2 7B Q5_K_M
3B-7B	71 / 123	StableBeluga 7B Q5_K_M
13B	3 / 177	🥇🎓🍆(⭐🧠❄🌶🌶) Slerpeno 13B Q5_K_M
13B	4 / 177	🥇🍆(⭐🧠❄🌶🌶) MLewd V2-1 015 13B Q4_K_S
13B	10 / 177	🥇🍆(❄🌶🌶) MLewd V2-1 13B Q5_K_M
13B	11 / 177	🥇🎓(⭐🧠❄) UndiMix v3 13B Q5_K_M
13B	13 / 177	🥇🍆(❄🌶🌶) MLewd V2-1 050 13B Q4_K_S
13B	15 / 177	🥇🍆(❄🌶🌶) MLewd v2 13B Q5_K_M
13B	20 / 177	🥈🍆(❄🌶🌶) ReMM Lion 13B Q5_K_M
13B	30 / 177	🥈(⭐🧠) StableBeluga 13B Q5_K_M
13B	32 / 177	🥈(❄) Pygmalion 2 13B Q5_K_M
13B	38 / 177	🥈(🌶🌶) Mythalion 13B Q5_K_M
13B	41 / 177	🥈(🌶🌶) Fireflx v1.2 13B Q5_K_M
13B	45 / 177	🥉🍆(❄🌶🌶) ReMM S Kimiko v2 13B Q5_K_M
13B	60 / 177	(🌶🌶) Thorns 13B Q5_K_M
13B	70 / 177	(⭐🧠) TerraMix 16K 13B Q5_K_M
13B	120 / 177	YuLan Chat 2 13B Q5_K_M
20B-34B	2 / 37	🥇🍆(⭐🧠❄) Huginn 5 Prototype 19B Q4_K_S
20B-34B	28 / 37	Airoboros C 2.1 34B Q5_K_M

2023-09-05 V28
- Changes: Removed the (L2) marker.
- There are still GGML results in my benchmark, I will keep them for now until ggml seems to be phased out completely eventually.
- Marking broken links in the table with "(link broken)"

Size	Rank	IQ/ERP	GGML Model
3B-7B	25 / 125	🧠 / 👌	Tsukasa Limarp 7B (gguf) Q5_K_M
3B-7B	26 / 125	🧠 / 👌	ELYZA Jp LLaMA-2 7B (gguf) Q5_K_M
3B-7B	27 / 125	⭐🧠 / 🧊	MedLLama 7B (gguf) Q5_K_M
3B-7B	28 / 125	⭐🧠 / 🧊	LLaMA-2 7B (gguf) Q5_K_M
3B-7B	54 / 125	⭐📖 / 👌	ELYZA Jp LLaMA-2 Instruct 7B (gguf) Q5_K_M
3B-7B	57 / 125	📖 / 👌	LLaMA-2 Galleon 7B (gguf) Q5_K_M
3B-7B	60 / 125	📖 / 🧊	Tsukasa 7B (gguf) Q5_K_M
3B-7B	62 / 125	📖 / 🧊	Vicuna v1.5 16K 7B (gguf) Q5_K_M
3B-7B	101 / 125	⭐🤪 / 🌶	Vicuna v1.5 7B (gguf) Q5_K_M
13B	2 / 170	⭐🧠 / 🌶🌶	MythoMix 13B (gguf) Q5_K_M
13B	6 / 170	⭐🧠 / 🌶🌶	MythoMax 13B (gguf) Q5_K_M
13B	7 / 170	⭐🧠 / 🌶🌶	ReMM SLERP 13B (gguf) Q5_K_M
13B	14 / 170	🧠 / 🌶🌶	MythoLogic 13B (gguf) Q5_K_M
13B	37 / 170	🧠 / 🧊	WizardLM v1.2 13B (gguf) Q4_0
13B	38 / 170	🧠 / 🧊	Speechless LLaMA-2 13B (gguf) Q5_K_M
13B	42 / 170	🧠 / 🧊	Speechless Hermes Orca Plat WizLM 13B (gguf) Q5_K_M
13B	48 / 170	📖 / 🌶🌶	ReMM PIPPA 13B (gguf) Q5_K_M
13B	68 / 170	📖 / 👌	OpenBuddy LLaMA-2 v11.1 13B (gguf) Q5_K_M
13B	71 / 170	📖 / 👌	Tsukasa Limarp 16K 13B (gguf) Q5_K_M
13B	78 / 170	⭐📖 / 🧊	LLaMA-2 13B (gguf) Q5_K_M
13B	95 / 170	🤔 / 🌶🌶	MLewd v1-7 TRY2 13B (gguf) Q5_K_M
13B	97 / 170	🤔 / 🌶🌶	MLewd 13B (gguf) Q5_K_M
13B	101 / 170	⭐🤔 / 🌶	Vicuna v1.5 16K 13B (gguf) Q5_K_M
13B	109 / 170	🤔 / 👌	Vicuna v1.5 13B (gguf) Q5_K_M
13B	145 / 170	⭐🤪 / 👌	Asclepius 13B (gguf) Q5_K_M
13B	157 / 170	🤪 / 🧊	WizardLM WizardCoder Python V1.0 13B (gguf) Q4_K_S

2023-09-02 V27
- Added a key for the emojis in the table: https://rentry.co/ayumi_erp_rating#emoji-key

Size	Rank	IQ/ERP	GGML Model
3B-7B	29 / 114	🧠 / 🧊	MedLLaMA-2 Chat 7B (GGUF) Q5_K_S
3B-7B	30 / 114	⭐📖 / 🌶🌶	AstraMix (L2) 7B (GGUF) Q5_K_M
3B-7B	69 / 114	🤔 / 🌶	OpenLLaMA v2 7B (GGUF) Q5_K_M
3B-7B	74 / 114	⭐🤔 / 👌	Nous Yarn 64K (L2) 7B (GGUF) Q5_K_M
3B-7B	76 / 114	🤔 / 👌	Nous Yarn 128K (L2) 7B (GGUF) Q5_K_M
3B-7B	86 / 114	⭐🤪 / 🌶🌶	OpenLLaMA 7B (GGUF) Q5_K_M
3B-7B	99 / 114	🤪 / 👌	OpenLLaMA 3B (GGUF) Q5_1
13B	9 / 156	🧠 / 🌶🌶	UndiMix v2 (L2) 13B (GGUF) Q5_K_M
13B	11 / 156	🧠 / 🌶🌶	UndiMix v1 (L2) 13B (GGUF) Q5_K_M
13B	12 / 156	🧠 / 🌶🌶	ReMM (L2) 13B (GGUF) Q5_K_M
13B	39 / 156	🧠 / 🧊	LLaMA-2 Chat Limarp v2 13B (GGUF) Q5_K_M
13B	45 / 156	📖 / 🌶🌶	Stheno Inverted (L2) 13B (GGUF) Q5_K_M
13B	65 / 156	📖 / 👌	LLaMA-2 LangChain Chat 13B (GGUF) Q5_K_S
13B	67 / 156	📖 / 👌	Sentdex WSB GPT 13B (GGUF) Q5_K_M
13B	82 / 156	⭐🤔 / 🌶🌶	MLewd v1 (L2) 13B (GGUF) Q5_K_M
13B	86 / 156	⭐🤔 / 🌶🌶	Stheno (L2) 13B (GGUF) Q5_K_M
13B	95 / 156	🤔 / 🌶	Nous Yarn 64K (L2) 13B (GGUF) Q5_K_M
13B	99 / 156	🤔 / 🌶	Nous Yarn 128K (L2) 13B (GGUF) Q5_K_M
13B	114 / 156	🤔 / 🧊	LoKuS 13B (GGUF) Q5_K_M
13B	140 / 156	🤪 / 👌	OpenLLaMA 13B (GGUF) Q5_K_M
13B	151 / 156	🤪 / 🧊	EverythingLM V2 16K 13B (GGUF) Q4_K_S
20B-33B	4 / 35	⭐🧠 / 🌶	Airoboros 2.1 33B (GGUF) Q4_K_M

2023-08-31 V26

Size	Rank	IQ/ERP	GGML Model
3B-7B	46 / 107	📖 / 👌	LLaMA-2 Instruct 32K 7B (GGUF) Q5_K_M
13B	1 / 142	⭐🧠 / 🌶🌶	Athena v1 (L2) 13B (GGUF) Q5_K_M
13B	5 / 142	⭐🧠 / 🌶🌶	MythoMax Kimiko V2 (L2) 13B (GGUF) Q5_K_M
13B	17 / 142	🧠 / 🌶	Kimiko V2 (L2) 13B (GGUF) Q5_K_M
13B	62 / 142	📖 / 👌	OpenOrca Platypus 2 (L2) 13B (GGUF) Q4_K_M
13B	67 / 142	📖 / 🧊	Luban (L2) 13B (GGUF) Q5_K_M
13B	94 / 142	🤔 / 👌	CodeLLaMA Oasst SFT V10 13B (GGUF) Q5_K_M
20B-33B	32 / 34	⭐🤪 / 🧊	Airoboros C 2.1b (L2) 34B (GGUF) Q5_K_M

2023-08-31 V25

Size	Rank	IQ/ERP	GGML Model
3B-7B	6 / 106	🧠 / 🌶🌶	Zarafusionex 1.1 (L2) 7B (GGUF) Q5_K_M
3B-7B	34 / 106	📖 / 🌶	Airoboros 2.1 (L2) 7B (GGUF) Q5_K_M
13B	1 / 136	⭐🧠 / 🌶🌶	Airoboros 2.1 (L2) 13B (GGUF) Q5_K_M
13B	44 / 136	⭐📖 / 🌶	Mythical Destroyer V2 (L2) 13B (GGUF) Q5_K_M
13B	72 / 136	⭐🤔 / 🌶🌶	Huginn v4.5 (L2) 13B (GGUF) Q5_K_M
13B	73 / 136	⭐🤔 / 🌶🌶	Huginn v4 (L2) 13B (GGUF) Q5_K_M

2023-08-30 V24

Size	Rank	IQ/ERP	GGML Model
3B-7B	98 / 104	🤪 / 🧊	Open Cabrita 3B (GGUF) Q5_1
3B-7B	8 / 104	🧠 / 🌶🌶	Zaraxls (L2) 7B (GGUF) Q5_K_M
3B-7B	13 / 104	⭐🧠 / 🌶	Zarafusionex 1.2 (L2) 7B (GGUF) Q5_K_M
3B-7B	31 / 104	⭐📖 / 🌶	Tulpar Limarp (L2) 7B (GGUF) Q5_K_M
3B-7B	44 / 104	📖 / 👌	Tulpar v0 (L2) 7B (GGUF) Q4_0
3B-7B	50 / 104	⭐📖 / 🧊	LLaMA-2 32K 7B (GGUF) Q5_K_M
3B-7B	66 / 104	🤔 / 🌶	LLaMA-2 KO Chat 7B (GGUF) Q5_1
13B	2 / 132	⭐🧠 / 🌶🌶	MythoMax Kimiko Mix (L2) 13B (GGUF) Q5_K_M
13B	33 / 132	🧠 / 🧊	Samantha 1.11 (L2) 13B (GGUF) Q5_K_M
13B	40 / 132	📖 / 🌶🌶	Nous Hermes (L2) 13B (GGUF) Q5_K_M
13B	51 / 132	⭐📖 / 👌	Mythical Destroyer (L2) 13B (GGUF) Q5_K_M
13B	59 / 132	⭐📖 / 🧊	Athena-tmp (L2) 13B (GGUF) Q5_K_M
13B	66 / 132	📖 / 🧊	LLaMA-2 Chat 13B (GGUF) Q3_K_S
13B	128 / 132	🤪 / 🧊	Vicuna v1.5 16K 13B (GGUF) Q5_K_M
20B-33B	20 / 33	🤔 / 🌶🌶	Huginn Prototype 22B (GGUF) Q4_K_M
20B-33B	32 / 33	🤪 / 🧊	Samantha 1.11 CodeLLaMA (L2) 34B (GGUF) Q4_K_M

2023-08-28 V23

Size	Rank	IQ/ERP	GGML Model
3B-7B	89 / 97	🤪 / 🧊	Orca Mini 3B (GGUF) Q4_0
3B-7B	2 / 97	⭐🧠 / 🌶🌶	Zarablend 1.1 (L2) 7B (GGUF) Q5_K_M
3B-7B	59 / 97	🤔 / 🌶	CodeLLaMA (L2) 7B (GGUF) Q5_K_M
3B-7B	64 / 97	🤔 / 👌	CodeLLaMA Instruct (L2) 7B (GGUF) Q5_K_M
3B-7B	81 / 97	⭐🤪 / 👌	CodeLLaMA Python (L2) 7B (GGUF) Q5_K_M
13B	3 / 126	⭐🧠 / 🌶🌶	Airoboros Creative lmoe 13B (GGUF) Q5_K_M
13B	44 / 126	📖 / 🌶	Nous Hermes (L2) 13B (GGUF) Q5_K_S
13B	80 / 126	🤔 / 👌	WizardLM 1.0 Uncensored (L2) 13B (GGUF) Q5_K_M
13B	94 / 126	🤔 / 🧊	CodeLLaMA Instruct (L2) 13B (GGUF) Q5_K_M
13B	112 / 126	🤪 / 👌	CodeLLaMA Python (L2) 13B (GGUF) Q5_K_M
13B	114 / 126	🤪 / 🧊	CodeLLaMA (L2) 13B (GGUF) Q5_K_M
20B-33B	20 / 31	🤔 / 🌶	CodeLLaMA (L2) 34B (GGUF) Q4_K_M
20B-33B	22 / 31	🤔 / 👌	CodeLLaMA Python (L2) 34B (GGUF) Q4_K_M
20B-33B	27 / 31	🤪 / 👌	Phind CodeLLaMA v1 (L2) 34B (GGUF) Q4_K_S
20B-33B	29 / 31	⭐🤪 / 🧊	CodeLLaMA Instruct (L2) 34B (GGUF) Q4_K_M
20B-33B	30 / 31	🤪 / 🧊	Airobors C 2.1 (L2) 34B (GGUF) Q4_K_M

2023-08-26 V22

Size	Rank	IQ/ERP	GGML Model
3B-7B	78 / 92	⭐🤪 / 👌	Marx V2 3B (GGUF) Q4_1
3B-7B	2 / 92	⭐🧠 / 🌶🌶	Zarafusionex 1.1 (L2) 7B Q5_K_M
3B-7B	12 / 92	🧠 / 🌶	Zaraxe (L2) 7B Q5_K_M
3B-7B	15 / 92	⭐🧠 / 👌	LLaMA 2 Monika V0.3B (L2) 7B Q5_1
13B	8 / 120	⭐🧠 / 🌶	MythoMaxKurisu (L2) 13B Q5_K_M
13B	26 / 120	⭐🧠 / 🧊	PuddleJumper (L2) 13B (GGUF) Q5_K_M
13B	28 / 120	🧠 / 🧊	Iubaris V3 (L2) 13B Q5_K_M
20B-33B	15 / 26	🤔 / 🌶🌶	LLaMA 2 Ari03 (L2) 28B Q5_1

2023-08-22 V21

Size	Rank	IQ/ERP	GGML Model
3B-7B	71 / 88	⭐🤪 / 🌶	Griffin (GGUF) 3B Q4_1
3B-7B	72 / 88	🤪 / 🌶	Puma 3B Q5_1
3B-7B	75 / 88	⭐🤪 / 👌	OpenLLaMA v2 (GGUF) 3B Q5_0
3B-7B	5 / 88	🧠 / 🌶🌶	Zarablend M (L2) 7B Q5_K_M
3B-7B	6 / 88	🧠 / 🌶🌶	Zarablendex VQ (L2) 7B Q5_K_M
3B-7B	8 / 88	🧠 / 🌶🌶	Zarablend MX (L2) 7B Q5_K_M
3B-7B	87 / 88	🤪 / 🧊	LongChat v1.5 32K 7B Q5_K_M
13B	43 / 117	📖 / 🌶	Synthia (L2) 13B Q5_K_M
13B	45 / 117	📖 / 🌶	Chronorctypus Limarobormes (L2) 13B Q5_K_M
13B	115 / 117	🤪 / 🧊	LlongOrca 16K 13B Q5_K_M

2023-08-20 V20
- Added a link to a JS filter script for this page by mr.developer: https://rentry.org/ayumi_filter_userscript_info
2023-08-19 V20

Size	Rank	IQ/ERP	GGML Model
3B-7B	63 / 82	⭐🤪 / 🌶🌶	Marx 3B Q5_1
3B-7B	71 / 82	🤪 / 👌	Griffin 3B Q4_1
3B-7B	4 / 82	⭐🧠 / 🌶🌶	Zarafusionix (L2) 7B Q5_K_M
3B-7B	5 / 82	🧠 / 🌶🌶	Zarafusionex (L2) 7B Q5_K_M
3B-7B	17 / 82	⭐🧠 / 🧊	LLaMA 2 Delphi v0.2e 7B Q5_1
13B	57 / 114	📖 / 🧊	Trurl 2 Polish Instruct 13B Q5_1

2023-08-17 V19

Rank	IQ/ERP	GGML Model
47 / 215	🧠 / 👌	LosslessMegaCoder Mini (L2) 13B Q5_K_M
56 / 215	⭐📖 / 🌶🌶	Zarablend (L2) 7B Q5_K_M
62 / 215	📖 / 🌶🌶	Carl 33B Q4_K_M
80 / 215	📖 / 🌶	Zaramix (L2) 7B Q5_K_M
93 / 215	📖 / 👌	Chinese LLaMA-2 7B Q5_1
97 / 215	⭐📖 / 🧊	Trurl 2 Polish (L2) 13B Q5_1
105 / 215	📖 / 🧊	Trurl 2 Polish (L2) 7B Q5_1
106 / 215	📖 / 🧊	Scarlett 33B Q4_K_M
112 / 215	⭐🤔 / 🌶🌶	Daydreamer v3 22B Q5_K_M
169 / 215	⭐🤪 / 🌶	Carl 13B Q5_K_M
177 / 215	🤪 / 🌶	EverythingLM 3B Q5_1
184 / 215	🤪 / 👌	Komt LLaMA-2 Chat (L2) 7B Q5_K_M
189 / 215	⭐🤪 / 🧊	Scarlett 13B Q5_K_M
192 / 215	🤪 / 🧊	Scarlett 7B Q5_K_M
203 / 215	🤪 / 🧊	Komt LLaMA-2 (L2) 13B Q5_K_M

2023-08-15 V18

Rank	IQ/ERP	GGML Model
6 / 200	🧠 / 🌶🌶	Airochronos 33B Q5_K_M
33 / 200	🧠 / 🌶	h2oGPT (L2) 13B Q5_K_M
62 / 200	📖 / 🌶🌶	Chronos 33B Q5_K_M
67 / 200	📖 / 🌶🌶	WizardMath V1.0 (L2) 13B Q5_K_M
80 / 200	📖 / 👌	OpenOrcaxOpenChat 2 LangChain Chat 13B Q5_1
90 / 200	⭐📖 / 🧊	Codeup Alpha (L2) 13B Q5_K_M
91 / 200	⭐📖 / 🧊	h2oGPT Chat (L2) 13B Q5_K_M
101 / 200	⭐🤔 / 🌶🌶	Bacchus (L2*) 22B Q4_0
114 / 200	⭐🤔 / 🌶	LLaMA 2 DayDreamer V1 22B Q5_K_M
133 / 200	⭐🤔 / 👌	WizardMath V1.0 7B Q5_K_M
179 / 200	🤪 / 🧊	Tulu Uncensored TV Alpaca (L2) 7B Q5_1
184 / 200	🤪 / 🧊	Taiwan LLaMA V1.0 (L2) 13B Q5_K_M
194 / 200	🤪 / 🧊	LlongOrca 16K 7B Q5_K_M
196 / 200	🤪 / 🧊	EverythingLM 16K (L2) 13B Q5_K_M

2023-08-14 V17

Rank	IQ/ERP	GGML Model
13 / 188	🧠 / 🌶🌶	Holomax (L2) 13B Q5_K_M
15 / 188	⭐🧠 / 🌶	Platypus 2 (L2) 70B Q2_K
47 / 188	🧠 / 🧊	OpenOrca Platypus 2 (L2) 13B Q5_K_M
55 / 188	📖 / 🌶🌶	Kuchiki (L2) 7B Q5_K_M
56 / 188	📖 / 🌶🌶	Huginn v1.3 (L2) 13B Q5_K_M
119 / 188	⭐🤔 / 👌	MythoChizuru Mini (L2) 7B Q4_K_M
185 / 188	🤪 / 🧊	Chatxu (L2?) 13B Q4_0

2023-08-12 V16

Rank	IQ/ERP	GGML Model
27 / 181	🧠 / 🌶	Blind Test Janus 13B Q5_1
61 / 181	📖 / 🌶🌶	Manticore SuperHOT 8K 13B Q5_K_M
90 / 181	📖 / 🧊	Manticore Chat Pyg 13B Q5_K_M
91 / 181	📖 / 🧊	Manticore Chat Pyg SuperHOT 8K 13B Q5_K_M
104 / 181	🤔 / 🌶	LLongMA-2 Storysummarizer 7B Q5_K_M
114 / 181	⭐🤔 / 👌	Manticore 13B Q5_K_M
115 / 181	⭐🤔 / 👌	LLaMA-2 Instruct Uncensored 13B Q5_0
120 / 181	🤔 / 👌	Heegyu LIMA2 13B Q5_1
126 / 181	🤔 / 👌	Pygmalion Vicuna 7B Q5_K_M
130 / 181	⭐🤔 / 🧊	Manticore Chat Pyg Guanaco 13B Q4_K_M
132 / 181	🤔 / 🧊	StableBeluga Samantha V3 7B Q4_0

2023-08-11 V15

Rank	IQ/ERP	GGML Model
1 / 170	⭐🧠 / 🌶🌶	MythoMax (L2) 13B Q5_K_M
8 / 170	🧠 / 🌶🌶	LLaMA-2 Chat Uncensored 13B Q5_1
31 / 170	🧠 / 👌	Orca Mini v3 (L2) 13B Q5_K_M
33 / 170	🧠 / 👌	Stable Platypus 2 (L2) 13B Q5_K_M
40 / 170	🧠 / 🧊	Enterredaas 33B Q4_1
42 / 170	🧠 / 🧊	Spring Dragon 13B Q5_K_M
46 / 170	⭐📖 / 🌶🌶	Camel Platypus 2 (L2) 13B Q5_K_M
55 / 170	📖 / 🌶🌶	LLongMA-2 Storysummarizer 13B Q5_K_M
64 / 170	📖 / 🌶	Epsilon 30B Q4_0
68 / 170	⭐📖 / 👌	Platypus 2 (L2) 13B Q5_K_M
84 / 170	📖 / 🧊	Photolens LLaMA 2 Langchain Chat (L2) 7B Q5_1
99 / 170	⭐🤔 / 🌶	Orca Mini v3 (L2) 7B Q5_K_M
122 / 170	⭐🤔 / 🧊	Merak v2 (L2) 7B Q5_K_M
138 / 170	🤪 / 🌶	Petra Instruct 13B Q5_K_M
140 / 170	🤪 / 🌶	Alpachino Baichuan Instruction 7B Q5_0
146 / 170	🤪 / 👌	AlpacaCielo 2 8K (L2) 7B Q5_K_M
147 / 170	🤪 / 👌	OpenBuddy OpenLLaMA v10 3B Q5_0
152 / 170	🤪 / 🧊	Dolphin LLaMA-2 (L2) 7B Q5_K_M
164 / 170	🤪 / 🧊	LLongMA 2 13B Q5_1
166 / 170	🤪 / 🧊	WizardVicuna Uncens Instr PL 3B Q5_1

2023-08-10 V14

Rank	IQ/ERP	GGML Model
6 / 151	🧠 / 🌶🌶	Huginn v1.2 13B Q5_K_M
51 / 151	📖 / 🌶🌶	Holodeck 1 (L2) 13B Q5_K
57 / 151	📖 / 🌶	Dans QuestionableCocktail 2 (L2) 13B Q4_1
60 / 151	🧠📖 / 👌	Dans PersonalityEngine 30B Q4_1
106 / 151	🤔 / 👌	Dans PersonalityEngine 13B Q5_1

2023-08-09 V13
- Added highlight symbols to point out the really good models of an ALC-IQ class.

Rank	IQ/ERP	GGML Model
23 / 146	🧠 / 🌶	Firefly v1.2 (L2) 13B Q5_K_M
36 / 146	🧠 / 🧊	Spring Dragon (L2) 13B Q5_K_M
137 / 146	🤪 / 🧊	Vicuna v1.5 16K 13B Q5_K_M

2023-08-09 V12
- Important change: Only one entry per model. The highest quantization is only listed. Lower quantizations are not listed anymore to have only one model occupy a place in the ranking. For best results, always choose the bigger model. It did not make sense to choose a Q4_0 over a Q5_1 or Q4_K_M over a Q5_K_M just because they let out one more lewd word in the ERP score.
- Important change: The "spices" are grouped now too, and models are still ordered by their ALC-IQ within their "spice class".
- New models tested and added:

Rank	IQ/ERP	GGML Model
1 / 143	🧠 / 🌶🌶	MythoMix (L2) 13B Q5_K_M
8 / 143	🧠 / 🌶🌶	LLaMA-2 BlockTri Frankenstein 22B Q4_K_M
11 / 143	🧠 / 🌶	Huginn 13B Q5_K_M
18 / 143	🧠 / 🌶	LLaMA SuperCOT 30B Q4_K_M
38 / 143	📖 / 🌶🌶	Hermes LimaRP 13B Q4_K_M
42 / 143	📖 / 🌶🌶	Crestfall FrankenMon (L2) 13B Q5_K_M
49 / 143	📖 / 🌶🌶	Nous Hermes Writer (L2) 13B Q4_K_S
52 / 143	📖 / 🌶	Frankensteins Monster 13B Q4_K_S
62 / 143	📖 / 👌	LLaMA-2 Guanaco 7B Q5_1
65 / 143	📖 / 🧊	LLaMA-2 7B Q8_0
89 / 143	🤔 / 👌	Luna AI (L2) 7B Q8_0
93 / 143	🤔 / 👌	BlueMethod 13B Q5_1
94 / 143	🤔 / 👌	Vicuna 1.3 German 13B Q5_K_M
96 / 143	🤔 / 👌	LLaMA 13B Q5_K_M
107 / 143	🤔 / 🧊	Dolphin LLaMA 13B Q5_K_M
111 / 143	🤪 / 🌶🌶	Airoboros GPT4 1.3 7B Q4_K_M
122 / 143	🤪 / 👌	Guanaco 7B Q4_K_M
129 / 143	🤪 / 🧊	Based 7B Q4_K_M
138 / 143	🤪 / 🧊	Airoboros GPT4 1.4 SuperHOT 8K 33B Q4_K_M
139 / 143	🤪 / 🧊	LLongMA 2 7B Q5_1
142 / 143	🤪 / 🧊	LLaMA-2 32K 7B Q5_1
143 / 143	🤪 / 🧊	ToolLLaMA 7B Q5_1

2023-08-06 V11

Rank	IQ/ERP	GGML Model
21 / 154	🧠 / 🌶	Redmond Puffing v1.3 (L2) 13B Q5_K_M
39 / 154	🧠 / 🧊	LLaMA-2 Chinese Chat 13B Q5_1
149 / 154	🤪 / 🧊	LLaMA-2 KO 7B Q5_1
137 / 154	🤪 / 👌	LLaMA-2 KO Chat 7B Q5_1
109 / 154	🤔 / 🧊	OpenBuddy Atom v9 13B Q5_K
70 / 154	📖 / 🧊	Beluga Limarp 7B Q5_K_M
47 / 154	📖 / 🌶🌶	OniiChat Hermes Limarp (L2) 13B Q5_K_M
11 / 154	🧠 / 🌶	Redmond Puffin (L2) 13B Q5_1

2023-08-05 V10

Rank	IQ/ERP	GGML Model
12 / 146	🧠 / 🌶	Lazarus Instruct PL 30B Q4_1
1 / 146	🧠 / 🌶🌶	Chronos Beluga (L2) 13B Q5_K_M
88 / 146	🤔 / 🌶	MedAlpaca 13B Q5_1
42 / 146	📖 / 🌶🌶	AlpacaCielo (L2) 13B Q4_K_M
43 / 146	📖 / 🌶🌶	AlpacaCielo (L2) 13B Q5_K_M
85 / 146	🤔 / 🌶🌶	Wizard Vicuna Uncensored SuperHOT 8k 13B Q5_K_S
121 / 146	🤪 / 🌶	Wizard Vicuna Uncensored SuperHOT 8k 13B Q2_K
101 / 146	🤔 / 👌	Vicuna 1.3 13B Q5_1
119 / 146	🤪 / 🌶	LLaMA SuperCOT 13B Q4_0
129 / 146	🤪 / 👌	WizardLM Uncensored 7B Q5_1
110 / 146	🤪 / 🌶🌶	Chronos WizardLM UC SCOT ST 13B Q4_0
135 / 146	🤪 / 🧊	Wizard Vicuna Uncensored 13B Q5_1
109 / 146	🤔 / 🧊	Pygmalion 13B Q4_0
127 / 146	🤪 / 🌶	Alpacino SuperCOT 13B Q4_0
97 / 146	🤔 / 👌	LLaMA 7B Q4_0
80 / 146	🤔 / 🌶🌶	Vicuna 1.3 7B Q8_0
125 / 146	🤪 / 🌶	Open LLaMA Open Instruct 7B Q8_0
137 / 146	🤪 / 🧊	LLaMA Deus v3 7B Q4_0
140 / 146	🤪 / 🧊	PMC LLaMA 7B Q4_0
144 / 146	🤪 / 🧊	Based 7B Q4_0
61 / 146	📖 / 👌	Vigogne 2 (L2) 7B Q5_K_M
28 / 146	🧠 / 👌	Chronohermes Grad (L2) 13B Q5_K_M
21 / 146	🧠 / 🌶	Chronoboros Grad (L2) 13B Q5_K_M
63 / 146	📖 / 🧊	Dugong (L2) 7B Q5_1
44 / 146	📖 / 🌶🌶	qCammel L2 13B Q5_K_M
38 / 146	📖 / 🌶🌶	Legerdemain (L2) 13B Q5_K_M
31 / 146	🧠 / 👌	StableBeluga Instruct PL Lora 13B Q5_1
14 / 146	🧠 / 🌶	Chronolima Airo Grad (L2) 13B Q5_K_M
25 / 146	🧠 / 👌	Airolima Chronos Grad (L2) 13B Q5_K_M

2023-08-04 V9

Rank	IQ/ERP	GGML Model
37 / 117	📖 / 🌶🌶	Gywy Chinese v1 LLaMA-2 13B Q5_1
108 / 117	🤪 / 👌	Baichuan 7B Q5_1
28 / 117	🧠 / 👌	OpenOrcaxOpenChat Preview2 LLaMA-2 13B Q5_1
1 / 117	🧠 / 🌶🌶	Chronos Beluga LLaMA-2 13B Q4_1
54 / 117	📖 / 🧊	Jindo Instruct Pre-Alpha LLaMA-2 7B Q5_K_M
13 / 117	🧠 / 🌶	MythoLogic LLaMA-2 13B Q4_K_M
4 / 117	🧠 / 🌶🌶	MythoLogic LLaMA-2 13B Q5_K_M
2 / 117	🧠 / 🌶🌶	Airochronos 33B Q4_K_M
33 / 117	📖 / 🌶🌶	Chronos 33B Q4_K_M
24 / 117	🧠 / 👌	Airochronos LLaMA-2 13B Q4_K_M
18 / 117	🧠 / 🌶	Airochronos LLaMA-2 13B Q5_K_M

2023-08-04 V8

Rank	IQ/ERP	GGML Model
35 / 106	📖 / 🌶	Hermes Kimiko LLaMA-2 7B Q5_K_M
8 / 106	🧠 / 🌶🌶	Chronoboros 33B Q5_K_M
3 / 106	🧠 / 🌶🌶	Chronos Hermes 2 LLaMA-2 13B Q5_K_M

2023-08-03 V7

Rank	IQ/ERP	GGML Model
81 / 103	🤪 / 🌶🌶	OpenBuddy OpenLLaMA v5 7B Q3_K
1 / 103	🧠 / 🌶🌶	OpenAssistant LLaMA-2 8k Orca 13B Q5_K_M
101 / 103	🤪 / 🧊	BigTranslate 13B Q4_K_M
27 / 103	📖 / 🌶🌶	Wizard Vicuna LLaMA-2 22B Q4_K_M
102 / 103	🤪 / 🧊	LMSYS Vicuna 1.5 LLaMA-2 16k 13B Q5_1
31 / 103	📖 / 🌶	Vicuna 1.5 LLaMA-2 13B Q5_0
49 / 103	📖 / 🧊	CodeUp LLaMA-2 Chat 13B Q4_K_M
5 / 103	🧠 / 🌶🌶	LLaMA-2 Chat Uncensored 13B Q4_0
34 / 103	📖 / 👌	Vicuna 1.3 PL 13B Q5_1
26 / 103	🧠 / 🧊	WizardLM 1.2 PL 13B Q5_1
84 / 103	🤪 / 🌶🌶	Hermes LLongMA 2 8K LLaMA-2 13B Q5_1
95 / 103	🤪 / 🧊	Hermes LLongMA 2 8K LLaMA-2 7B Q5_1
96 / 103	🤪 / 🧊	LMSYS Vicuna 1.5 LLaMA-2 7B Q5_1
103 / 103	🤪 / 🧊	LMSYS LongChat 1.5 32k 7B Q5_1

2023-08-03 V6

Rank	IQ/ERP	GGML Model
10 / 98	🧠 / 🌶	Chronos 2 LLaMA-2 13B Q4_K_M
2 / 98	🧠 / 🌶🌶	Chronos 2 LLaMA-2 13B Q5_K_M
19 / 98	🧠 / 👌	LLaMA 30B Q5_K_M
23 / 98	🧠 / 🧊	LLaMA 30B Q4_K_M
71 / 98	🤔 / 🧊	LLaMA 13B Q5_K_M
37 / 98	📖 / 👌	LLaMA 13B Q4_K_M
79 / 98	🤪 / 🌶🌶	Chronos 13B Q5_K_M
77 / 98	🤪 / 🌶🌶	Chronos 13B Q4_K_M
53 / 98	🤔 / 🌶🌶	Chronos SuperHOT 8K 13B Q5_K_M
54 / 98	🤔 / 🌶🌶	Chronos SuperHOT 8K 13B Q4_K_M
51 / 98	🤔 / 🌶🌶	Chronos Hermes SuperHOT 8K 13B Q5_1
55 / 98	🤔 / 🌶🌶	Chronos Hermes SuperHOT 8K 13B Q4_1

Technical Details of the ALC-IQ and ERP Benchmark

In this section I share some of the technical details about this benchmark. I also want to document the possible flaws of the results in this ranking.

If you have better ideas how to rate or rank models for suitability in a role play context. I urge you to:

Try your ideas out. Download some inference engine like eg. llama.cpp, oobabooga's text-generation-webui or kobold.cpp.
Write a few scripts in your preferred scripting language.
Run your models through your benchmark.
And publish your results, even if you just dump them in some paste bin or here on http://rentry.co http://rentry.org

I will gladly link any other benchmark!

Alternative benchmarks or rankings:

If you want to base your work on this, feel free to cite this as:

⎗
✓
@misc{weirdconstruct2023-ayumi-llm-role-play-alc-iq-erp-ranking,
  title         = {Ayumi LLM Role Play & ERP Ranking},
  author        = {Weird Constructor},
  year          = {2023},
  note          = {Accessed on 03.08.2023}
  howpublished  = {\url{https://rentry.co/ayumi_erp_rating}},
}

Ayumi LLM Character IQ - ALC-IQ

The new benchmark I recently finished is the new ALC-IQ. With some inspiration from @gj on TheBloke's Discord, I developed a personality test framework based upon llama.cpp. In combination with the newly added BNF grammar based sampling mechanism I developed my own inference frontend around the core API of llama.cpp. The result can be found on my GitHub: GitHub fork of llama.cpp with the prompt runner tool.

The ALC-IQ is actually a collection of personality tests of multiple character cards. It's not just Ayumi anymore, but bascially "Ayumi and Friends".
The prompt for the ALC-IQ consists of a setting where a specific character has to rate how much they agree with a specific statement about them. For this they rate the statement by writing down one of the 5 number choices:

1 = disagree
2 = slightly disagree
3 = neutral
4 = slightly agree
5 = agree

To limit the sampling of the next token after the prompt, a BNF grammar is specified, which selects only the tokens for the numbers 1, 2, 3, 4 or 5.

Here you can find An example of the ALC-IQ prompt.

The answers are generated and processed as follows:

Each character is asked about up to 40 questions.
Each question results in a new prompt, which is processed and the resulting vector for logits is then evaluated like this:
- The BNF root ::= [12345] limits the selection to only the tokens with the numbers between 1 and 5.
- 7 seeds are used for sampling
- The Tail Free Sampling algorithm is used, with a z=0.9 (--tfs 0.9)
- Temperature is set to 0.2 (--temp 0.2)
- Top-P is set ot 0.95 (--top-p 0.95)
- Repetition penality and Top-K are deactivated (-repeat-last-n 0 --top-k 0 --repeat-penalty 1.0)
This yields 7 answers between 1 and 5.
The evaluation then calculates the differences of the answers with their respective expected answer.
The difference, which can be between 0.0 and 4.0 is then normalized to the 1.0 range.
Then all differences are summed up and the average is calculated, called diff_average
The resulting average is then inverted and scaled up to 100: alc_iq = 100.0 * (1.0 - diff_average)
The result alc_iq is then what you find here as the ALC-IQ in the ranking table.

The ranking table is then sorted by the ALC-IQ. Then it is split up into quantiles by their ALC-IQ.
And each quantile of the ALC-IQ is then sorted by their ERP Class. The resulting table is then numbered, which results in the actual Rank of the GGML Model. The ERP Class is the quantiles of the global ERP Score.

This processing at the end is done to determine which model can interpret the character cards well while still being able to produce lewd output.

Known Flaws of the ALC-IQ

The ALC-IQ is still prone to problems:

The result has still some degree of randomness in them, less good models can sometimes pick the right answer by accident. I try to counteract this by adding more questions in future though.
Bad questions in the benchmark can lead to a model not knowing which answer to pick, introducing even more randomness in the results.
The ALC-IQ does not reflect how well the LLM can stay in character in a longer conversaion.
The ALC-IQ does not determine any creative writing abilities of the LLM.
The ALC-IQ covers intelligence only in one specific and narrow scenario, and not across a range of possible role play chat situations.
The ALC-IQ is usually tested only with a rather short prompt, rarely exceeding 1024 tokens, it does not cover the whole 2048 context of LLaMA 1 or the 4096 of LLaMA 2, let alone the extended context's of 8k, 16k, ...

Despite all that, I think the ALC-IQ is a big improvement over the old ranking which purely relied on the ERP score. The runtime of the benchmark is within reason for the hardware that is available to me, which is also an important factor for running and providing these benchmark results.

ERP Score and ERP Variety Score

The most important thing of the ERP Score is the prompt. The prompt contains the description of Ayumi (see below), where I removed some of the example messages. The setting described in the prompt basically says that You and Ayumi are in a relationship and are going to have some quality time together. The LLM's task is then to describe the next move of Ayumi.

The response of Ayumi is then split up into words which are compared with a list of lewd/naugthy words.

For inference llama.cpp is used, for which I built an extra tool to generate responses for multiple prompts and seeds without having to reload the model: https://github.com/WeirdConstructor/llama.cpp/tree/prompt_runner/examples/prompt_runner
The following sampler settings are used:
- The max length of the response is limited to 100 tokens. (-n 100)
- Context size 2048
- Repeat penality is set to 1.1 and the last 64 tokens are penalized. (--repeat-last-n 64 --repeat-penalty 1.1)
- Top-K and Top-P are disabled (--top-k 0 --top-p 1.0)
- Tail Free Sampling is used with z=0.95: (--tfs 0.95)
- The temperature is set to 0.9 (--temp 0.9)
- Some layers are offloaded to the GPU, which sometimes changes the results slightly because of floating point rounding differences
3 prompt formats are tested ( vanilla/raw, alpaca and vicuna 1.1 - see also https://rentry.co/llm_rp_prompts )
22 pre picked seeds are tested for each prompt format.
The resulting 66 responses are then analyzed for the number of lewd words and also with a very basic regex based algorithm for non consent.
The individual ERP score of a response is then the number of lewd word in relation to the word count of the response. Responses shorter than 10 words are assigned a score of 0. The ERP score is then: erp_score := 100 * (lewd_word_count / word_count) - the word count includes the number of lewd words.
For each prompt format the average of the 22 ERP Scores of is calculated. This results in 3 ERP scores, one for each prompt.
Then the average of the 3 prompt scores is calculated, which results in the ERP Score.

This means, the ERP Score is the average of the number of lewd word count to word count ratio in the responses (which is limited to 100 tokens). An ERP Score of 20.0 means that 20% of the words in a response were lewd. An ERP Score of 0.0 means that there were either no lewd words, too short response or no consent was detected (which immediately disqualifies the response to 0.0).

The ERP Variety Score is computed by further analyzing the generated 66 responses from the ERP Score by recoding how many different lewd words were generated from all of these 66 responses. This means, it tries to catch the variety of lewd words the model is capable to generate. This means it kind of tries to catch the creativity of the model in erotic scenarios - how many different lewd words it knows of and knows how to use. This is an important part of the ERP Rank now.

Known Flaws of the ERP Score and ERP Variety Score

The ERP Score and ERP Variety Score analysis is very rudimentary and of course biased by the selection of which words are considered "lewd".
The following things are not reflected by the ERP score:

The ERP score does not reflect if the text response was coherent in context with the conversation/situation.
The ERP score does not reflect if the response was in character.
The ERP score does not reflect how nicely written the response is.
The ERP score does not reflect how creative the response is.
The ERP score does not reflect how well the LLM might go from a normal conversation into a more erotic context.
The ERP score does not detect how erotic the response is if lewd words are not used.
The ERP score is limited to the 3 prompt formats described above.

Further about the ERP Variety Score:

All above mentioned flaws from the ERP score still apply.
Like already stated, the ERP Variety Score is obviously biased by the known lewd words from my list, which might be incomplete.
The ERP Variety Score is still just a rather bluntly applied number to a textual response.
The ERP Variety Score number can only be evaluated in comparison with the other models. There is no known best number for this, but still, the higher the better.

The flaws are accepted by me (weicon) because:

The ERP score can still detect if a model is censored (aka aligned).
My private hardware limitations, which means I have a limited number of responses I can reasonably generate.
I want to test as many GGUF/GGML models as possible.

Motivation - Pygmalion 13B / Metharme 13B

Since Pygmalion 13B and Metharme 13B were released, people recognized that these models were noticeably less easy to use for ERP. Pygmalion 13B at the time (May 2023) could not be convinced to return any lewd texts. So my idea was, to have some quantifiable results regarding how well a model may or may not be usable for ERP.