tl;dr: Big sparse MoE models like GLM-5 are surprisingly usable even with significant fractions of the weights on SSD, due to caching dynamics.
I've recently been experimenting with MoE expert weights partially spilling over to SSD. It's much better than I expected. I realized my previous intuition for weights on SSD ("sounds painfully slow") came from analogy to GPU->CPU spillover with dense models, where Amdahl's Law guarantees things will be pretty bad. However, MoE experts on SSD is a very different situation.
Two factors make it work really well. First, you don't have a chunk of weights permanently living on SSD. Rather, you have an mmap'd file paging chunks of the model in/out of RAM, based on the kernel's caching logic, I think basically LRU. If an expert on SSD is not used by a token, then for that token it doesn't matter that you spilled that expert onto SSD. Second, experts' hotness follows a power law distribution, which is very friendly to caching.
Specifically, I recorded activations of GLM-5 from a diverse set of prompts, ~100k tokens generated. I analyzed 128 token chunks within each response, looking at how many experts are used within a given chunk (U), and how many used experts tend to carry over from chunk to chunk - or rather, how many don't (D). The idea: if you have enough RAM to hold at least U experts at once, then every chunk you have to page in no more than D. If you have a shortfall of S (i.e. can only fit U - S experts), then you have to page in D + S. Particularly unfortunate access patterns could make that worse (more likely the less RAM you have, I bet), but for these purposes I think it's ok to simplify. Anyways, for GLM-5 I found U to be 13190 (69.6% of all GLM-5 experts) and D to be 2412. If down/up/gate are all IQ4_XS, that's 19.125MiB per expert, so U is 246 GiB and D is 45GiB. D, and your machine's RAM's shortfall from U, are how much latency you will pay every 128 tokens from SSD reads; beyond that it's just the same latency as if your machine had infinite RAM. So with 256GB RAM and a 7GB/s SSD, the SSD penalty averages 45/(128*7) = 50ms per token.
I wrote a script following this logic to estimate token generation speed for a given hardware config. Can't guarantee I'm not missing something, but at least it reasonably accurately predicted my own machine's performance (just under 3tok/s on GLM-5 with IQ4_XS experts; 3xP40, 288GiB of 2400 DDR4, ~3.5GB/s SSD). It suggests that a 3090 + 128GiB DDR5 (and 7GB/s SSD) should get 3.5tok/s with IQ4_XS down, IQ3_S gate+up, or over 6tok/s if increasing SSD bandwidth to 28GB/s (4 drives in RAID0). Pretty good for a 3.7-bit quant of a ~800B model on 152GB VRAM+RAM!
I think this logic is significantly over-optimistic when RAM is really low, though. I modeled it as, for every expert of the average used expert set size that you can't keep in RAM, you have to load that expert once per 128 tokens. But that's clearly wrong if you imagine 0GB RAM: there you have to load every active expert every token. I think this estimation transitions from wildly optimistic at very low RAM amounts, to basically perfect when you have enough RAM to hold the typical expert set size (S=0, in my terminology). Not sure what shape that transition has, though.
Side notes
Not all layers are equal for this caching logic. The more uniform a layer's activations, the worse for caching. I thought to identify these to prioritize GPU offloading, but it turns out to be pretty consistently the earliest layers with the most uniform activations. So nothing fancy needed; simply offload the first few layers. You will help yourself beyond what would be expected from just upgrading a small chunk of weights from RAM to VRAM bandwidths, by taking pressure off of your SSD.
I believe the SSD friendliness described here should increase as sparsity increases. GLM-5's 8-of-256 is pretty sparse. I think you would not haved wanted to try SSD spillover with Mixtral 8x22B and its 2-of-8. Maybe the lower sparsity of those early open MoE models contributed to all of this not being more common knowledge.
The Smart Expert Reduction feature of ik_llama.cpp could help here. You would add a condition to never skip an already-paged-in expert, and maybe probabilistically page-in/skip based on probability relative to top expert to avoid colder experts being paged out forever.
Code
Here is what I used for this investigation. You can use estimate_speed.py as-is to estimate any hardware's speed on GLM-5 at any quant. You could use the llama.cpp instrumentation patch + analyze_experts.py to gather the necessary --average_active_per_batch and --average_expert_turnover_per_batch parameters to use estimate_speed.py on other models.
The llama.cpp instrumentation for collecting activation data through stdout:
analyze_experts.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | import glob
from collections import Counter
def process_chunk(flat_experts, thresholds):
"""
Given a flat list of expert activations for a 128-token chunk,
return the minimum K experts needed to hit each threshold,
and the set of those top-K experts.
"""
if not flat_experts:
return None, None
counts = Counter(flat_experts)
total_activations = len(flat_experts)
# Sort experts by frequency descending
sorted_experts = counts.most_common()
k_dict = {}
set_dict = {}
cum_sum = 0
for i, (expert, count) in enumerate(sorted_experts):
cum_sum += count
# Check if we've crossed any thresholds we haven't recorded yet
for t in thresholds:
if cum_sum >= total_activations * t and t not in k_dict:
k_dict[t] = i + 1 # +1 because i is 0-indexed
# Store the set of experts up to this point
set_dict[t] = {e for e, _ in sorted_experts[:i+1]}
# Early exit: if we've found all thresholds, stop looping
if len(k_dict) == len(thresholds):
break
return k_dict, set_dict
def main():
files = sorted(glob.glob('single_response_*.txt'))
if not files:
print("No single_response_*.txt files found in the current directory.")
return
thresholds = [0.10, 0.25, 0.50, 0.80, 0.95, 0.99, 0.999, 1.0]
# Accumulators for averages
sum_k = {t: 0.0 for t in thresholds}
count_k = {t: 0 for t in thresholds}
sum_delta = {t: 0.0 for t in thresholds}
count_delta = {t: 0 for t in thresholds}
print(f"Processing {len(files)} files...")
for fpath in files:
with open(fpath, 'r', encoding='utf-8') as f:
current_token_experts = []
all_tokens = []
for line in f:
line = line.strip()
if 'prompt' in line:
raise ValueError('you must first remove all ---began/ended prompt processing--- blocks')
if line.startswith('---token'):
if current_token_experts:
all_tokens.append(current_token_experts)
current_token_experts = []
elif line:
# Line is like "layer 7 expert 188"
current_token_experts.append(line)
# Catch the last token if the file doesn't end with a separator
if current_token_experts:
all_tokens.append(current_token_experts)
# Chunk into exactly 128 tokens, discard remainder
num_chunks = len(all_tokens) // 128
chunks = [all_tokens[i*128 : (i+1)*128] for i in range(num_chunks)]
chunk_results = []
for chunk in chunks:
# Flatten the 128 tokens into a single list of all expert activations in the chunk
flat_experts = [exp for token in chunk for exp in token]
k_dict, set_dict = process_chunk(flat_experts, thresholds)
if k_dict:
chunk_results.append((k_dict, set_dict))
# Accumulate K averages
for k_dict, _ in chunk_results:
for t in thresholds:
sum_k[t] += k_dict[t]
count_k[t] += 1
# Accumulate Delta averages (Turnover)
# Compare chunk i to chunk i+1 ONLY within the same file
for i in range(len(chunk_results) - 1):
_, set_prev = chunk_results[i]
_, set_curr = chunk_results[i+1]
for t in thresholds:
# Experts in current chunk's hot-set that weren't in previous chunk's hot-set
new_experts = set_curr[t] - set_prev[t]
sum_delta[t] += len(new_experts)
count_delta[t] += 1
# Print Results
print("\n" + "="*80)
print(f"{'Threshold':<12} | {'Avg Active Experts (K)':<25} | {'Avg New Experts (Delta)':<25}")
print("-" * 80)
for t in thresholds:
avg_k = sum_k[t] / count_k[t] if count_k[t] > 0 else 0
avg_d = sum_delta[t] / count_delta[t] if count_delta[t] > 0 else 0
# Format percentages nicely
t_str = f"{t*100:>6.1f}%"
if t == 0.999:
t_str = " 99.9%"
print(f"{t_str:<12} | {avg_k:<25.2f} | {avg_d:<25.2f}")
print("="*80)
if __name__ == '__main__':
main()
|
estimate_speed.py