updated 24 Jul 2026

Open Source LLM Leaderboard

This open source LLM leaderboard displays the latest public benchmark performance for open-weight and open-source models released after April 2024. The data comes from model providers as well as independently run evaluations by Vellum or the open-source community. We feature results from non-saturated benchmarks, excluding outdated benchmarks (e.g. MMLU).

Best Overall (Humanity's Last Exam)

60%45%30%15%0%

Kimi K3

54.7

GLM 5.2

Kimi K2.6

51.6

DeepSeek V4 Flash

48.2

DeepSeek V4 Pro

44.9

Kimi K2 Thinking

30.1

Kimi K2.5

14.9

GPT oss 120b

10.9

GPT oss 20b

8.6

DeepSeek-R1

Best Overall (Humanity's Last Exam)
Model	Score
Kimi K3	56%
GLM 5.2	54.7%
Kimi K2.6	54%
DeepSeek V4 Flash	51.6%
DeepSeek V4 Pro	48.2%
Kimi K2 Thinking	44.9%
Kimi K2.5	30.1%
GPT oss 120b	14.9%
GPT oss 20b	10.9%
DeepSeek-R1	8.6%

Top open source models per tasks

Best in Reasoning (GPQA Diamond)

95%92%88%85%81%

93.5

Kimi K3

MiniMax M3

91.2

GLM 5.2

90.5

Kimi K2.6

90.1

DeepSeek V4 Pro

Best in Reasoning (GPQA Diamond)
Model	Score
Kimi K3	93.5%
MiniMax M3	93%
GLM 5.2	91.2%
Kimi K2.6	90.5%
DeepSeek V4 Pro	90.1%

Best in Agentic Coding (SWE Bench)

85%81%76%72%68%

80.6

DeepSeek V4 Pro

80.5

MiniMax M3

80.2

Kimi K2.6

DeepSeek V4 Flash

76.8

Kimi K2.5

Best in Agentic Coding (SWE Bench)
Model	Score
DeepSeek V4 Pro	80.6%
MiniMax M3	80.5%
Kimi K2.6	80.2%
DeepSeek V4 Flash	79%
Kimi K2.5	76.8%

New

Best in Computer Use (OSWorld)

75%72%69%66%63%

73.1

Kimi K2.6

70.1

MiniMax M3

Best in Computer Use (OSWorld)
Model	Score
Kimi K2.6	73.1%
MiniMax M3	70.1%

New

Best in Browsing (BrowseComp)

95%89%84%78%72%

91.2

Kimi K3

85.9

DeepSeek V4 Flash

83.5

MiniMax M3

83.4

DeepSeek V4 Pro

83.2

Kimi K2.6

Best in Browsing (BrowseComp)
Model	Score
Kimi K3	91.2%
DeepSeek V4 Flash	85.9%
MiniMax M3	83.5%
DeepSeek V4 Pro	83.4%
Kimi K2.6	83.2%

New

Best in Terminal Use (Terminal-Bench 2.1)

90%68%45%23%0%

88.3

Kimi K3

GLM 5.2

MiniMax M3

50.8

Kimi K2.5

35.7

Kimi K2 Thinking

Best in Terminal Use (Terminal-Bench 2.1)
Model	Score
Kimi K3	88.3%
GLM 5.2	81%
MiniMax M3	66%
Kimi K2.5	50.8%
Kimi K2 Thinking	35.7%

Best in Visual Reasoning (ARC-AGI 2)

15%14%12%11%9%

Kimi K2.5

Best in Visual Reasoning (ARC-AGI 2)
Model	Score
Kimi K2.5	12%

Inference provider comparison

Fastest (Throughput)

2012150910065030

1828.8

Cerebras

749.2

Fireworks AI

592.9

Together AI

476.8

Groq

283

Baseten

Fastest (Throughput)
Provider	Value
Cerebras	1828.8
Fireworks AI	749.2
Together AI	592.9
Groq	476.8
Baseten	283

Lowest Latency (TTFT)

43210

0.27s

Baseten

0.47s

Together AI

0.54s

Cerebras

0.7s

Groq

1.12s

Novita AI

3.43s

Fireworks AI

Lowest Latency (TTFT)
Provider	Value
Baseten	0.27s
Together AI	0.47s
Cerebras	0.54s
Groq	0.7s
Novita AI	1.12s
Fireworks AI	3.43s

Lowest Cost (per 1M tokens)InputOutput

$1$0.75$0.5$0.25$0

$0.05

$0.25

Novita AI

$0.1

$0.5

Baseten

$0.15

$0.6

Fireworks AI

$0.15

$0.6

Together AI

$0.15

$0.6

Groq

$0.35

$0.75

Cerebras

Lowest Cost (per 1M tokens)
Provider	Input	Output
Novita AI	$0.05	$0.25
Baseten	$0.1	$0.5
Fireworks AI	$0.15	$0.6
Together AI	$0.15	$0.6
Groq	$0.15	$0.6
Cerebras	$0.35	$0.75

Compare open source models

	Kimi K3	GLM 5.2
Context size	1,048,576	1,000,000
Cutoff date	-	Mar 2026
I/O cost	$3 / $15	$0.95 / $3
Max output	-	128,000
Latency	4.46s	1.14s
Speed	35.2 t/s	347 t/s

Best Overall (HLE)

Kimi K3

GLM 5.2

54.7

Best in Terminal Use (Terminal-Bench 2.1)

Kimi K3

88.3

GLM 5.2

Best in Agentic Coding (SWE-Bench)

Kimi K3

76.8

GLM 5.2-

Best in Reasoning (GPQA Diamond)

Kimi K3

93.5

GLM 5.2

91.2

Model Comparison

Model	Context size	Cutoff date	I/O cost	Max output	Latency	Speed
Kimi K3	1,048,576	-	$3 / $15	-	4.46s	35.2 t/s
GLM 5.2	1,000,000	Mar 2026	$0.95 / $3	128,000	1.14s	347 t/s
Kimi K2.6	256,000	-	$0.95 / $4	-	0.68s	342.6 t/s
DeepSeek V4 Flash	1000000	Jan 2026	$0.14 / $0.28	384000	1.42s	107.9 t/s
DeepSeek V4 Pro	1000000	Jan 2026	$0.435 / $0.87	384000	1.2s	174.9 t/s
Kimi K2.5	256,000	Apr 2024	$0.6 / $2.5	33,000	0.69s	337.7 t/s
DeepSeek-R1	128,000	Dec 2024	$0.55 / $2.19	8,000	1.18s	30.1 t/s
Llama 3.1 405b	128,000	Dec 2023	$3.5 / $3.5	4096	0.73s	969 t/s
Llama 3.3 70b	128,000	July 2024	$0.59 / $0.7	32,768	0.52s	2500 t/s
DeepSeek V3 0324	128,000	Dec 2024	$0.27 / $1.1	8,000	1.9s	36.4 t/s
Qwen2.5-VL-32B	131,000	Dec 2024	-	8,000	-	-
Gemma 3 27b	128,000	Nov 2024	$0.07 / $0.07	8192	0.72s	59 t/s
Llama 4 Maverick	10,000,000	November 2024	$0.2 / $0.6	8,000	0.45s	126 t/s
Llama 4 Scout	10,000,000	November 2024	$0.11 / $0.34	8,000	0.33s	2600 t/s
MiniMax M3	1,048,576	Mar 2026	$0.6 / $2.4	512,000	0.85s	98.6 t/s

Context window, cost and speed comparison

Models	Context Window	Input Cost / 1M tokens	Output Cost / 1M tokens	Speed (tokens/second)	Latency
Kimi K3	1,048,576	$3	$15	35.2 t/s	4.46 seconds
GLM 5.2	1,000,000	$0.95	$3	347 t/s	1.14 seconds
Kimi K2.6	256,000	$0.95	$4	342.6 t/s	0.68 seconds
DeepSeek V4 Flash	1000000	$0.14	$0.28	107.9 t/s	1.42 seconds
DeepSeek V4 Pro	1000000	$0.435	$0.87	174.9 t/s	1.2 seconds
Kimi K2.5	256,000	$0.6	$2.5	337.7 t/s	0.69 seconds
DeepSeek-R1	128,000	$0.55	$2.19	30.1 t/s	1.18 seconds
Llama 3.1 405b	128,000	$3.5	$3.5	969 t/s	0.73 seconds
Llama 3.3 70b	128,000	$0.59	$0.7	2500 t/s	0.52 seconds
DeepSeek V3 0324	128,000	$0.27	$1.1	36.4 t/s	1.9 seconds
Qwen2.5-VL-32B	131,000	n/a	n/a	n/a	n/a
Gemma 3 27b	128,000	$0.07	$0.07	59 t/s	0.72 seconds
Llama 4 Maverick	10,000,000	$0.2	$0.6	126 t/s	0.45 seconds
Llama 4 Scout	10,000,000	$0.11	$0.34	2600 t/s	0.33 seconds
MiniMax M3	1,048,576	$0.6	$2.4	98.6 t/s	0.85 seconds

Benchmark glossary

Humanity's Last Exam: A crowd-sourced exam of extremely hard questions spanning every academic discipline. Designed to be the final exam before superhuman AI.
GPQA Diamond: Graduate-level science questions curated by domain experts. Tests advanced reasoning across physics, chemistry, and biology.
SWE-Bench Verified: Real GitHub issues from popular Python repos that the model must resolve end-to-end. Measures agentic software engineering ability.
AutoBench: Automation benchmark evaluating a model's ability to complete real-world work automation tasks using tools and multi-step workflows.
OSWorld-Verified: Real-world computer use tasks requiring GUI interaction in desktop environments. Measures end-to-end task completion on a real OS.
BrowseComp: Agentic web search benchmark testing a model's ability to browse and extract information from the web to answer complex questions.
Terminal-Bench 2.1: Terminal and tool use benchmark evaluating a model's ability to execute multi-step tasks in a terminal environment.
ARC-AGI 2: Abstract visual puzzles requiring novel pattern recognition. Tests fluid intelligence and generalization beyond training data.

Open Source LLM Leaderboard

Best Overall (Humanity's Last Exam)

Top open source models per tasks

Best in Reasoning (GPQA Diamond)

Best in Agentic Coding (SWE Bench)

Best in Computer Use (OSWorld)

Best in Browsing (BrowseComp)

Best in Terminal Use (Terminal-Bench 2.1)

Best in Visual Reasoning (ARC-AGI 2)

Inference provider comparison

Fastest (Throughput)

Lowest Latency (TTFT)

Lowest Cost (per 1M tokens)InputOutput

Compare open source models

Model Comparison

Context window, cost and speed comparison

Benchmark glossary

The Personal AI you were promised