Curiosity · AI Model
Orca-Math 7B
Orca-Math 7B is a math-specialised small language model from Microsoft Research, released in 2024. Built on top of Mistral-7B, it is fine-tuned with a synthetic GPT-4-generated 'math dialogue' dataset plus iterative preference-learning. Orca-Math scored roughly 86% on GSM8K, leading the pack of 7B math models at the time.
Model specs
- Vendor
- Microsoft
- Family
- Orca
- Released
- 2024-02
- Context window
- 32,000 tokens
- Modalities
- text
Strengths
- High GSM8K score for a 7B model
- Demonstrates the impact of synthetic training data
- Permissively released via HuggingFace for research
Limitations
- Weaker on competition-level MATH than later math-RL models
- Not a general-purpose chat model
- Small for 2026 deployment — overtaken by Phi-4 and Qwen-Math
Use cases
- School and early-undergraduate math tutoring apps
- SLM research into synthetic-data math training
- Baseline model for fine-tuning STEM chatbots
- Offline math solvers on single-GPU machines
Benchmarks
| Benchmark | Score | As of |
|---|---|---|
| GSM8K | ~86% | 2026-04 |
| MATH | ~32% | 2026-04 |
| MMLU | ~58% | 2026-04 |
Frequently asked questions
What is Orca-Math 7B?
Orca-Math 7B is a Microsoft-released small language model fine-tuned from Mistral-7B on synthetic math dialogues and preference feedback, achieving roughly 86% on GSM8K.
How is Orca-Math different from DeepSeek-Math?
DeepSeek-Math continue-pretrained on a huge math corpus, while Orca-Math focused on high-quality synthetic dialogue fine-tuning and iterative DPO. Both are strong 7B math SLMs with slightly different recipes.
Sources
- Orca-Math on HuggingFace — accessed 2026-04-20
- Orca-Math paper (arXiv) — accessed 2026-04-20