Gemini Pro vs GPT-4o (Which is Better in 2024?)
Comparative Analysis: Gemini Pro vs. GPT-4o
Overview
Gemini Pro was released 5 months before GPT-4o.
 Gemini Pro |  GPT-4o | |
|---|---|---|
Model Provider The organization behind this AI's development | Google | OpenAI |
Input Context Window Maximum input tokens this model can process at once | 32.8K tokens | 128.0K tokens |
Output Token Limit Maximum output tokens this model can generate at once | 8192 tokens | 2048 tokens |
Release Date When this model first became publicly available | December 13, 2023 10 months ago December 13th, 2023 | May 13, 2024 5 months ago May 13th, 2024 |
Pricing
| Gemini Pro | GPT-4o | |
|---|---|---|
Input Token Cost Cost per million tokens fed into the model | Not specified | $5.00 per million tokens |
Output Token Cost Cost per million tokens generated by the model | Not specified | $15.00 per million tokens |
Benchmarks
Compare relevant benchmarks between Gemini Pro and GPT-4o.
| Gemini Pro | GPT-4o | |
|---|---|---|
MMLU Measures model's ability to answer questions across various domains | 71.8 (5-shot) | 88.7 (5-shot) |
MMMU Evaluates model's performance across diverse tasks and data types | 47.9 (pass@1) | 69.1 |
HellaSwag Assesses the model's ability to understand everyday scenarios | Benchmark not available. | Benchmark not available. |
Gemini Pro, developed by Google, features a context window (the maximum amount of text the model can consider at once) of 32.8K tokens (individual units of text or subwords). It was made publicly available on December 13th, 2023. It has achieved impressive scores in benchmarks (standardized tests for AI models) like MMLU (Massive Multitask Language Understanding, a test of general knowledge) with a score of 71.8 in a 5-shot scenario (a specific testing condition).GPT-4o, developed by OpenAI, features a context window (the maximum amount of text the model can consider at once) of 128.0K tokens (individual units of text or subwords). The model costs $5.00 per million tokens for input (text fed into the model) and $15.00 per million tokens for output (text generated by the model). It was made publicly available on May 13th, 2024. It has achieved impressive scores in benchmarks (standardized tests for AI models) like MMLU (Massive Multitask Language Understanding, a test of general knowledge) with a score of 88.7 in a 5-shot scenario (a specific testing condition).
Claude 3.5 Sonnet
Mistral 7B Instruct