Claude 3.5 Sonnet vs Gemini Pro (Which is Better in 2024?)

Compare

Comparative Analysis: Claude 3.5 Sonnet vs. Gemini Pro

Interested in trying out these models in one collaborative workspace?Try out Flamingo.ai for free

Overview

Gemini Pro was released 6 months before Claude 3.5 Sonnet.

	Claude 3.5 Sonnet	Gemini Pro
Model Provider The organization behind this AI's development	Anthropic	Google
Input Context Window Maximum input tokens this model can process at once	200.0K tokens	32.8K tokens
Output Token Limit Maximum output tokens this model can generate at once	4096 tokens	8192 tokens
Release Date When this model first became publicly available	June 20, 2024 3 months ago June 20th, 2024	December 13, 2023 9 months ago December 13th, 2023

Pricing

	Claude 3.5 Sonnet	Gemini Pro
Input Token Cost Cost per million tokens fed into the model	$3.00 per million tokens	Not specified
Output Token Cost Cost per million tokens generated by the model	$15.00 per million tokens	Not specified

Benchmarks

Compare relevant benchmarks between Claude 3.5 Sonnet and Gemini Pro.

	Claude 3.5 Sonnet	Gemini Pro
MMLU Measures model's ability to answer questions across various domains	90.4 (5-shot CoT) Source	71.8 (5-shot) Source
MMMU Evaluates model's performance across diverse tasks and data types	68.3 (0-shot CoT) Source	47.9 (pass@1) Source
HellaSwag Assesses the model's ability to understand everyday scenarios	Benchmark not available.	Benchmark not available.

Claude 3.5 Sonnet, developed by Anthropic, features a context window (the maximum amount of text the model can consider at once) of 200.0K tokens (individual units of text or subwords). The model costs $3.00 per million tokens for input (text fed into the model) and $15.00 per million tokens for output (text generated by the model). It was made publicly available on June 20th, 2024. It has achieved impressive scores in benchmarks (standardized tests for AI models) like MMLU (Massive Multitask Language Understanding, a test of general knowledge) with a score of 90.4 in a 5-shot CoT scenario (a specific testing condition).

Gemini Pro, developed by Google, features a context window (the maximum amount of text the model can consider at once) of 32.8K tokens (individual units of text or subwords). It was made publicly available on December 13th, 2023. It has achieved impressive scores in benchmarks (standardized tests for AI models) like MMLU (Massive Multitask Language Understanding, a test of general knowledge) with a score of 71.8 in a 5-shot scenario (a specific testing condition). Google logo