Expected Calibration Error (ECE) Calculator

Paste predicted probabilities and 0/1 labels to get ECE, MCE, Brier score and log loss, a per-bin table and a reliability diagram.

Binning matches scikit-learn calibration_curve with uniform or quantile bins.

Last updatedHow we build & check our tools
Results will appear here once you enter your values.

How expected calibration error is computed

The formula. Sort every prediction into M bins by its predicted probability. For each bin B, take the mean predicted probability (confidence) and the fraction of rows whose label is 1 (accuracy for the positive class). Guo et al. (2017) define ECE = sum over bins of |B|/n times |acc(B) - conf(B)|, so each bin's gap is weighted by how many predictions it holds. MCE is simply the largest gap in any non-empty bin.

Worked example. The default 20 rows with 10 uniform bins give bins such as [0.2, 0.3] holding 0.25, 0.30 and 0.30: mean confidence 0.2833, fraction positive 0.6667, gap 0.3833, weight 3/20. Summing all ten weighted gaps gives ECE = 0.2015, and the largest gap, 0.45 in the (0.4, 0.5] bin, is the MCE. These numbers match scikit-learn 1.7.1 output in our unit tests.

Edge rules. A probability exactly on an inner edge such as 0.3 falls in the lower bin, and 1.0 falls in the last bin, which is how calibration_curve assigns bins with searchsorted.

Reading the reliability diagram

The diagonal is perfect calibration. Each point plots a bin's mean predicted probability on the x axis against its observed fraction of positives on the y axis. Points below the dashed line mean the model is overconfident in that range: it said 0.9 but only 75 percent were positive. Points above the line mean underconfidence.

Typical patterns. Large modern neural networks trained with cross entropy often sit below the diagonal at high confidence, which is the effect Guo et al. documented for ResNets. Gradient boosted trees and naive Bayes often push predictions toward 0 and 1, producing an S shape. Logistic regression is usually close to the diagonal because it directly optimizes log loss.

Fixing it. Temperature scaling, Platt scaling and isotonic regression fit a mapping on a held out set. After fitting, rerun this calculator on the calibrated probabilities from a separate test set; ECE on the same data you fit on will look better than it really is.

Pitfalls: bin count, sample size and strategy

ECE depends on the bins. The same 20 predictions give ECE 0.2015 with 10 uniform bins, 0.1665 with 5 uniform bins and 0.2265 with 10 quantile bins. Always report the bin count and strategy next to the number, and compare models only with identical settings.

Small samples inflate ECE. A bin with 2 rows can only have a fraction positive of 0, 0.5 or 1, so random noise alone creates gaps. With a few hundred rows, 10 to 15 bins is common; with fewer than about 100 rows, use 5 bins or quantile bins so each bin has enough samples.

Uniform versus quantile. Uniform bins have equal width, so most predictions may crowd into one or two bins when a model is very confident. Quantile bins use np.percentile edges so each bin holds roughly the same count, which gives steadier estimates but uneven widths. Empty bins are shown as n/a and skipped.

ECE versus Brier score and log loss

Different questions. ECE measures only calibration. Brier score, the mean of (y - p) squared, and log loss, the mean negative log likelihood, are proper scoring rules that mix calibration with discrimination and the inherent noise of the data. The scikit-learn user guide notes that a lower Brier score does not by itself prove better calibration.

Log loss clipping. A prediction of exactly 0 for a positive label would make log loss infinite. Like scikit-learn, this tool clips probabilities to [eps, 1 - eps] with eps = 2.22e-16, so one such row adds about 36.04 to the summed loss instead of breaking the result.

Binary only. The inputs are the probability of the positive class. For a multi-class model you can compute top-label ECE by entering the top class confidence and 1 if that prediction was correct, which is exactly the setting in the Guo et al. paper.

Frequently Asked Questions

Common questions about the Expected Calibration Error (ECE) Calculator

There is no universal cutoff, but values below about 0.02 to 0.05 are usually considered well calibrated for datasets with thousands of rows. Guo et al. reported ECE above 0.10 for some uncalibrated deep networks that dropped to around 0.01 to 0.02 after temperature scaling. Always compare ECE computed with the same bin count, strategy and test set, because those settings change the number.
From the same team

Stop paying per token — route AI requests to your own GPU

Wide Area AI is a local-first AI gateway: repeated requests hit an edge cache, the rest run free on your own hardware, and the cloud is only a failover. OpenAI-compatible endpoint, free tier.

Start routing — free
VRAM CalculatorCan I Run This AI?AI API Cost CalculatorGPU for Model

Sources & References

On Calibration of Modern Neural Networks

Defines ECE and MCE with reliability diagrams and shows temperature scaling calibrates deep networks.
→Source: arXiv (Guo, Pleiss, Sun, Weinberger, ICML 2017)

sklearn.calibration.calibration_curve

Uniform and quantile binning, prob_true and prob_pred outputs, and the docs example used as a unit test.
→Source: scikit-learn documentation

Probability calibration

Explains calibration curves and why Brier score and log loss mix calibration with discrimination.
→Source: scikit-learn user guide

Explore More Tools

Explore these related tools for your next step.

ai

AI Text Cleaner: Remove Hidden Characters

Strip zero width spaces, no-break spaces, soft hyphens and other invisible Unicode from ChatGPT, Claude or Gemini text, with a count of every change and a highlighted before and after diff.

Try it now
ai

AGENTS.md Generator for AI Coding Agents

Describe your repo once and get an AGENTS.md, CLAUDE.md, Cursor rule, .cursorrules or GitHub Copilot instructions file, with stack presets for Node, Python, Go and Rust.

Try it now
ai

AI Crawler robots.txt Generator

Block GPTBot, ClaudeBot, Google-Extended and 18 other AI user agents while keeping ChatGPT, Claude and Perplexity search citations, then copy a ready robots.txt. Crawler list as of September 2026.

Try it now
ai

llms.txt Generator: Build an llms.txt From Your Sitemap

Paste a URL list or your sitemap.xml and get a ready-to-upload llms.txt that follows the llmstxt.org proposal, with sections grouped by folder and low-priority links moved to Optional.

Try it now
ai

AI Image Resolution Calculator for SDXL, FLUX and DALL-E

Pick an aspect ratio and get the native generation size for SDXL, SD 1.5, FLUX.1, SD 3.5, DALL-E 3 and gpt-image-1, plus the Midjourney --ar value, how far each is off your ratio and the upscale needed for your final size.

Try it now
ai

Confusion Matrix Calculator: F1, Precision, Recall, MCC

Enter TP, FP, FN and TN, or a 3 to 8 class matrix, to get accuracy, precision, recall, F1, F-beta, MCC, Cohen's kappa and balanced accuracy with the formula behind each number.

Try it now