KL Divergence and JS Divergence Calculator

Paste two discrete distributions to get KL(P||Q), KL(Q||P), Jensen-Shannon divergence and distance, cross-entropy and entropy in bits, nats or bans, with a per-element contribution table.

Results match scipy.stats.entropy and scipy.spatial.distance.jensenshannon.

Last updatedHow we build & check our tools
Results will appear here once you enter your values.

How KL and Jensen-Shannon divergence are computed

KL divergence. For discrete distributions P and Q over the same outcomes, KL(P||Q) = sum of p(i) times log(p(i) / q(i)). It measures the extra information, in bits for log base 2 or nats for the natural log, you pay when you encode data drawn from P with a code built for Q. It is always at least 0 and equals 0 only when P and Q are identical.

Worked example. Wikipedia's example uses P = (0.36, 0.48, 0.16), a binomial with N = 2 and p = 0.4, and Q uniform at 1/3 each. The terms are 0.36 ln(1.08) = 0.02771, 0.48 ln(1.44) = 0.17503 and 0.16 ln(0.48) = -0.11744, so KL(P||Q) = 0.0852996 nats. Reversed, KL(Q||P) = 0.097455 nats. The same inputs give 0.123061 bits, and scipy.stats.entropy returns identical values in our unit tests.

Jensen-Shannon. Build the mixture M = (P + Q) / 2, then JSD = 0.5 KL(P||M) + 0.5 KL(Q||M). With log base 2 it always falls between 0 and 1, and its square root, the JS distance, is a true metric. For the example above the JS distance is 0.180008 in base 2, which matches scipy.spatial.distance.jensenshannon.

Reading the results

Direction matters. KL(P||Q) and KL(Q||P) are usually different, so KL is not a distance. Put the true or reference distribution in P and the model in Q when you want the forward KL used in maximum likelihood training. Cross-entropy H(P, Q) equals H(P) + KL(P||Q), which is why minimizing cross-entropy loss against fixed labels also minimizes KL.

Per-element table. Each row shows p log(p/q), q log(q/p) and the JS term for that outcome. Individual KL terms can be negative, as the third term in the example shows, but they always sum to a non-negative total. The rows with the largest values tell you where Q misplaces the most probability mass.

Choosing a measure. Use JS divergence or JS distance to compare two histograms symmetrically, for example feature drift between a training set and live traffic. Use KL when one side is clearly the reference, such as a teacher model in distillation or a prior in a variational autoencoder.

Zeros, infinite values and bad input

Zero in P. An outcome with p = 0 contributes nothing, using the convention 0 log 0 = 0. So P = (0.5, 0.5, 0) against Q = (0.25, 0.25, 0.5) gives exactly 1 bit.

Zero in Q. If q = 0 where p is positive, KL(P||Q) is infinite, because Q says an observed outcome is impossible. The calculator shows this as the text infinite (Q has zero where P > 0) rather than a number. JS divergence stays finite in that case, which is one reason it is popular for comparing empirical histograms. If you need a finite KL, add a small smoothing count such as 1 to every bin before comparing.

Validation. Lists must have the same length, values cannot be negative, and a list of all zeros is rejected. Lists that do not sum to 1, such as raw counts 10 20 30 40, are divided by their own total and a note says so, the same thing scipy does.

Units and log base

Converting units. Bits use log base 2, nats use the natural log and bans (also called hartleys) use base 10. One nat equals 1 / ln 2 = 1.4427 bits, and one ban equals log2(10) = 3.3219 bits, so you can convert any result by multiplying. The maximum JS divergence is 1 bit, 0.6931 nats or 0.3010 bans.

Which to pick. PyTorch and TensorFlow loss functions use nats, information theory texts usually use bits, and scipy.stats.entropy defaults to base e. Keep the base consistent when comparing numbers across papers or tools.

Frequently Asked Questions

Common questions about the KL Divergence and JS Divergence Calculator

KL divergence measures how much information is lost when Q is used to approximate P, and it is asymmetric and can be infinite. Jensen-Shannon divergence averages the KL of each distribution against their 50/50 mixture, so it is symmetric, always finite and bounded by 1 bit. Its square root, the JS distance, satisfies the triangle inequality.
From the same team

Stop paying per token — route AI requests to your own GPU

Wide Area AI is a local-first AI gateway: repeated requests hit an edge cache, the rest run free on your own hardware, and the cloud is only a failover. OpenAI-compatible endpoint, free tier.

Start routing — free
VRAM CalculatorCan I Run This AI?AI API Cost CalculatorGPU for Model

Sources & References

Kullback-Leibler divergence

Definition and the basic example P = (0.36, 0.48, 0.16) vs uniform Q giving 0.0852996 and 0.097455 nats.
→Source: Wikipedia

Jensen-Shannon divergence

Mixture definition M = (P+Q)/2, the 0 to 1 bound in base 2 and that its square root is a metric.
→Source: Wikipedia

scipy.stats.entropy

Reference implementation of KL divergence with automatic normalization and default base e.
→Source: SciPy documentation

scipy.spatial.distance.jensenshannon

Reference implementation of JS distance as the square root of JS divergence, with normalization.
→Source: SciPy documentation

Explore More Tools

Explore these related tools for your next step.

ai

LLM Token Usage Calculator

Calculate token usage and associated costs for LLM prompts.

Try it now
ai

AI Embedding Cost Calculator

Estimate the total token count and API cost to embed a document corpus using OpenAI, Cohere, or Voyage embedding models.

Try it now
ai

Neural Network Playground

Train a real neural network in your browser — no code, no installs. Pick a sample dataset (Titanic survival, vehicle classifier, handwritten digits) or upload your own CSV, watch the network learn with live visuals, then ask it questions and export the model as Python/Keras code.

Try it now
ai

LLM Token Counter

Count tokens in text for GPT, Claude, Gemini, and open models like Llama, Gemma, and Qwen — or search any model on Hugging Face. Estimate API costs, check context window fit, and optimize prompts

Try it now
ai

LLM VRAM Calculator

Calculate how much VRAM any LLM needs to run locally — pick a model (Llama, Gemma, Qwen, DeepSeek, or search Hugging Face), choose a quantization and context size, and see which GPUs it fits on, including multi-GPU setups.

Try it now
ai

LLM Inference Speed Calculator

Estimate LLM tokens per second from memory bandwidth, model size, quantization, and context window. Compare generation speed across GPUs (RTX 4090, 5090, A100, H100, Apple Silicon) and understand the memory-bandwidth bottleneck.

Try it now