Activation Function Calculator (Sigmoid, ReLU, GELU, Tanh)

Compute the value and derivative of sigmoid, tanh, ReLU, Leaky ReLU, ELU, SELU, GELU, Swish, softplus and Mish for any x or list of x values, plus softmax for a vector.

Plots the chosen function and its derivative from -6 to 6.

Last updatedHow we build & check our tools
Results will appear here once you enter your values.

What each activation function computes

An activation function is applied to each neuron's weighted sum so a network can learn nonlinear patterns. This calculator evaluates eleven common choices at any x and returns both f(x) and the derivative f'(x), the quantity backpropagation multiplies into every gradient.

  • Sigmoid 1 / (1 + e^(-x)) squashes to (0, 1). Sigmoid(1) = 0.731059.
  • Tanh squashes to (-1, 1) and is zero centered, with slope 1 at x = 0.
  • ReLU max(0, x) and Leaky ReLU, which keeps a small slope (0.01 by default) for negative inputs.
  • ELU alpha(e^x - 1) for x at or below 0, and SELU, a scaled ELU with lambda = 1.0507009873554805 and alpha = 1.6732632423543772.
  • GELU x times the standard normal CDF, used in BERT and GPT style transformers, and Swish/SiLU x times sigmoid(x).
  • Softplus ln(1 + e^x), a smooth ReLU whose derivative is exactly the sigmoid, and Mish x times tanh(softplus(x)).

Worked examples you can check

Sigmoid at x = 1. sigma(1) = 1 / (1 + e^(-1)) = 0.7310586, and its derivative is sigma(1)(1 - sigma(1)) = 0.1966119. The derivative peaks at 0.25 when x = 0, which is why deep stacks of sigmoids shrink gradients.

GELU at x = 1. The exact form is x times Phi(x) = 0.5x(1 + erf(x / sqrt(2))) = 0.8413447. Hendrycks and Gimpel also give the approximation 0.5x(1 + tanh(sqrt(2/pi)(x + 0.044715x^3))). Across x from -6 to 6 the two differ by at most about 0.00047, largest near x = -2.7, so the table shows both.

Softmax of [1, 2, 3]. The outputs are 0.0900, 0.2447 and 0.6652. Adding the same constant to every logit does not change them, so [1000, 1001, 999] gives the same three numbers in a different order. The calculator subtracts the largest logit first so that e^1000 never overflows.

All values match PyTorch float64 outputs and autograd derivatives to within 1e-7 in the unit tests.

Pitfalls: saturation, dead units and overflow

Saturation. At x = 5 the sigmoid derivative is only 0.0066, and tanh behaves similarly, so large pre-activations stall learning. Initialization schemes and normalization layers exist largely to keep inputs in the responsive range.

Dead ReLUs. ReLU's derivative is 0 for every negative input. This tool uses 0 at exactly x = 0, the same subgradient PyTorch autograd uses. A neuron whose input stays negative stops updating, which Leaky ReLU, ELU and GELU avoid by keeping a nonzero gradient on the left side.

Overflow. A naive softplus computes e^1000, which is Infinity in 64-bit floating point. Here softplus uses max(x, 0) + ln(1 + e^(-|x|)), and sigmoid switches formulas for negative x, so x = 1000 and x = -1000 return clean results such as softplus(1000) = 1000 and sigmoid(-1000) = 0 instead of NaN.

Choosing an activation function

ReLU remains a fast default for convolutional and multilayer perceptron hidden layers. GELU is the usual choice in transformer feed-forward blocks. Swish, which Ramachandran et al. found through automated search, and Mish are smooth and non-monotonic: each dips slightly below zero for negative inputs, with minimums of about -0.278 near x = -1.28 for Swish and -0.309 near x = -1.19 for Mish, while GELU bottoms out near -0.170 at x = -0.75.

SELU is designed to self-normalize when weights use LeCun normal initialization, as described by Klambauer et al., so it is usually paired with that initializer instead of batch normalization. Sigmoid belongs in output layers for binary probabilities, and softmax for mutually exclusive classes. Use the list mode to tabulate a function over a grid of inputs, and the plot to see where the derivative vanishes.

Frequently Asked Questions

Common questions about the Activation Function Calculator (Sigmoid, ReLU, GELU, Tanh)

The derivative is sigma(x) times (1 - sigma(x)). At x = 0 sigmoid is 0.5, so the derivative reaches its maximum of 0.25. At x = 1 the value is 0.7310586 and the derivative is 0.1966119. The derivative shrinks quickly as |x| grows, falling to about 0.0066 at x = 5, which contributes to vanishing gradients in deep sigmoid networks.
From the same team

Stop paying per token — route AI requests to your own GPU

Wide Area AI is a local-first AI gateway: repeated requests hit an edge cache, the rest run free on your own hardware, and the cloud is only a failover. OpenAI-compatible endpoint, free tier.

Start routing — free
VRAM CalculatorCan I Run This AI?AI API Cost CalculatorGPU for Model

Sources & References

Gaussian Error Linear Units (GELUs)

Defines GELU as x times Phi(x), the tanh approximation with the 0.044715 coefficient, and the SiLU x times sigmoid(x).
→Source: arXiv (Hendrycks and Gimpel, 2016)

Self-Normalizing Neural Networks

Introduces SELU and derives the alpha and lambda constants for self-normalizing networks.
→Source: arXiv (Klambauer et al., 2017)

SELU

Full precision SELU constants used by the calculator; PyTorch float64 outputs and autograd derivatives were the reference values in the unit tests.
→Source: PyTorch documentation

Searching for Activation Functions

Proposes Swish, x times sigmoid(beta x), found by automated search.
→Source: arXiv (Ramachandran, Zoph and Le, 2017)

Mish: A Self Regularized Non-Monotonic Activation Function

Defines Mish as x times tanh(softplus(x)).
→Source: arXiv (Misra, 2019)

Explore More Tools

Explore these related tools for your next step.

ai

LLM Token Usage Calculator

Calculate token usage and associated costs for LLM prompts.

Try it now
ai

AI Embedding Cost Calculator

Estimate the total token count and API cost to embed a document corpus using OpenAI, Cohere, or Voyage embedding models.

Try it now
ai

Neural Network Playground

Train a real neural network in your browser — no code, no installs. Pick a sample dataset (Titanic survival, vehicle classifier, handwritten digits) or upload your own CSV, watch the network learn with live visuals, then ask it questions and export the model as Python/Keras code.

Try it now
ai

LLM Token Counter

Count tokens in text for GPT, Claude, Gemini, and open models like Llama, Gemma, and Qwen — or search any model on Hugging Face. Estimate API costs, check context window fit, and optimize prompts

Try it now
ai

LLM VRAM Calculator

Calculate how much VRAM any LLM needs to run locally — pick a model (Llama, Gemma, Qwen, DeepSeek, or search Hugging Face), choose a quantization and context size, and see which GPUs it fits on, including multi-GPU setups.

Try it now
ai

LLM Inference Speed Calculator

Estimate LLM tokens per second from memory bandwidth, model size, quantization, and context window. Compare generation speed across GPUs (RTX 4090, 5090, A100, H100, Apple Silicon) and understand the memory-bandwidth bottleneck.

Try it now