Activation Functions in Neural Networks: ReLU, Sigmoid, GELU and More

An activation function is the small nonlinear function a neuron applies to its weighted input before passing the result on: a = f(w · x + b). Without it, a neural network of any depth collapses into a single linear model. The most common choices today are ReLU for convolutional and fully connected networks, GELU and SwiGLU inside Transformers and large language models, sigmoid for binary outputs and softmax for multi-class outputs.

This guide explains why activation functions are needed, compares the main types with formulas, ranges and derivatives, shows what causes the vanishing gradient problem, and gives NumPy and PyTorch code for each function.

What Is an Activation Function?

An artificial neuron does two things. First it computes a weighted sum of its inputs plus a bias: z = w₁x₁ + w₂x₂ + … + b. Then it passes that number through an activation function f, and the result a = f(z) becomes an input to the next layer. The weighted sum decides how strongly the neuron responds; the activation function decides the shape of that response.

The earliest neurons, such as Rosenblatt’s perceptron, used a hard threshold: output 1 if z is above zero and 0 otherwise. Modern networks use smooth or piecewise-linear functions instead, because training with gradient descent needs a derivative that tells each weight which way to move. A step function has a derivative of zero almost everywhere, so it gives the optimizer nothing to work with.

Why Neural Networks Need Nonlinearity

Suppose every layer used the identity f(z) = z. A two-layer network would compute W₂(W₁x + b₁) + b₂, which simplifies to Wx + b with W = W₂W₁. A hundred linear layers simplify the same way. However deep the network, it can only draw straight decision boundaries and fit linear relationships — no better than linear regression.

Putting a nonlinear function between layers breaks this collapse. With enough neurons, a network with a nonlinear activation can approximate essentially any continuous function; this is the universal approximation theorem. Depth then lets the network build complex features out of simpler ones: edges into textures, textures into object parts.

Types of Activation Functions Compared

Φ(x) is the cumulative distribution function of the standard normal distribution, and φ(x) is its density.

Sigmoid

The sigmoid (logistic) function squeezes any real number into the range from 0 to 1: σ(x) = 1 / (1 + e^(−x)). Large positive inputs give values close to 1, large negative inputs give values close to 0, and σ(0) = 0.5. Because the output looks like a probability, sigmoid is the standard choice for the output of a binary classifier and for multi-label problems where each class is a separate yes-or-no decision. It is also used for the gates inside LSTM and GRU cells, where a value between 0 and 1 decides how much information to let through.

Sigmoid is a poor choice for hidden layers of deep networks. Its derivative σ(x)(1 − σ(x)) peaks at only 0.25 at x = 0 and falls to about 0.0066 at x = 5. When inputs drift away from zero, the neuron saturates: its output barely changes, and almost no gradient flows back through it. Its outputs are also always positive, which makes weight updates zigzag and slows convergence.

Tanh

The hyperbolic tangent is a rescaled sigmoid: tanh(x) = 2σ(2x) − 1. It maps inputs to the range from −1 to 1 and is centered at zero, so its outputs average out closer to zero and gradient updates are better behaved than with sigmoid. Its derivative 1 − tanh²(x) reaches 1 at the origin, four times the peak of sigmoid.

Tanh still saturates: at x = 5 its derivative is about 0.0002. It remains common in recurrent networks — the hidden state of an LSTM is passed through tanh — but feed-forward and convolutional networks moved to ReLU long ago.

The Vanishing Gradient Problem

Backpropagation computes the gradient for an early layer by multiplying together the derivatives of every layer after it. If each of those derivatives is small, the product shrinks exponentially with depth. With sigmoid, even in the best case the derivative is 0.25, so after 10 layers the gradient is multiplied by at most 0.25¹⁰ ≈ 0.00000095. The first layers receive almost no learning signal and stay close to their random initialization. This is the vanishing gradient problem, and it is the main reason deep networks were so hard to train before 2010.

Three changes largely solved it: activation functions whose derivative is 1 over a wide range (ReLU and its successors), better weight initialization (Xavier/Glorot for tanh, He initialization for ReLU), and architectural tricks such as normalization layers and the residual connections of ResNet. The opposite failure, exploding gradients, happens when the product grows instead; gradient clipping is the usual fix.

ReLU and Its Variants

ReLU

The rectified linear unit is simply ReLU(x) = max(0, x). For positive inputs it passes the value through unchanged, with a derivative of exactly 1, so gradients do not shrink as they move back through active neurons. For negative inputs it outputs 0. It is also extremely cheap to compute — a comparison instead of an exponential. ReLU became the default after AlexNet showed in 2012 that it trains deep convolutional networks several times faster than tanh, and it is used throughout classic architectures such as VGG16 and ResNet.

Its weakness is the dying ReLU problem. If a neuron’s weights shift so that its input is negative for every example — for instance after a large update with a high learning rate — it outputs zero everywhere, its gradient is zero everywhere, and it never recovers. In practice a moderate learning rate, He initialization and normalization layers keep the fraction of dead neurons small.

Leaky ReLU, PReLU and ELU

• Leaky ReLU keeps a small slope for negative inputs, αx with α around 0.01, so the gradient never becomes exactly zero.
• PReLU (parametric ReLU) makes α a learnable parameter instead of a fixed constant.
• ELU uses a smooth exponential curve for negative inputs, α(e^x − 1), which pushes the mean activation toward zero. SELU is a scaled version designed to keep activations normalized in deep fully connected networks.

GELU, SiLU (Swish) and SwiGLU

Modern Transformers mostly use smooth relatives of ReLU. They behave like ReLU for large positive and large negative inputs, but curve gently near zero and let small negative values through, which tends to give slightly better accuracy in large models.

• GELU (Gaussian Error Linear Unit), x · Φ(x), weights each input by the probability that a standard normal variable is smaller than it. It was proposed by Hendrycks and Gimpel in 2016 and is the activation in BERT, GPT-2 and many vision Transformers. A fast approximation, 0.5x(1 + tanh(√(2/π)(x + 0.044715x³))), is often used in practice.
• SiLU, also called Swish, is x · σ(x). It looks very similar to GELU and is used in EfficientNet and many diffusion models.
• SwiGLU is not a single function but a gated feed-forward block: SwiGLU(x) = SiLU(xW) ⊙ (xV), where one linear projection is passed through SiLU and multiplied element-wise by a second projection. Introduced by Noam Shazeer in 2020 in “GLU Variants Improve Transformer”, it is used in the feed-forward layers of PaLM, LLaMA and most open large language models that followed.

Softmax

Softmax works on a whole vector rather than a single number. It exponentiates every score and divides by the sum, softmax(z)ᵢ = e^(zᵢ) / Σⱼ e^(zⱼ), turning arbitrary scores (logits) into positive values that add up to 1. For the scores [2.0, 1.0, 0.1] it gives approximately [0.659, 0.242, 0.099]. That makes it the standard output layer for multi-class classification, where exactly one class is correct, and the core of the attention mechanism in Transformers.

Two practical details matter. Subtract the maximum score before exponentiating to avoid overflow; the result is mathematically the same. And in PyTorch, do not put a softmax layer before nn.CrossEntropyLoss: the loss already applies log-softmax to raw logits internally.

How to Choose an Activation Function

Change the activation function last. Learning rate, initialization and normalization usually matter far more, and the differences between ReLU, GELU and SiLU are small for most models outside very large Transformers.

Activation Functions in NumPy and PyTorch

Every function in this guide fits in a line or two of NumPy:

import numpy as np
from math import erf, sqrt

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def tanh(x):
    return np.tanh(x)

def relu(x):
    return np.maximum(0, x)

def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

def gelu(x):
    # exact GELU: x * Phi(x), where Phi is the standard normal CDF
    return x * 0.5 * (1 + np.vectorize(erf)(x / sqrt(2)))

def silu(x):
    # also called Swish
    return x * sigmoid(x)

def softmax(z):
    z = z - np.max(z)          # subtract the max for numerical stability
    e = np.exp(z)
    return e / e.sum()

x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
for f in [sigmoid, tanh, relu, leaky_relu, gelu, silu]:
    print(f"{f.__name__:>10}: {np.round(f(x), 3)}")

print("   softmax:", np.round(softmax(np.array([2.0, 1.0, 0.1])), 3))

Output:

   sigmoid: [0.119 0.269 0.5   0.731 0.881]
      tanh: [-0.964 -0.762  0.     0.762  0.964]
      relu: [0. 0. 0. 1. 2.]
leaky_relu: [-0.02 -0.01  0.    1.    2.  ]
      gelu: [-0.046 -0.159  0.     0.841  1.954]
      silu: [-0.238 -0.269  0.     0.731  1.762]
   softmax: [0.659 0.242 0.099]

In PyTorch every activation exists both as a layer in torch.nn (nn.ReLU(), nn.GELU(), nn.SiLU(), nn.Sigmoid(), nn.Tanh(), nn.LeakyReLU(0.01)) and as a function in torch.nn.functional. A typical classifier puts the activation after each hidden linear layer and leaves the output as raw logits:

import torch
import torch.nn as nn
import torch.nn.functional as F

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
print(F.relu(x))
print(F.gelu(x))
print(F.silu(x))

# A small MLP with GELU in the hidden layers
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.GELU(),
    nn.Linear(256, 128),
    nn.GELU(),
    nn.Linear(128, 10),   # raw logits: no softmax here
)

logits = model(torch.randn(32, 784))
labels = torch.randint(0, 10, (32,))
loss = nn.CrossEntropyLoss()(logits, labels)   # applies log-softmax internally
probs = torch.softmax(logits, dim=1)           # only for inference / reporting
print(logits.shape, probs.sum(dim=1)[:3])

Output (PyTorch 2.14; the model weights are random, so only the shapes matter in the last line):

tensor([0., 0., 0., 1., 2.])
tensor([-0.0455, -0.1587,  0.0000,  0.8413,  1.9545])
tensor([-0.2384, -0.2689,  0.0000,  0.7311,  1.7616])
torch.Size([32, 10]) tensor([1.0000, 1.0000, 1.0000], grad_fn=<SliceBackward0>)

FAQ

What is the most commonly used activation function?

ReLU is the most widely used activation for hidden layers in convolutional and fully connected networks. In Transformers and large language models, GELU and SwiGLU have largely taken its place.

Why is ReLU better than sigmoid?

For hidden layers, ReLU does not saturate for positive inputs, so its gradient stays at 1 instead of shrinking, and it is much cheaper to compute. Sigmoid remains the better choice for a binary output layer, where you need a value between 0 and 1.

What is the difference between sigmoid and softmax?

Sigmoid treats each output independently and maps it to a value between 0 and 1, so several classes can be “on” at once. Softmax normalizes a whole vector so the values sum to 1, which fits problems where exactly one class is correct.

Does the output layer need an activation function?

For regression, usually not: a linear output can take any value. For classification, the output passes through sigmoid or softmax, although in PyTorch you normally feed raw logits to a loss function that applies it internally.

What activation function does GPT use?

GPT-2 and GPT-3 use GELU in their feed-forward layers. Many newer open models, including the LLaMA family, use SwiGLU.

See also: our breakdowns of AlexNet, the network that made ReLU the default, and ResNet, which made very deep networks trainable, plus the rest of our popular neural network architectures.