If you have trained a neural network, you have written something like model.fit(X, y, epochs=10, batch_size=32) and wondered what exactly the numbers mean and how they relate to the “steps” or “iterations” in the training log. The three terms look similar, but they measure different things. An epoch is one complete pass of the whole training dataset through the model. The batch size is how many training examples the model looks at before it updates its weights once. An iteration, or step, is one such update, so the number of iterations in an epoch equals the number of batches.
This guide explains the three terms through gradient descent, shows how to calculate them, and runs two experiments in PyTorch: one shows how the number of epochs leads from underfitting to overfitting and how early stopping picks the right moment, the other shows what the batch size changes in training.
Key takeaways
- Epoch: the model has seen every training example once.
- Batch size: the number of examples used to compute one gradient and make one weight update.
- Iteration (step): one weight update on one batch.
iterations per epoch = ceil(training examples / batch size). - Example: 2,000 examples with a batch size of 500 give 4 iterations per epoch; 10 epochs give 40 weight updates.
- Choosing them: use early stopping on a validation set instead of guessing the number of epochs; start with a batch size of 32 to 256 and adjust it to your memory and learning rate.
Epoch, Batch Size and Iteration at a Glance
| Term | Meaning | Example: 2,000 examples, batch size 500, 10 epochs |
|---|---|---|
| Epoch | One full pass over the training dataset | 10 epochs |
| Batch | A subset of the training data processed together | Each epoch is split into 4 batches |
| Batch size | The number of examples in one batch | 500 |
| Iteration (step) | One forward and backward pass on one batch, followed by one weight update | 4 per epoch, 40 in total |
Note that the batch size and the number of batches are two different numbers: the batch size is how big each batch is, the number of batches is how many of them make up an epoch. For one epoch, the number of batches equals the number of iterations.
Why We Need Them: Gradient Descent
To see why training is organized into epochs and batches, recall how neural networks learn. Gradient descent is an iterative optimization algorithm: it computes the gradient of the loss function with respect to the model’s weights, which shows the direction in which the loss grows fastest, and takes a small step in the opposite direction. The size of the step is controlled by the learning rate. Repeating this many times moves the model to a point where the loss, also called the cost, is low:
The animation shows exactly this: each step moves the parameter towards the minimum of the cost function, the steps get smaller as the slope flattens, and the fitted regression line on the right gets closer to the data. One step is never enough; training needs many of them. The question is how much data to use to compute each step:
| Variant | Batch size | Iterations per epoch | Properties |
|---|---|---|---|
| Batch gradient descent | The whole dataset | 1 | Exact gradient, but slow updates and the whole dataset must fit in memory |
| Stochastic gradient descent (SGD) | 1 | Number of examples | Very frequent, very noisy updates; cannot use the parallelism of GPUs |
| Mini-batch gradient descent | Usually 16 to a few thousand | Examples / batch size | The standard in deep learning: fits in memory, uses GPUs well, reasonably accurate gradients |
Real datasets are far too large to push through a network at once, so the data is divided into batches, loaded one after another, and the weights are updated after each one. In deep learning, “SGD” in practice almost always means mini-batch gradient descent, and the optimizers you use, such as SGD with momentum or Adam, all work on mini-batches. The gradients themselves are computed with backpropagation.
What Is an Epoch in Machine Learning?
An epoch is complete when the entire training dataset has passed forward and backward through the neural network once. Because one epoch is usually too big to process at once, it is divided into batches, and the epoch consists of as many iterations as there are batches. Before each epoch, the training examples are usually shuffled, so that the batches are different every time.
Why use more than one epoch? At first it may seem strange that passing the whole dataset through the network once is not enough. But gradient descent is an iterative process, and one pass gives only a limited number of small steps, typically too few to reach a good minimum. As the number of epochs grows, the weights are updated more times, and the model moves from underfitting, where it has not yet learned the patterns in the data, to an optimal fit. If training continues too long, it moves on to overfitting: the model fits the training examples, including their noise, so closely that it loses the ability to generalize to new data.
How Many Epochs? Overfitting and Early Stopping
There is no universal right number of epochs: it depends on the size and diversity of the dataset, the model and the learning rate. A dataset with only black cats needs fewer epochs to learn than one with every kind of cat. The reliable way to decide is to measure: keep a validation set that the model does not train on, check its loss after every epoch, and stop when it stops improving. This is called early stopping.
The experiment below trains a small multilayer perceptron on just 300 handwritten digits, so that overfitting shows up quickly, and evaluates it on 500 other digits after every epoch. It also counts iterations: with 300 examples and a batch size of 32, each epoch has 10 of them:
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from torch.utils.data import DataLoader, TensorDataset
torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16, dtype=torch.float32)
y = torch.tensor(y)
# a deliberately small training set, so that overfitting shows up quickly
X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=300, test_size=500, random_state=0, stratify=y)
loader = DataLoader(TensorDataset(X_train, y_train), batch_size=32, shuffle=True)
model = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
best_val, best_epoch, patience, iteration = float("inf"), 0, 15, 0
for epoch in range(1, 201): # one epoch = one pass over all 300 training examples
model.train()
for xb, yb in loader: # one iteration = one batch of 32 = one weight update
loss = loss_fn(model(xb), yb)
opt.zero_grad()
loss.backward()
opt.step()
iteration += 1
model.eval()
with torch.no_grad():
train_loss = loss_fn(model(X_train), y_train).item()
val_loss = loss_fn(model(X_val), y_val).item()
val_acc = (model(X_val).argmax(1) == y_val).float().mean().item()
if epoch in (1, 5, 10, 20, 50, 100, 200) or epoch == best_epoch + patience:
print(f"epoch {epoch:3d} (iteration {iteration:4d}): train loss {train_loss:.4f}, "
f"val loss {val_loss:.4f}, val accuracy {val_acc:.3f}")
if val_loss < best_val:
best_val, best_epoch = val_loss, epoch
elif epoch == best_epoch + patience:
print(f"early stopping would stop here; best epoch was {best_epoch} (val loss {best_val:.4f})")epoch 1 (iteration 10): train loss 2.1173, val loss 2.1250, val accuracy 0.642
epoch 5 (iteration 50): train loss 0.5194, val loss 0.5886, val accuracy 0.880
epoch 10 (iteration 100): train loss 0.1504, val loss 0.2733, val accuracy 0.940
epoch 20 (iteration 200): train loss 0.0475, val loss 0.2133, val accuracy 0.942
epoch 41 (iteration 410): train loss 0.0101, val loss 0.2232, val accuracy 0.950
early stopping would stop here; best epoch was 26 (val loss 0.2053)
epoch 50 (iteration 500): train loss 0.0061, val loss 0.2339, val accuracy 0.948
epoch 100 (iteration 1000): train loss 0.0013, val loss 0.2816, val accuracy 0.950
epoch 200 (iteration 2000): train loss 0.0003, val loss 0.3338, val accuracy 0.948
For the first 10 to 20 epochs both losses fall quickly: the model is underfitting and every epoch helps. The validation loss reaches its minimum at epoch 26. After that, the training loss keeps falling towards zero, but the validation loss rises by more than 60% by epoch 200: the model is memorizing the training examples and becoming overconfident on new ones. Accuracy hardly changes here, but the growing loss shows that its predictions are getting worse calibrated. With a patience of 15 epochs, early stopping would end training at epoch 41 and keep the weights from epoch 26, saving about 80% of the training time.
Typical numbers of epochs in practice:
- Small datasets and models, such as tabular data or the example above: tens to hundreds of epochs, controlled by early stopping.
- Image classification from scratch, for example ResNets on ImageNet: roughly 90 to a few hundred epochs, with a learning rate schedule.
- Fine-tuning a pretrained model: often 1 to 5 epochs, because the model already knows most of what it needs.
- Large language models: pretraining datasets are so large that models usually see most of the data only once, about one epoch, and training length is measured in tokens or steps instead.
Besides early stopping, overfitting is fought with more data, data augmentation, dropout, weight decay and smaller models.
What Is Batch Size?
The batch size is the number of training examples in one batch, and therefore the number of examples used to compute one gradient. It works like splitting a long article into sections: the content stays the same, but it is easier to process piece by piece. The gradient of a batch is an estimate of the gradient over the whole dataset; the bigger the batch, the more accurate the estimate, and the more memory and computation each step needs.
Choosing the batch size is a trade-off:
- Memory. The batch has to fit into GPU memory together with the model and its activations. This is often the hard upper limit. When a larger batch does not fit, gradient accumulation sums the gradients of several smaller batches before making one update, which gives almost the same result as a larger batch.
- Speed per epoch. GPUs process many examples in parallel, so larger batches finish an epoch faster, up to the point where the hardware is saturated.
- Number of updates. Larger batches mean fewer weight updates per epoch. With the same number of epochs and the same learning rate, a model trained with large batches learns less.
- Noise and generalization. Small batches give noisy gradients, which acts as a regularizer and can help the model find solutions that generalize well. Nitish Keskar and colleagues showed in 2017 that very large batches tend to converge to “sharp” minima that generalize worse.
Batch size and learning rate are linked. A common rule of thumb, the linear scaling rule from the 2017 paper “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour” by Priya Goyal and colleagues at Facebook, says: when you multiply the batch size by k, multiply the learning rate by k too, and warm it up over the first few epochs. With this rule they trained a ResNet-50 on ImageNet with batches of 8,192 images without losing accuracy. Powers of two such as 32, 64 or 128 are traditional, but there is nothing magical about them.
How Batch Size Affects Training
Here the same network is trained for exactly 20 epochs on 1,257 digits with six different batch sizes, from a single example to the whole training set, keeping everything else the same:
import time
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from torch.utils.data import DataLoader, TensorDataset
X, y = load_digits(return_X_y=True)
X, y = torch.tensor(X / 16, dtype=torch.float32), torch.tensor(y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
def run(batch_size, epochs=20):
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loader = DataLoader(TensorDataset(X_train, y_train), batch_size=batch_size, shuffle=True)
start = time.perf_counter()
for epoch in range(epochs):
for xb, yb in loader:
loss = nn.functional.cross_entropy(model(xb), yb)
opt.zero_grad()
loss.backward()
opt.step()
seconds = time.perf_counter() - start
with torch.no_grad():
acc = (model(X_test).argmax(1) == y_test).float().mean().item()
return len(loader) * epochs, seconds, acc
print(f"{len(X_train)} training examples, 20 epochs, Adam with learning rate 0.001")
for batch_size in (1, 8, 32, 128, 512, len(X_train)):
updates, seconds, acc = run(batch_size)
print(f"batch size {batch_size:4d}: {updates:5d} updates, {seconds:5.1f} s, test accuracy {acc:.3f}")1257 training examples, 20 epochs, Adam with learning rate 0.001
batch size 1: 25140 updates, 16.6 s, test accuracy 0.976
batch size 8: 3160 updates, 2.4 s, test accuracy 0.978
batch size 32: 800 updates, 0.6 s, test accuracy 0.963
batch size 128: 200 updates, 0.3 s, test accuracy 0.939
batch size 512: 60 updates, 0.2 s, test accuracy 0.885
batch size 1257: 20 updates, 0.4 s, test accuracy 0.793
Twenty epochs mean very different amounts of training. With a batch size of 1, the model makes 25,140 updates; with the full batch, only 20, and it is still far from trained, with 79% accuracy. Small batches reach about 98% but take much longer: processing examples one at a time is almost 30 times slower than in batches of 32, even on a CPU. The times were measured on a CPU and will differ on your machine, but the pattern is general. Large batches are not worse in themselves: given more epochs or a larger learning rate, they catch up. The lesson is that the number of epochs alone does not tell you how much a model has been trained; the number of iterations does.
A practical recipe: start with a batch size between 32 and 256, the largest that fits comfortably in memory, tune the learning rate for it, and if you change the batch size later, scale the learning rate with it.
What Is an Iteration?
An iteration, also called a step or an update, is one pass of one batch forward and backward through the network followed by one update of the weights. The number of iterations needed to complete one epoch is:
iterations per epoch = ceil(number of training examples / batch size)
total iterations = epochs × iterations per epoch
For example, if we have 2,000 training examples and divide them into batches of 500, one epoch takes 4 iterations. If the dataset does not divide evenly, the last batch is smaller, unless you drop it, for example with drop_last=True in PyTorch. In PyTorch, len(loader) gives the number of iterations per epoch:
import math
import torch
from torch.utils.data import DataLoader, TensorDataset
dataset = TensorDataset(torch.randn(2000, 10), torch.randint(0, 2, (2000,))) # 2,000 training examples
epochs = 10
for batch_size in (500, 64, 1):
loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
iterations = len(loader) # batches per epoch = ceil(2000 / batch_size)
print(f"batch size {batch_size:3d}: {iterations:4d} iterations per epoch, "
f"{iterations * epochs:5d} weight updates in {epochs} epochs")
loader = DataLoader(dataset, batch_size=64, shuffle=True)
sizes = [len(x) for x, y in loader]
print("batch sizes in one epoch:", sizes[:3], "...", sizes[-2:], "| ceil(2000 / 64) =", math.ceil(2000 / 64))batch size 500: 4 iterations per epoch, 40 weight updates in 10 epochs
batch size 64: 32 iterations per epoch, 320 weight updates in 10 epochs
batch size 1: 2000 iterations per epoch, 20000 weight updates in 10 epochs
batch sizes in one epoch: [64, 64, 64] ... [64, 16] | ceil(2000 / 64) = 32With a batch size of 64, the epoch has 31 full batches and a last one of 16 examples, 32 iterations in total. In Keras, the same quantity is called steps_per_epoch; when you train on an endless data stream, you set it yourself to define how many steps count as an epoch. Learning rate schedules, logging and checkpoints are often defined in steps rather than epochs, especially for large models.
Epoch vs Batch vs Iteration
| Epoch | Batch size | Iteration | |
|---|---|---|---|
| What it counts | Passes over the whole dataset | Examples per update | Weight updates |
| Set by | You, or early stopping | You, limited by memory | Follows from the other two |
| Too small | Underfitting | Slow, noisy training | Model not trained enough |
| Too large | Overfitting, wasted compute | Out of memory, fewer updates, possibly worse generalization | Overfitting, wasted compute |
| In Keras | epochs |
batch_size |
steps_per_epoch |
| In PyTorch | Outer training loop | DataLoader(batch_size=...) |
Inner loop over the DataLoader |
In short: the batch size decides how many examples go into one step, an iteration is one step, and an epoch is as many steps as it takes to see all the data once.
FAQ
What is an epoch in machine learning?
An epoch is one complete pass of the entire training dataset through the model. If a model is trained for 10 epochs, it has seen every training example 10 times.
What is the difference between an epoch and a batch?
A batch is a subset of the training data used for one weight update; an epoch is a full pass over all the data, which consists of all the batches. With 1,000 examples and a batch size of 100, one epoch contains 10 batches.
What is the difference between an epoch and an iteration?
An iteration is one weight update on one batch. An epoch is completed after as many iterations as there are batches in the dataset. Iterations count updates, epochs count passes over the data.
How many epochs should I train for?
There is no fixed number. Monitor the loss on a validation set and use early stopping: stop when the validation loss has not improved for a number of epochs, and keep the best weights. Small datasets often need tens to hundreds of epochs; fine-tuning pretrained models usually needs only a few.
What is a good batch size?
Values between 32 and 256 are a common starting point. Use the largest batch that fits in memory without hurting results, and when you change it, adjust the learning rate proportionally. Very small batches are slow; very large ones need more epochs or a larger learning rate and may generalize slightly worse.
Does a larger batch size train faster?
A larger batch processes an epoch faster on a GPU, but it makes fewer updates per epoch, so it may need more epochs to reach the same accuracy. Whether total training time goes down depends on the hardware and on adjusting the learning rate.
Can the batch size be 1?
Yes. This is stochastic gradient descent in its original form. It works, but it is slow because it does not use parallel hardware, and its updates are very noisy.
See also: our guides to gradient descent, which explains the optimization behind epochs and batches, and to backpropagation, which computes the gradient in every iteration.
