This is a technique used in Neural network training to make learning more stable.
Some notes
Common usages of Batch Norm keeps some batch statistics of the mean and standard deviation. These are computed during training and then used during inference. This makes it possible to do inference on single samples instead of using entire batches with the same random distribution as the training data.
I once had a problem where I had to train and do inference on entire batches. I wanted the network to learn things while taking into account the entire batch, so I didn’t want to use the batch statistics that were calculated during training. I ended up writing a custom batch norm layer that didn’t keep this statistic and only used the current batch for both training and inference.
Usage with frameworks
Keras
You can use Batch Normalization in Keras like this.
keras.layers.BatchNormalization()
Documentation of the Keras batch norm layer: https://keras.io/api/layers/normalization_layers/batch_normalization/
PyTorch
torch.nn.BatchNorm1d(out_dim)
Torch documentation for batch norm: https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html
