27 Dec 2016
Tensorflow Guide: Batch Normalization
Update [11-21-2017]: Please see this code snippet for my current preferred implementation.
I recently made the switch to TensorFlow and am very happy with how easy it was to get things done using this awesome library. Tensorflow has come a long way since I first experimented with it in 2015, and I am happy to be back.
Since I am getting myself re-acquainted with TensorFlow, I decided that I should write a post about how to do batch normalization in TensorFlow. It’s kind of weird that batch normalization still presents such a challenge for new TensorFlow users, especially since TensorFlow comes with invaluable functions like
tf.nn.batch_normalization, and even
tf.contrib.layers.batch_norm. One would think that using batch normalization in TensorFlow will be a cinch. But alas, confusion still crops up from time to time, and the devil really lies in the details.
Batch Normalization The Easy Way
Perhaps the easiest way to use batch normalization would be to simply use the
tf.contrib.layers.batch_norm layer. So let’s give that a go! Let’s get some imports and data loading out of the way first.
Next, we define our typical
fully-connected + batch normalization + nonlinearity set-up
One thing that might stand out is the
phase term. We are going to use as a placeholder for a boolean which we will insert into
feed_dict. It will serve as a binary indicator for whether we are in training
phase=True or testing
phase=False mode. Recall that batch normalization has distinct behaviors during training verus test time:
Normalize layer activations according to mini-batch statistics.
During the training step, update population statistics approximation via moving average of mini-batch statistics.
Normalize layer activations according to estimated population statistics.
Do not update population statistics according to mini-batch statistcs from test data.
Now we can define our very simple neural network for MNIST classification.
Now that we have defined our data and computational graph (a.k.a. model), we can train the model! Here is where we need to notice a very important note in the
Note: When is_training is True the moving_mean and moving_variance need to be updated, by default the update_ops are placed in tf.GraphKeys.UPDATE_OPS so they need to be added as a dependency to the train_op, example:
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) if update_ops: updates = tf.group(*update_ops) total_loss = control_flow_ops.with_dependencies([updates], total_loss)
If you are comfortable with TensorFlow’s underlying graph/ops mechanism, the note is fairly straight-forward. If not, here’s a simple way to think of it: when you execute an operation (such as
train_step), only the subgraph components relevant to
train_step will be executed. Unfortunately, the
update_moving_averages operation is not a parent of
train_step in the computational graph, so we will never update the moving averages! To get around this, we have to explicitly tell the graph:
Hey graph, update the moving averages before you finish the training step!
Unfortunately, the instructions in the documentation are a little out of date. Furthermore, if you think about it a little more, you may conclude that attaching the
update ops to
total_loss may not be desirable if you wish to compute the
total_loss of the test set during test time. Personally, I think it makes more sense to attach the
update ops to the
train_step itself. So I modified the code a little and created the following training function
And we’re done! We can now train our model and see what happens. Below, I provide a comparison of the model without batch normalization, the model with pre-activation batch normalization, and the model with post-activation batch normalization.
As you can see, batch normalization really does help with training (not always, but it certainly did in this simple example).
You have the choice of applying batch normalization either before or after the non-linearity, depending on your definition of the “activation distribution of interest” that you wish to normalize. It will probably end up being a hyperparameter that you’ll just have to tinker with.
There is also the question of what it means to share the same batch normalization layer across multiple models. I think this question should be treated with some care, and it really depends on what you think you’ll gain out of sharing the batch normalization layer. In particular, we should be careful about sharing the mini-batch statistics across multiple data streams if we expect the data streams to have distinct distributions, as is the case when using batch normalization in recurrent neural networks. As of the moment, the
tf.contrib.layers.batch_norm function does not allow this level of control.
But I do!… Kind of. It’s a fairly short piece of code, so it should be easy to modify it to fit your own purposes. *runs away*
End of post