Interpretability of Neural Network

Lets see how we can interpret different neural network architecture such as CNN and all.

Saliency maps

This means backpropagating upto pixels. What we do is we backpropagate upto pixels and check which pixels contribute the most.

You know that VGG16 has 1000 softmax at final layer. Can we backpropagate the softmax layer to image ?

No. We have to take the final layer without softmax. Its because softmax comprises of contribution from all the layers and not just that layer, so we cannot take softmax otherwise backpropagation would also affect other pixels.

Now we got 2 options:

1. Take the current score of the image, and its score that it is maximizing.
2. Take all other score and minimize them

Lets take all other score and minimize the gradient because that would give us better result than the first one.

The given image is of peacock. I ran it through the VGGNET and its index is 84.

Here is its saliency map.

So, this part of the image is maximizing its output to have the index of 84. Why does this work and why it doesn’t affect other part of network? Its because we have set training to False which means parameter is not affected.

This is one time gradient ascent. I iterated 5 times though!

Here is the link: https://github.com/SanjivG10/SaliencyMapCNN/blob/master/CNNDeepViz.ipynb

Occlusion Sensitivity:

The figure above shows the mask where the prediction is within the tolerable range or out of it. So whenever we cover the bounding box (shown in right), and try to predict the probability of detecting face, shown by the bounding box in the right, the probability of the image having face decreases beneath the threshold, (network is less confident that it has face). What can we conclude from the figure?

Those boxes (portion) of image are very important for our system. They contribute much to our system.

This process is called Occlusion (blocking).

Class Activation Map

GAP stands for GlobalAveragePooling. We check all those weights connecting a to final softmax and determine which one activated our network the most.

How can I achieve this? We take the network cut off those final layers, and use global average pooling connecting it to softmax and then visualize.So what we do is we upsample the resultant activation map to the input.Requires a bit of fine tuning.

W stands for weight, the rectangular box shows the heat map of each activation in previous layer. We then add it by multiplying it with the weight to get final activation. We can even check which activation has highest weight, which means that activation contributes maximum.