Saved Bookmarks
| 1. |
Why Is Zero Initialization Not A Recommended Weight Initialization Technique? |
|
Answer» As a result of setting weights in the network to ZERO, all the neurons at each layer are producing the same OUTPUT and the same gradients during backpropagation. The network can’t learn at all because there is no SOURCE of asymmetry between neurons. That is why we NEED to add randomness to WEIGHT initialization process. As a result of setting weights in the network to zero, all the neurons at each layer are producing the same output and the same gradients during backpropagation. The network can’t learn at all because there is no source of asymmetry between neurons. That is why we need to add randomness to weight initialization process. |
|