The lottery ticket hypothesis says:
A randomly initialized, dense neural network contains a sub-network that is initialized such that, when trained in isolation, can match the test accuracy of the original network after training for at most the same number of iterations.
This hypothesis can be used for Network pruning, by doing the following.
- Train the big network
- Detect the sub-network of the lottery ticket, we can do that by pruning, once pruned we will have a smaller network but our accuracy would have also decreased
- Get the random initialized values for the subnetwork.
- Train again, this time only the subnetwork, but it’s important, with the same random values the big one had.
- Voila! you have a smaller network that has an accuracy similar to the first one.
You can check the https://www.youtube.com/watch?v=LXm_6eq0Cs4 for more information.
Conclusions: 🔗
- Initialization plays a key role in Neural networks