Neural Network Pruning
Youtube search... ...Google search
- The Lottery Ticket Hypothesis: Finfing Spare, Trainable Neural Networks | Jonathan Frankle & Michael Carbin - MIT CSAIL
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask | Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski - Uber Engineering
- Pruning algorithms of neural networks - a comparative study | M. Gethsiyal Augasta, T. Kathirvalavakumar
- Model pruning: Training tensorflow models to have masked connections
Lottery Ticket Hypothesis: The discovery hinges on the reality that the random connection strengths assigned during initialization aren’t, in fact, random in their consequences: they predispose different parts of the network to fail or succeed before training even happens. Put another way, the initial configuration influences which final configuration the network will arrive at. By focusing on this idea, the researchers found that if you prune an oversize network after training, you can actually reuse the resultant smaller network to train on new data and preserve high performance—as long as you reset each connection within this downsized network back to its initial strength. From this finding, Frankle and his coauthor Michael Carbin, an assistant professor at MIT, propose what they call the “lottery ticket hypothesis.” When you randomly initialize a neural network’s connection strengths, it’s almost like buying a bag of lottery tickets. Within your bag, you hope, is a winning ticket—i.e., an initial configuration that will be easy to train and result in a successful model. A new way to build tiny neural networks could create powerful AI on your phone | Karen Hao - MIT Technology Review
