BatchNorm + Dropout = DNN Success! – Synced
What happened: A group of researchers from Tencent Technology, the Chinese University of Hong Kong, and Nankai University have proposed a new deep neural network (DNN) training method that aims to improve training efficiency in machine learning. By combining the Batch Normalization (BatchNorm) and Dropout techniques, the researchers were able to improve the stability of the training process and improve the convergence limit and speed of their network. According to their paper’s abstract on arXiv, the researchers’ “work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed.”
Why it’s important: Deep neural networks can become cumbersome when faced with large amounts of data and complex training models, so a new training method that improves efficiency is valuable to the development of this category of neural network. The new research also proposes a way for BatchNorm to refocus on improved whitening, which, according to Synced, “is a preprocessing technique that seeks to make data less correlated and standardize variance.” More recently, BatchNorm had shifted away from its goal of whitening due to increased computational cost.