Yaroslav Ganin, Skoltech PhD student in Skoltech Computer Vision Group, and his supervisor, Victor Lempitsky, have proposed a novel approach for deep neural networks training in case of insufficient amount of labeled data. The paper that describes this method has been accepted for the International Conference on Machine Learning.
Training a top-performing model (e.g. for image classification) usually requires enormous amounts of pre-labeled data. For example for deep neural network to distinguish a cat from a dog in a reliable way, one needs to have a few hundreds of training images of cats and dogs labeled with the kind of animal that is depicted there. Unfortunately, for the majority of challenging tasks, the difficulty of acquiring such data in sufficient quantity often times varies from “hard” to “impossible” for a lot of tasks. But also often times one can try to re-use the model already trained with the data that is relatively easy to collect and adjust it for the more difficult tasks with similar input data. This approach is called “domain adaptation”.
Domain adaptation methods aid in solving the problem of insufficient labeled data for the new task. This is made possible by adjusting the parameters of the already well-trained model in the cases where the new input data has similar nature to the labeled data that has been used to train the first model but is in fact different. Consider this example: one can train the model to recognize objects in labeled images acquired from internet and then adjust it to the domain of web-cam images:
Example has been taken from the dataset collected by U. of Berkeley researchers
It’s important to note that one of the key desired features of this approach is that the new input data is not required to be labeled (and thus the learning is unsupervised).
The essence of the method called “Unsupervised Domain Adaptation by Backpropagation” is in adding just a few layers to the existing network architecture to perform domain classification:
Network architecture (picture taken from the paper)
The resulting slightly more complex structure is then trained using the widely known backpropagation method. This approach is universal, easily implemented and does not require any labeled data for the new domain. It also has a property of promoting the features that are both discriminative for the source domain and invariant with respect to the shift between domains.
The method that authors had proposed was accepted by the researchers community with lively interest. Despite its recency, it is already being discussed as part of the Advances in Computer Vision course taught at MIT. The paper that describes this method is currently being expanded with the help of Evgenia Ustinova, PhD student at Skoltech, and prepared to be published in Journal of Machine Learning Research. http://jmlr.org/proceedings/papers/v37/ganin15.html
Media contacts – Alexander Zolotarev, +7 916 686 73 34, , https://www.skoltech.ru
*****
About Skolkovo Institute of Science and Technology:
The Skolkovo Institute of Science and Technology (Skoltech) is a private graduate research university in Skolkovo, Russia, a suburb of Moscow. Established in 2011 in collaboration with The Massachusetts Institute of Technology (MIT), Skoltech will educate global leaders in innovation, advance scientific knowledge, and foster new technologies to address critical issues facing Russia and the world. Applying international research and educational models, the university integrates the best Russian scientific traditions with twenty-first century entrepreneurship and innovation.
Contact information:
Skoltech Communications
+7 (495) 280 14 81
