ift6266: writeup/nips2010_cameraready.tex annotate

annotate writeup/nips2010_cameraready.tex @ 615:337253b82409

repair the class/fct that allow to read pnist07 and others by allowing them to read gziped file.

author	Frederic Bastien <nouiz@nouiz.org>
date	Fri, 07 Jan 2011 11:44:23 -0500
parents	d840139444fe
children

rev	line source
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 \documentclass{article} % For LaTeX2e
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 \usepackage{nips10submit_e,times}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3 \usepackage{wrapfig}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	4 \usepackage{amsthm,amsmath,bbm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	5 \usepackage[psamsfonts]{amssymb}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 \usepackage{algorithm,algorithmic}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 \usepackage[utf8]{inputenc}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 \usepackage{graphicx,subfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	9 \usepackage[numbers]{natbib}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	11 \addtolength{\textwidth}{10mm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	12 \addtolength{\textheight}{10mm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	13 \addtolength{\topmargin}{-5mm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	14 \addtolength{\evensidemargin}{-5mm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	15 \addtolength{\oddsidemargin}{-5mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17 %\setlength\parindent{0mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20 \author{
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	21 Frédéric Bastien,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 Yoshua Bengio,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	23 Arnaud Bergeron,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 Nicolas Boulanger-Lewandowski,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	25 Thomas Breuel,\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26 {\bf Youssouf Chherawala,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 Moustapha Cisse,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	28 Myriam Côté,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29 Dumitru Erhan,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 Jeremy Eustache,}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	31 {\bf Xavier Glorot,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 Xavier Muller,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 Sylvain Pannetier Lebeuf,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 Razvan Pascanu,} \\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 {\bf Salah Rifai,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	36 Francois Savard,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 Guillaume Sicard}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 Dept. IRO, U. Montreal
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	39 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	40
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	41 \begin{document}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 %\makeanontitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44 \maketitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47 \begin{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 Recent theoretical and empirical work in statistical machine learning has
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49 demonstrated the importance of learning algorithms for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 architectures, i.e., function classes obtained by composing multiple
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 non-linear transformations. Self-taught learning (exploiting unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 examples or examples from other distributions) has already been applied
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 to deep learners, but mostly to show the advantage of unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 examples. Here we explore the advantage brought by {\em out-of-distribution examples}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 For this purpose we
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	56 developed a powerful generator of stochastic variations and noise
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 processes for character images, including not only affine transformations
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	58 but also slant, local elastic deformations, changes in thickness,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	59 background images, grey level changes, contrast, occlusion, and various
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60 types of noise. The out-of-distribution examples are obtained from these
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 highly distorted images or by including examples of object classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 different from those in the target test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63 We show that {\em deep learners benefit
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64 more from them than a corresponding shallow learner}, at least in the area of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65 handwritten character recognition. In fact, we show that they reach
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 human-level performance on both handwritten digit classification and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 62-class handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 \end{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 \section{Introduction}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	74 {\bf Deep Learning} has emerged as a promising new area of research in
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	75 statistical machine learning~\citep{Hinton06}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	76 (see \citet{Bengio-2009} for a review).
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 Learning algorithms for deep architectures are centered on the learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78 of useful representations of data, which are better suited to the task at hand,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 and are organized in a hierarchy with multiple levels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 This is in part inspired by observations of the mammalian visual cortex,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	81 which consists of a chain of processing elements, each of which is associated with a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 different representation of the raw visual input. In fact,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	83 it was found recently that the features learnt in deep architectures resemble
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 those observed in the first two of these stages (in areas V1 and V2
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85 of visual cortex)~\citep{HonglakL2008}, and that they become more and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	86 more invariant to factors of variation (such as camera movement) in
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	87 higher layers~\citep{Goodfellow2009}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	88 It has been hypothesized that learning a hierarchy of features increases the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	89 ease and practicality of developing representations that are at once
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	90 tailored to specific tasks, yet are able to borrow statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	91 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	92 feature representation can lead to higher-level (more abstract, more
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	93 general) features that are more robust to unanticipated sources of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	94 variance extant in real data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	95
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	96 {\bf Self-taught learning}~\citep{RainaR2007} is a paradigm that combines principles
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	97 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	98 that are unlabeled and possibly come from a distribution different from the target
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	99 distribution, e.g., from other classes than those of interest.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	100 It has already been shown that deep learners can clearly take advantage of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	101 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	102 but more needs to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	103 of {\em out-of-distribution} examples and of the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	104 (one exception is~\citep{CollobertR2008}, which uses a different kind
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	105 of learning algorithm). In particular the {\em relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	106 advantage} of deep learning for these settings has not been evaluated.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	107 The hypothesis discussed in the conclusion is that a deep hierarchy of features
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	108 may be better able to provide sharing of statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	109 between different regions in input space or different tasks.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	110
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	111 Previous comparative experimental results with stacking of RBMs and DAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	112 to build deep supervised predictors had shown that they could outperform
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	113 shallow architectures in a variety of settings, especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	114 when the data involves complex interactions between many factors of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	115 variation~\citep{LarochelleH2007,Bengio-2009}. Other experiments have suggested
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116 that the unsupervised layer-wise pre-training acted as a useful
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	117 prior~\citep{Erhan+al-2010} that allows one to initialize a deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	118 neural network in a relatively much smaller region of parameter space,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119 corresponding to better generalization.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	120
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	121 To further the understanding of the reasons for the good performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	122 observed with deep learners, we focus here on the following {\em hypothesis}:
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	123 intermediate levels of representation, especially when there are
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	124 more such levels, can be exploited to {\bf share
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	125 statistical strength across different but related types of examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	126 such as examples coming from other tasks than the task of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127 (the multi-task setting), or examples coming from an overlapping
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	128 but different distribution (images with different kinds of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	129 and noises, here). This is consistent with the hypotheses discussed
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	130 in~\citet{Bengio-2009} regarding the potential advantage
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	131 of deep learning and the idea that more levels of representation can
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132 give rise to more abstract, more general features of the raw input.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	133
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	134 This hypothesis is related to the
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	135 {\bf self-taught learning} setting~\citep{RainaR2007}, which combines principles
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137 that are unlabeled and possibly come from a distribution different from the target
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	138 distribution, e.g., from classes other than those of interest.
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	139 It has already been shown that deep learners can take advantage of
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	141 but more needed to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	142 of {\em out-of-distribution} examples and of the {\em multi-task} setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	143 (one exception is~\citep{CollobertR2008}, which shares and uses unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144 pre-training only with the first layer). In particular the {\em relative
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	145 advantage of deep learning} for these settings had not been evaluated.
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	146
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	147
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	148 %
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	149 The {\bf main claim} of this paper is that deep learners (with several levels of representation) can
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150 {\bf benefit more from out-of-distribution examples than shallow learners} (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	151 level), both in the context of the multi-task setting and from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	152 perturbed examples. Because we are able to improve on state-of-the-art
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	153 performance and reach human-level performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	154 on a large-scale task, we consider that this paper is also a contribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	155 to advance the application of machine learning to handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156 More precisely, we ask and answer the following questions:
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	157
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	158 %\begin{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	159 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	160 Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	161 MNIST digit images generalize to the setting of a similar but much larger and richer
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	162 dataset, the NIST special database 19, with 62 classes and around 800k examples?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	163
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	165 To what extent does the perturbation of input images (e.g. adding
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	166 noise, affine transformations, background images) make the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	167 classifiers better not only on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168 the {\em original clean examples}? We study this question in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	169 context of the 62-class and 10-class tasks of the NIST special database 19.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	170
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	171 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	172 Do deep architectures {\em benefit {\bf more} from such out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	173 examples, in particular do they benefit more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	174 examples that are perturbed versions of the examples from the task of interest?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	175
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	176 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	177 Similarly, does the feature learning step in deep learning algorithms benefit {\bf more}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	178 from training with moderately {\em different classes} (i.e. a multi-task learning scenario) than
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	179 a corresponding shallow and purely supervised architecture?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 We train on 62 classes and test on 10 (digits) or 26 (upper case or lower case)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181 to answer this question.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	182 %\end{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	183
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	184 Our experimental results provide positive evidence towards all of these questions,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	185 as well as {\bf classifiers that reach human-level performance on 62-class isolated character
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	186 recognition and beat previously published results on the NIST dataset (special database 19)}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	187 To achieve these results, we introduce in the next section a sophisticated system
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	188 for stochastically transforming character images and then explain the methodology,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	189 which is based on training with or without these transformed images and testing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	190 clean ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	191 Code for generating these transformations as well as for the deep learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	192 algorithms are made available at {\tt http://hg.assembla.com/ift6266}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	193
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	194 \vspace*{-3mm}
606 bd7d11089a47 more fix to newpage command. Frederic Bastien <nouiz@nouiz.org> parents: 605 diff changeset	195 %%\newpage
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	196 \section{Perturbed and Transformed Character Images}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	197 \label{s:perturbations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	198 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	199
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	200 %\begin{minipage}[h]{\linewidth}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	201 \begin{wrapfigure}[8]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	202 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	203 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	204 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	205 \includegraphics[scale=.4]{images/Original.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	206 {\bf Original}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	208 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209 %\vspace{0.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	210 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	211 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	212 This section describes the different transformations we used to stochastically
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	213 transform $32 \times 32$ source images (such as the one on the left)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	214 in order to obtain data from a larger distribution which
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	215 covers a domain substantially larger than the clean characters distribution from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	216 which we start.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	217 Although character transformations have been used before to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	218 improve character recognizers, this effort is on a large scale both
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	219 in number of classes and in the complexity of the transformations, hence
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	220 in the complexity of the learning task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	221 More details can
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	222 be found in this technical report~\citep{ARXIV-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	223 The code for these transformations (mostly python) is available at
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	224 {\tt http://hg.assembla.com/ift6266}. All the modules in the pipeline share
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	225 a global control parameter ($0 \le complexity \le 1$) modulating the
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	226 amount of deformation or noise.
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	227 There are two main parts in the pipeline. The first one,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	228 from thickness to pinch, performs transformations. The second
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	229 part, from blur to contrast, adds different kinds of noise.
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	230 %\end{minipage}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	231
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	232 %\newpage
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	233 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	234 %\subsection{Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	235 {\large\bf 2.1 Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	236 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	237
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	238
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	239 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	240 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	241 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	242 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	243 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	244 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	245 \includegraphics[scale=.4]{images/Thick_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	246 {\bf Thickness}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	247 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	248 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	249 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	250 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	251 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	252 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	253 are applied. The neighborhood of each pixel is multiplied
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	254 element-wise with a {\em structuring element} matrix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	255 The pixel value is replaced by the maximum or the minimum of the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	256 matrix, respectively for dilation or erosion. Ten different structural elements with
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	257 increasing dimensions (largest is $5\times5$) were used. For each image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	258 randomly sample the operator type (dilation or erosion) with equal probability and one structural
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	259 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	260 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	261 A neutral element (no transformation)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	262 is always present in the set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	263 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	264 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	265 \vspace*{3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	266
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	267 \begin{minipage}[h]{\linewidth}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	268 \begin{wrapfigure}[7]{l}{0.15\textwidth}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	269 %\begin{minipage}[b]{0.14\linewidth}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	270 %\centering
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	271 \begin{center}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	272 \vspace*{-5mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	273 \includegraphics[scale=.4]{images/Slant_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	274 {\bf Slant}
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	275 \end{center}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	276 \end{wrapfigure}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	277
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	278 %\end{minipage}%
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	279 %\hspace{0.3cm}
63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	280 %\begin{minipage}[b]{0.83\linewidth}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	281 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	282 To produce {\bf slant}, each row of the image is shifted
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	283 proportionally to its height: $shift = round(slant \times height)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	284 $slant \sim U[-complexity,complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	285 The shift is randomly chosen to be either to the left or to the right.
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	286 %\vspace{8mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	287 \end{minipage}
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	288 \vspace*{10mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	289
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	290 \begin{minipage}[h]{\linewidth}
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	291 %\begin{minipage}[b]{0.14\linewidth}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	292 %\centering
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	293 \begin{wrapfigure}[7]{l}{0.15\textwidth}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	294 \begin{center}
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	295 \vspace*{-5mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	296 \includegraphics[scale=.4]{images/Affine_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	297 {\small {\bf Affine \mbox{Transformation}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	298 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	299 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	300 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	301 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	302 A $2 \times 3$ {\bf affine transform} matrix (with
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	303 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	304 Output pixel $(x,y)$ takes the value of input pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	305 nearest to $(ax+by+c,dx+ey+f)$,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	306 producing scaling, translation, rotation and shearing.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	307 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	308 forbid large rotations (to avoid confusing classes) but to give good
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	309 variability of the transformation: $a$ and $d$ $\sim U[1-3
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	310 complexity,1+3\,complexity]$, $b$ and $e$ $\sim U[-3 \,complexity,3\,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	311 complexity]$, and $c$ and $f \sim U[-4 \,complexity, 4 \,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	312 complexity]$.\\
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	313 %\end{minipage}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	314 \end{minipage}
605 63f838479510 fix transformation description printing. Frederic Bastien <nouiz@nouiz.org> parents: 604 diff changeset	315 \vspace*{3mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	316
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	317 \vspace*{-4.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	318
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	319 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	320 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	321 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	322 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	323 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	324 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	325 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	326 {\bf Local Elastic Deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	327 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	328 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	329 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	330 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	331 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	332 The {\bf local elastic deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	333 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	334 which provides more details.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	335 The intensity of the displacement fields is given by
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	336 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	337 convolved with a Gaussian 2D kernel (resulting in a blur) of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	338 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	339 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	340 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	341
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	342 \vspace*{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	343
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	344 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	345 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	346 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	347 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	348 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	349 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	350 \includegraphics[scale=.4]{images/Pinch_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	351 {\bf Pinch}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	352 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	353 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	354 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	355 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	356 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	357 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl set to 0.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	358 A pinch is ``similar to projecting the image onto an elastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	359 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	360 For a square input image, draw a radius-$r$ disk
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	361 around its center $C$. Any pixel $P$ belonging to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	362 that disk has its value replaced by
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	363 the value of a ``source'' pixel in the original image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	364 on the line that goes through $C$ and $P$, but
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	365 at some other distance $d_2$. Define $d_1=distance(P,C)$
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	366 and $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	367 d_1$, where $pinch$ is a parameter of the filter.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	368 The actual value is given by bilinear interpolation considering the pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	369 around the (non-integer) source position thus found.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	370 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	371 %\vspace{1.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	372 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	373
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	374 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	375
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	376 {\large\bf 2.2 Injecting Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	377 %\subsection{Injecting Noise}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	378 %\vspace{2mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	379
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	380 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	381 %\vspace*{-.2cm}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	382 %\begin{minipage}[t]{0.14\linewidth}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	383 \begin{wrapfigure}[8]{l}{0.15\textwidth}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	384 \begin{center}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	385 \vspace*{-5mm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	386 %\vspace*{-2mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	387 \includegraphics[scale=.4]{images/Motionblur_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	388 {\bf Motion Blur}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	389 %\end{minipage}%
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	390 \end{center}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	391 \end{wrapfigure}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	392 %\hspace{0.3cm}
d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	393 %\begin{minipage}[t]{0.83\linewidth}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	394 %\vspace*{.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	395 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	396 has parameters $length$ and $angle$. The value of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	397 a pixel in the final image is approximately the mean of the first $length$ pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	398 found by moving in the $angle$ direction,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	399 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	400 \vspace{5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	401 \end{minipage}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	402 %\end{minipage}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	403
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	404 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	405
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	406 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	407 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	408 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	409 \includegraphics[scale=.4]{images/occlusion_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	410 {\bf Occlusion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	411 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	412 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	413 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	414 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	415 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	416 image and places it over the original {\em occluded}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	417 image. Pixels are combined by taking the max(occluder, occluded),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	418 i.e. keeping the lighter ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	419 The rectangle corners
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	420 are sampled so that larger complexity gives larger rectangles.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	421 The destination position in the occluded image are also sampled
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	422 according to a normal distribution (more details in~\citet{ARXIV-2010}).
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	423 This module is skipped with probability 60\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	424 %\vspace{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	425 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	426 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	427
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	428 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	429
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	430 \begin{wrapfigure}[8]{l}{0.15\textwidth}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	431 \vspace*{-3mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	432 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	433 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	434 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	435 \includegraphics[scale=.4]{images/Bruitgauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	436 {\bf Gaussian Smoothing}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	437 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	438 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	439 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	440 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	441 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	442 With the {\bf Gaussian smoothing} module,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	443 different regions of the image are spatially smoothed.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	444 This is achieved by first convolving
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	445 the image with an isotropic Gaussian kernel of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	446 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	447 complexity]$ and $[2,2 + 6 \times complexity]$. This filtered image is normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	448 between $0$ and $1$. We also create an isotropic weighted averaging window, of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	449 kernel size, with maximum value at the center. For each image we sample
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	450 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	451 averaging centers between the original image and the filtered one. We
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	452 initialize to zero a mask matrix of the image size. For each selected pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	453 we add to the mask the averaging window centered on it. The final image is
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	454 computed from the following element-wise operation: $\frac{image + filtered\_image
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	455 \times mask}{mask+1}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	456 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	457 %\end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	458
606 bd7d11089a47 more fix to newpage command. Frederic Bastien <nouiz@nouiz.org> parents: 605 diff changeset	459 %\newpage
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	460
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	461 \vspace*{1mm}
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	462
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	463 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	464 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	465 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	466 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	467 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	468 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	469 \includegraphics[scale=.4]{images/Permutpixel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	470 {\small\bf Permute Pixels}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	471 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	472 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	473 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	474 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	475 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	476 This module {\bf permutes neighbouring pixels}. It first selects a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	477 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	478 of these pixels is then sequentially exchanged with a random pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	479 among its four nearest neighbors (on its left, right, top or bottom).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	480 This module is skipped with probability 80\%.\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	481 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	482 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	483
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	484 \vspace{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	485
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	486 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	487 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	488 %\vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	489 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	490 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	491 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	492 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	493 \includegraphics[scale=.4]{images/Distorsiongauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	494 {\small \bf Gauss. Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	495 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	496 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	497 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	498 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	499 \vspace*{12mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	500 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	501 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	502 This module is skipped with probability 70\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	503 %\vspace{1.1cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	504 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	505
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	506 \vspace*{1.2cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	507
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	508 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	509 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	510 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	511 \includegraphics[scale=.4]{images/background_other_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	512 {\small \bf Bg Image}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	513 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	514 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	515 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	516 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	517 background image behind the letter, from a randomly chosen natural image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	518 with contrast adjustments depending on $complexity$, to preserve
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	519 more or less of the original character image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	520 %\vspace{.8cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	521 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	522 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	523 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	524
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	525 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	526 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	527 \includegraphics[scale=.4]{images/Poivresel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	528 {\small \bf Salt \& Pepper}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	529 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	530 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	531 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	532 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	533 The number of selected pixels is $0.2 \times complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	534 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	535 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	536 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	537 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	538
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	539 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	540
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	541 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	542 \begin{wrapfigure}[7]{l}{0.14\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	543 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	544 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	545 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	546 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	547 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	548 {\bf Scratches}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	549 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	550 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	551 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	552 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	553 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	554 The {\bf scratches} module places line-like white patches on the image. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	555 lines are heavily transformed images of the digit ``1'' (one), chosen
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	556 at random among 500 such 1 images,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	557 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	558 complexity)^2$ (in degrees), using bi-cubic interpolation.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	559 Two passes of a grey-scale morphological erosion filter
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	560 are applied, reducing the width of the line
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	561 by an amount controlled by $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	562 This module is skipped with probability 85\%. The probabilities
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	563 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	564 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	565
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	566 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	567
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	568 \begin{minipage}[t]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	569 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	570 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	571 {\bf Grey Level \& Contrast}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	572 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	573 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	574 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	575 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	576 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	577 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	578 polarity is inverted with probability 50\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	579 %\vspace{.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	580 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	581 \vspace{2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	582
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	583 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	584 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	585 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	586 \caption{Illustration of the pipeline of stochastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	587 transformations applied to the image of a lower-case \emph{t}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	588 (the upper left image). Each image in the pipeline (going from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	589 left to right, first top line, then bottom line) shows the result
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	590 of applying one of the modules in the pipeline. The last image
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	591 (bottom right) is used as training example.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	592 \label{fig:pipeline}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	593 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	594 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	595
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	596 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	597 \section{Experimental Setup}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	598 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	599
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	600 Much previous work on deep learning had been performed on
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	601 the MNIST digits task
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	602 with 60~000 examples, and variants involving 10~000
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	603 examples~\citep{VincentPLarochelleH2008-very-small}.
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	604 The focus here is on much larger training sets, from 10 times to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	605 to 1000 times larger, and 62 classes.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	606
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	607 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	608 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	609 and {\bf OCR data} (scanned machine printed characters). Once a character
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	610 is sampled from one of these {\em data sources} (chosen randomly), the second step is to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	611 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	612
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	613 To provide a baseline of error rate comparison we also estimate human performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	614 on both the 62-class task and the 10-class digits task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	615 We compare the best Multi-Layer Perceptrons (MLP) against
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	616 the best Stacked Denoising Auto-encoders (SDA), when
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	617 both models' hyper-parameters are selected to minimize the validation set error.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	618 We also provide a comparison against a precise estimate
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	619 of human performance obtained via Amazon's Mechanical Turk (AMT)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	620 service ({\tt http://mturk.com}).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	621 AMT users are paid small amounts
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	622 of money to perform tasks for which human intelligence is required.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	623 An incentive for them to do the job right is that payment can be denied
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	624 if the job is not properly done.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	625 Mechanical Turk has been used extensively in natural language processing and vision.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	626 %processing \citep{SnowEtAl2008} and vision
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	627 %\citep{SorokinAndForsyth2008,whitehill09}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	628 AMT users were presented
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	629 with 10 character images at a time (from a test set) and asked to choose 10 corresponding ASCII
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	630 characters. They were forced to choose a single character class (either among the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	631 62 or 10 character classes) for each image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	632 80 subjects classified 2500 images per (dataset,task) pair.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	633 Different humans labelers sometimes provided a different label for the same
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	634 example, and we were able to estimate the error variance due to this effect
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	635 because each image was classified by 3 different persons.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	636 The average error of humans on the 62-class task NIST test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	637 is 18.2\%, with a standard error of 0.1\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	638
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	639 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	640 \subsection{Data Sources}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	641 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	642
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	643 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	644 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	645 {\bf NIST.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	646 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	647 widely used for training and testing character
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	648 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	649 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	650 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	651 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	652 The fourth partition (called $hsf_4$, 82587 examples),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	653 experimentally recognized to be the most difficult one, is the one recommended
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	654 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	655 for that purpose. We randomly split the remainder (731,668 examples) into a training set and a validation set for
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	656 model selection.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	657 The performances reported by previous work on that dataset mostly use only the digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	658 Here we use all the classes both in the training and testing phase. This is especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	659 useful to estimate the effect of a multi-task setting.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	660 The distribution of the classes in the NIST training and test sets differs
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	661 substantially, with relatively many more digits in the test set, and a more uniform distribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	662 of letters in the test set (whereas in the training set they are distributed
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	663 more like in natural text).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	664 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	665
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	666 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	667 {\bf Fonts.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	668 In order to have a good variety of sources we downloaded an important number of free fonts from:
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	669 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	670 % TODO: pointless to anonymize, it's not pointing to our work
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	671 Including an operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	672 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	673 directly as input to our models.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	674 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	675
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	676 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	677 {\bf Captchas.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	678 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a Python-based captcha generator library) for
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	679 generating characters of the same format as the NIST dataset. This software is based on
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	680 a random character class generator and various kinds of transformations similar to those described in the previous sections.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	681 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	682 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	683 depending on the value of the complexity parameter provided by the user of the data source.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	684 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	685 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	686
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	687 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	688 {\bf OCR data.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	689 A large set (2 million) of scanned, OCRed and manually verified machine-printed
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	690 characters where included as an
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	691 additional source. This set is part of a larger corpus being collected by the Image Understanding
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	692 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	693 ({\tt http://www.iupr.com}), and which will be publicly released.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	694 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	695 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	696
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	697 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	698 \subsection{Data Sets}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	699 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	700
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	701 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	702 from one of the 62 character classes. They are obtained from the optional application of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	703 perturbation pipeline to iid samples from the datasources, and they are randomly split into
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	704 training set, validation set, and test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	705 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	706 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	707
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	708 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	709 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	710 \{651668 / 80000 / 82587\} \{training / validation / test\} examples, containing
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	711 upper case, lower case, and digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	712 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	713
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	714 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	715 {\bf P07.} This dataset of upper case, lower case and digit images
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	716 is obtained by taking raw characters from all four of the above sources
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	717 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	718 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	719 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	720 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	721 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	722 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	723
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	724 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	725 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	726 except that we only apply
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	727 transformations from slant to pinch. Therefore, the character is
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	728 transformed but no additional noise is added to the image, giving images
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	729 closer to the NIST dataset.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	730 It has \{81,920,000 / 80,000 / 20,000\} \{training / validation / test\} examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	731 obtained from the corresponding NIST sets plus other sources.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	732 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	733
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	734 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	735 \subsection{Models and their Hyperparameters}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	736 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	737
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	738 The experiments are performed using MLPs (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	739 hidden layer) and deep SDAs.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	740 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	741
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	742 {\bf Multi-Layer Perceptrons (MLP).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	743 Whereas previous work had compared deep architectures to both shallow MLPs and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	744 SVMs, we only compared to MLPs here because of the very large datasets used
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	745 (making the use of SVMs computationally challenging because of their quadratic
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	746 scaling behavior). Preliminary experiments on training SVMs (libSVM) with subsets of the training
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	747 set allowing the program to fit in memory yielded substantially worse results
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	748 than those obtained with MLPs. For training on nearly a hundred million examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	749 (with the perturbed data), the MLPs and SDA are much more convenient than
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	750 classifiers based on kernel methods.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	751 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	752 exponentials) on the output layer for estimating $P(class \| image)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	753 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	754 Training examples are presented in minibatches of size 20. A constant learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	755 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	756 %through preliminary experiments (measuring performance on a validation set),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	757 %and $0.1$ (which was found to work best) was then selected for optimizing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	758 %the whole training sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	759 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	760
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	761
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	762 {\bf Stacked Denoising Auto-encoders (SDA).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	763 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	764 can be used to initialize the weights of each layer of a deep MLP (with many hidden
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	765 layers)
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	766 apparently setting parameters in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	767 basin of attraction of supervised gradient descent yielding better
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	768 generalization~\citep{Erhan+al-2010}. This initial {\em unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	769 pre-training phase} uses all of the training images but not the training labels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	770 Each layer is trained in turn to produce a new representation of its input
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	771 (starting from the raw pixels).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	772 It is hypothesized that the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	773 advantage brought by this procedure stems from a better prior,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	774 on the one hand taking advantage of the link between the input
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	775 distribution $P(x)$ and the conditional distribution of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	776 $P(y\|x)$ (like in semi-supervised learning), and on the other hand
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	777 taking advantage of the expressive power and bias implicit in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	778 deep architecture (whereby complex concepts are expressed as
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	779 compositions of simpler ones through a deep hierarchy).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	780
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	781 \iffalse
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	782 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	783 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	784 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	785 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	786 \caption{Illustration of the computations and training criterion for the denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	787 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	788 the layer (i.e. raw input or output of previous layer)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	789 s corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	790 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	791 is compared to the uncorrupted input $x$ through the loss function
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	792 $L_H(x,z)$, whose expected value is approximately minimized during training
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	793 by tuning $\theta$ and $\theta'$.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	794 \label{fig:da}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	795 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	796 \end{figure}
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	797 \fi
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	798
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	799 Here we chose to use the Denoising
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	800 Auto-encoder~\citep{VincentPLarochelleH2008-very-small} as the building block for
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	801 these deep hierarchies of features, as it is simple to train and
607 d840139444fe NIPS workshop spotlight Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 606 diff changeset	802 explain (see % Figure~\ref{fig:da}, as well as
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	803 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	804 provides efficient inference, and yielded results
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	805 comparable or better than RBMs in series of experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	806 \citep{VincentPLarochelleH2008-very-small}. It really corresponds to a Gaussian
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	807 RBM trained by a Score Matching criterion~\cite{Vincent-SM-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	808 During training, a Denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	809 Auto-encoder is presented with a stochastically corrupted version
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	810 of the input and trained to reconstruct the uncorrupted input,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	811 forcing the hidden units to represent the leading regularities in
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	812 the data. Here we use the random binary masking corruption
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	813 (which sets to 0 a random subset of the inputs).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	814 Once it is trained, in a purely unsupervised way,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	815 its hidden units' activations can
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	816 be used as inputs for training a second one, etc.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	817 After this unsupervised pre-training stage, the parameters
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	818 are used to initialize a deep MLP, which is fine-tuned by
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	819 the same standard procedure used to train them (see previous section).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	820 The SDA hyper-parameters are the same as for the MLP, with the addition of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	821 amount of corruption noise (we used the masking noise process, whereby a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	822 fixed proportion of the input values, randomly selected, are zeroed), and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	823 separate learning rate for the unsupervised pre-training stage (selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	824 from the same above set). The fraction of inputs corrupted was selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	825 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	826 of hidden layers but it was fixed to 3 based on previous work with
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	827 SDAs on MNIST~\citep{VincentPLarochelleH2008-very-small}. The size of the hidden
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	828 layers was kept constant across hidden layers, and the best results
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	829 were obtained with the largest values that we could experiment
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	830 with given our patience, with 1000 hidden units.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	831
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	832 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	833
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	834 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	835 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	836 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	837 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	838 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	839 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	840 of all models, on NIST and NISTP test sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	841 Right: error rates on NIST test digits only, along with the previous results from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	842 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	843 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	844 \label{fig:error-rates-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	845 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	846 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	847
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	848
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	849 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	850 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	851 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	852 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	853 \caption{Relative improvement in error rate due to self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	854 Left: Improvement (or loss, when negative)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	855 induced by out-of-distribution examples (perturbed data).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	856 Right: Improvement (or loss, when negative) induced by multi-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	857 learning (training on all classes and testing only on either digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	858 upper case, or lower-case). The deep learner (SDA) benefits more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	859 both self-taught learning scenarios, compared to the shallow MLP.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	860 \label{fig:improvements-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	861 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	862 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	863
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	864 \section{Experimental Results}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	865 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	866
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	867 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	868 %\subsection{SDA vs MLP vs Humans}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	869 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	870 The models are either trained on NIST (MLP0 and SDA0),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	871 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	872 on either NIST, NISTP or P07 (regardless of the data set used for training),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	873 either on the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	874 or on the 10-digits task. Training time (including about half
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	875 for unsupervised pre-training, for DAs) on the larger
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	876 datasets is around one day on a GPU (GTX 285).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	877 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	878 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	879 SDA2), along with the previous results on the digits NIST special database
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	880 19 test set from the literature, respectively based on ARTMAP neural
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	881 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	882 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	883 ~\citep{Milgram+al-2005}.% More detailed and complete numerical results
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	884 %(figures and tables, including standard errors on the error rates) can be
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	885 %found in Appendix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	886 The deep learner not only outperformed the shallow ones and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	887 previously published performance (in a statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	888 significant way) but when trained with perturbed data
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	889 reaches human performance on both the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	890 and the 10-class (digits) task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	891 17\% error (SDA1) or 18\% error (humans) may seem large but a large
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	892 majority of the errors from humans and from SDA1 are from out-of-context
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	893 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	894 ``c'' and a ``C'' are often indistinguishible).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	895
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	896 In addition, as shown in the left of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	897 Figure~\ref{fig:improvements-charts}, the relative improvement in error
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	898 rate brought by self-taught learning is greater for the SDA, and these
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	899 differences with the MLP are statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	900 significant.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	901 The left side of the figure shows the improvement to the clean
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	902 NIST test set error brought by the use of out-of-distribution examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	903 (i.e. the perturbed examples examples from NISTP or P07),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	904 over the models trained exclusively on NIST (respectively SDA0 and MLP0).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	905 Relative percent change is measured by taking
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	906 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	907 The right side of
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	908 Figure~\ref{fig:improvements-charts} shows the relative improvement
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	909 brought by the use of a multi-task setting, in which the same model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	910 trained for more classes than the target classes of interest (i.e. training
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	911 with all 62 classes when the target classes are respectively the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	912 lower-case, or upper-case characters). Again, whereas the gain from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	913 multi-task setting is marginal or negative for the MLP, it is substantial
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	914 for the SDA. Note that to simplify these multi-task experiments, only the original
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	915 NIST dataset is used. For example, the MLP-digits bar shows the relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	916 percent improvement in MLP error rate on the NIST digits test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	917 is $100\% \times$ (single-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	918 model's error / multi-task model's error - 1). The single-task model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	919 trained with only 10 outputs (one per digit), seeing only digit examples,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	920 whereas the multi-task model is trained with 62 outputs, with all 62
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	921 character classes as examples. Hence the hidden units are shared across
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	922 all tasks. For the multi-task model, the digit error rate is measured by
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	923 comparing the correct digit class with the output class associated with the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	924 maximum conditional probability among only the digit classes outputs. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	925 setting is similar for the other two target classes (lower case characters
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	926 and upper case characters). Note however that some types of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	927 (NISTP) help more than others (P07) when testing on the clean images.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	928 %%\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	929 %\subsection{Perturbed Training Data More Helpful for SDA}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	930 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	931
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	932 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	933 %\subsection{Multi-Task Learning Effects}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	934 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	935
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	936 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	937 As previously seen, the SDA is better able to benefit from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	938 transformations applied to the data than the MLP. In this experiment we
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	939 define three tasks: recognizing digits (knowing that the input is a digit),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	940 recognizing upper case characters (knowing that the input is one), and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	941 recognizing lower case characters (knowing that the input is one). We
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	942 consider the digit classification task as the target task and we want to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	943 evaluate whether training with the other tasks can help or hurt, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	944 whether the effect is different for MLPs versus SDAs. The goal is to find
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	945 out if deep learning can benefit more (or less) from multiple related tasks
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	946 (i.e. the multi-task setting) compared to a corresponding purely supervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	947 shallow learner.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	948
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	949 We use a single hidden layer MLP with 1000 hidden units, and a SDA
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	950 with 3 hidden layers (1000 hidden units per layer), pre-trained and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	951 fine-tuned on NIST.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	952
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	953 Our results show that the MLP benefits marginally from the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	954 in the case of digits (5\% relative improvement) but is actually hurt in the case
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	955 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	956 On the other hand the SDA benefited from the multi-task setting, with relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	957 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	958 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	959 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	960
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	961
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	962 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	963 \section{Conclusions and Discussion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	964 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	965
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	966 We have found that the self-taught learning framework is more beneficial
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	967 to a deep learner than to a traditional shallow and purely
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	968 supervised learner. More precisely,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	969 the answers are positive for all the questions asked in the introduction.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	970 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	971
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	972 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	973 {\bf Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	974 MNIST digits generalize to a much larger and richer (but similar)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	975 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	976 Yes, the SDA {\em systematically outperformed the MLP and all the previously
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	977 published results on this dataset} (the ones that we are aware of), {\em in fact reaching human-level
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	978 performance} at around 17\% error on the 62-class task and 1.4\% on the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	979 and beating previously published results on the same data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	980
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	981 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	982 {\bf To what extent do self-taught learning scenarios help deep learners,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	983 and do they help them more than shallow supervised ones}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	984 We found that distorted training examples not only made the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	985 classifier better on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	986 the {\em original clean examples}, and more importantly and more novel,
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	987 that deep architectures benefit more from such {\em out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	988 examples. MLPs were helped by perturbed training examples when tested on perturbed input
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	989 images (65\% relative improvement on NISTP)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	990 but only marginally helped (5\% relative improvement on all classes)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	991 or even hurt (10\% relative loss on digits)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	992 with respect to clean examples. On the other hand, the deep SDAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	993 were significantly boosted by these out-of-distribution examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	994 Similarly, whereas the improvement due to the multi-task setting was marginal or
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	995 negative for the MLP (from +5.6\% to -3.6\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	996 it was quite significant for the SDA (from +13\% to +27\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	997 which may be explained by the arguments below.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	998 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	999
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1000 In the original self-taught learning framework~\citep{RainaR2007}, the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1001 out-of-sample examples were used as a source of unsupervised data, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1002 experiments showed its positive effects in a \emph{limited labeled data}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1003 scenario. However, many of the results by \citet{RainaR2007} (who used a
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1004 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1005 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1006 We note instead that, for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1007 architectures, our experiments show that such a positive effect is accomplished
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1008 even in a scenario with a \emph{large number of labeled examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1009 i.e., here, the relative gain of self-taught learning and
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1010 out-of-distribution examples is probably preserved
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1011 in the asymptotic regime. However, note that in our perturbation experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1012 (but not in our multi-task experiments),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1013 even the out-of-distribution examples are labeled, unlike in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1014 earlier self-taught learning experiments~\citep{RainaR2007}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1015
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1016 {\bf Why would deep learners benefit more from the self-taught learning framework}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1017 The key idea is that the lower layers of the predictor compute a hierarchy
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1018 of features that can be shared across tasks or across variants of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1019 input distribution. A theoretical analysis of generalization improvements
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1020 due to sharing of intermediate features across tasks already points
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1021 towards that explanation~\cite{baxter95a}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1022 Intermediate features that can be used in different
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1023 contexts can be estimated in a way that allows to share statistical
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1024 strength. Features extracted through many levels are more likely to
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1025 be more abstract and more invariant to some of the factors of variation
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1026 in the underlying distribution (as the experiments in~\citet{Goodfellow2009} suggest),
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1027 increasing the likelihood that they would be useful for a larger array
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1028 of tasks and input conditions.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1029 Therefore, we hypothesize that both depth and unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1030 pre-training play a part in explaining the advantages observed here, and future
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1031 experiments could attempt at teasing apart these factors.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1032 And why would deep learners benefit from the self-taught learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1033 scenarios even when the number of labeled examples is very large?
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1034 We hypothesize that this is related to the hypotheses studied
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1035 in~\citet{Erhan+al-2010}. In~\citet{Erhan+al-2010}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1036 it was found that online learning on a huge dataset did not make the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1037 advantage of the deep learning bias vanish, and a similar phenomenon
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1038 may be happening here. We hypothesize that unsupervised pre-training
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1039 of a deep hierarchy with self-taught learning initializes the
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1040 model in the basin of attraction of supervised gradient descent
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1041 that corresponds to better generalization. Furthermore, such good
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1042 basins of attraction are not discovered by pure supervised learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1043 (with or without self-taught settings), and more labeled examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1044 does not allow the model to go from the poorer basins of attraction discovered
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1045 by the purely supervised shallow models to the kind of better basins associated
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1046 with deep learning and self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1047
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1048 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1049 can be executed on-line at {\tt http://deep.host22.com}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1050
606 bd7d11089a47 more fix to newpage command. Frederic Bastien <nouiz@nouiz.org> parents: 605 diff changeset	1051 %\newpage
604 51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1052 {
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1053 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1054 %\bibliographystyle{plainnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1055 \bibliographystyle{unsrtnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1056 %\bibliographystyle{apalike}
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1057 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1058
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1059
51213beaed8b draft of NIPS 2010 workshop camera-ready version Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1060 \end{document}

Mercurial > ift6266

annotate writeup/nips2010_cameraready.tex @ 615:337253b82409