Mercurial > ift6266

annotate writeup/nips2010_cameraready.tex @ 604:51213beaed8b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
draft of NIPS 2010 workshop camera-ready version
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Mon, 22 Nov 2010 14:52:33 -0500
parents
children 63f838479510
rev   line source
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 \usepackage{wrapfig}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4 \usepackage{amsthm,amsmath,bbm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 \usepackage[psamsfonts]{amssymb}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage{algorithm,algorithmic}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage[utf8]{inputenc}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{graphicx,subfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9 \usepackage[numbers]{natbib}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 \addtolength{\textwidth}{20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12 \addtolength{\textheight}{20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13 \addtolength{\topmargin}{-10mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14 \addtolength{\evensidemargin}{-10mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15 \addtolength{\oddsidemargin}{-10mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17 %\setlength\parindent{0mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 \author{
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21 Frédéric Bastien,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 Yoshua Bengio,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 Arnaud Bergeron,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 Nicolas Boulanger-Lewandowski,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 Thomas Breuel,\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 {\bf Youssouf Chherawala,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 Moustapha Cisse,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 Myriam Côté,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29 Dumitru Erhan,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 Jeremy Eustache,}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31 {\bf Xavier Glorot,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 Xavier Muller,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 Sylvain Pannetier Lebeuf,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 Razvan Pascanu,} \\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 {\bf Salah Rifai,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 Francois Savard,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 Guillaume Sicard}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 Dept. IRO, U. Montreal
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 \begin{document}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 %\makeanontitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44 \maketitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 \begin{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 Recent theoretical and empirical work in statistical machine learning has
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 demonstrated the importance of learning algorithms for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 architectures, i.e., function classes obtained by composing multiple
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 non-linear transformations. Self-taught learning (exploiting unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 examples or examples from other distributions) has already been applied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 to deep learners, but mostly to show the advantage of unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 examples. Here we explore the advantage brought by {\em out-of-distribution examples}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 For this purpose we
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 developed a powerful generator of stochastic variations and noise
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
57 processes for character images, including not only affine transformations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 but also slant, local elastic deformations, changes in thickness,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59 background images, grey level changes, contrast, occlusion, and various
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 types of noise. The out-of-distribution examples are obtained from these
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 highly distorted images or by including examples of object classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 different from those in the target test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 We show that {\em deep learners benefit
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 more from them than a corresponding shallow learner}, at least in the area of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 handwritten character recognition. In fact, we show that they reach
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 human-level performance on both handwritten digit classification and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 62-class handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 \end{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 \section{Introduction}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 {\bf Deep Learning} has emerged as a promising new area of research in
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 statistical machine learning~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,VincentPLarochelleH2008-very-small,ranzato-08,TaylorHintonICML2009,Larochelle-jmlr-2009,Salakhutdinov+Hinton-2009,HonglakL2009,HonglakLNIPS2009,Jarrett-ICCV2009,Taylor-cvpr-2010}. See \citet{Bengio-2009} for a review.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76 Learning algorithms for deep architectures are centered on the learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 of useful representations of data, which are better suited to the task at hand,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 and are organized in a hierarchy with multiple levels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 This is in part inspired by observations of the mammalian visual cortex,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 which consists of a chain of processing elements, each of which is associated with a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 different representation of the raw visual input. In fact,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 it was found recently that the features learnt in deep architectures resemble
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 those observed in the first two of these stages (in areas V1 and V2
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 of visual cortex)~\citep{HonglakL2008}, and that they become more and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 more invariant to factors of variation (such as camera movement) in
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86 higher layers~\citep{Goodfellow2009}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 It has been hypothesized that learning a hierarchy of features increases the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88 ease and practicality of developing representations that are at once
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89 tailored to specific tasks, yet are able to borrow statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91 feature representation can lead to higher-level (more abstract, more
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
92 general) features that are more robust to unanticipated sources of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
93 variance extant in real data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
95 {\bf Self-taught learning}~\citep{RainaR2007} is a paradigm that combines principles
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 that are unlabeled and possibly come from a distribution different from the target
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
98 distribution, e.g., from other classes than those of interest.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
99 It has already been shown that deep learners can clearly take advantage of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
100 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
101 but more needs to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
102 of {\em out-of-distribution} examples and of the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 (one exception is~\citep{CollobertR2008}, which uses a different kind
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 of learning algorithm). In particular the {\em relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 advantage} of deep learning for these settings has not been evaluated.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106 The hypothesis discussed in the conclusion is that a deep hierarchy of features
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 may be better able to provide sharing of statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108 between different regions in input space or different tasks.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
109
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
111 Whereas a deep architecture can in principle be more powerful than a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
112 shallow one in terms of representation, depth appears to render the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 training problem more difficult in terms of optimization and local minima.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 It is also only recently that successful algorithms were proposed to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
115 overcome some of these difficulties. All are based on unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116 learning, often in an greedy layer-wise ``unsupervised pre-training''
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
117 stage~\citep{Bengio-2009}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118 The principle is that each layer starting from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
119 the bottom is trained to represent its input (the output of the previous
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
120 layer). After this
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
121 unsupervised initialization, the stack of layers can be
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122 converted into a deep supervised feedforward neural network and fine-tuned by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
123 stochastic gradient descent.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124 One of these layer initialization techniques,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
125 applied here, is the Denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 Figure~\ref{fig:da}), which performed similarly or
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
128 better~\citep{VincentPLarochelleH2008-very-small} than previously
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
129 proposed Restricted Boltzmann Machines (RBM)~\citep{Hinton06}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130 in terms of unsupervised extraction
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
131 of a hierarchy of features useful for classification. Each layer is trained
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
132 to denoise its input, creating a layer of features that can be used as
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133 input for the next layer, forming a Stacked Denoising Auto-encoder (SDA).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
134 Note that training a Denoising Auto-encoder
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 can actually been seen as training a particular RBM by an inductive
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 principle different from maximum likelihood~\citep{Vincent-SM-2010},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 namely by Score Matching~\citep{Hyvarinen-2005,HyvarinenA2008}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
138 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
139
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 Previous comparative experimental results with stacking of RBMs and DAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 to build deep supervised predictors had shown that they could outperform
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
142 shallow architectures in a variety of settings, especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143 when the data involves complex interactions between many factors of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 variation~\citep{LarochelleH2007,Bengio-2009}. Other experiments have suggested
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
145 that the unsupervised layer-wise pre-training acted as a useful
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
146 prior~\citep{Erhan+al-2010} that allows one to initialize a deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
147 neural network in a relatively much smaller region of parameter space,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
148 corresponding to better generalization.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
149
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
150 To further the understanding of the reasons for the good performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
151 observed with deep learners, we focus here on the following {\em hypothesis}:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
152 intermediate levels of representation, especially when there are
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 more such levels, can be exploited to {\bf share
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
154 statistical strength across different but related types of examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
155 such as examples coming from other tasks than the task of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
156 (the multi-task setting), or examples coming from an overlapping
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
157 but different distribution (images with different kinds of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
158 and noises, here). This is consistent with the hypotheses discussed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
159 in~\citet{Bengio-2009} regarding the potential advantage
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
160 of deep learning and the idea that more levels of representation can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
161 give rise to more abstract, more general features of the raw input.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163 This hypothesis is related to a learning setting called
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 {\bf self-taught learning}~\citep{RainaR2007}, which combines principles
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
166 that are unlabeled and possibly come from a distribution different from the target
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 distribution, e.g., from other classes than those of interest.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
168 It has already been shown that deep learners can clearly take advantage of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
170 but more needed to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 of {\em out-of-distribution} examples and of the {\em multi-task} setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
172 (one exception is~\citep{CollobertR2008}, which shares and uses unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 pre-training only with the first layer). In particular the {\em relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174 advantage of deep learning} for these settings has not been evaluated.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177 %
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178 The {\bf main claim} of this paper is that deep learners (with several levels of representation) can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 {\bf benefit more from out-of-distribution examples than shallow learners} (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 level), both in the context of the multi-task setting and from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181 perturbed examples. Because we are able to improve on state-of-the-art
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 performance and reach human-level performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 on a large-scale task, we consider that this paper is also a contribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
184 to advance the application of machine learning to handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185 More precisely, we ask and answer the following questions:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187 %\begin{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
189 Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190 MNIST digit images generalize to the setting of a similar but much larger and richer
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
191 dataset, the NIST special database 19, with 62 classes and around 800k examples?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
192
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 To what extent does the perturbation of input images (e.g. adding
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
195 noise, affine transformations, background images) make the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
196 classifiers better not only on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 the {\em original clean examples}? We study this question in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 context of the 62-class and 10-class tasks of the NIST special database 19.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
200 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
201 Do deep architectures {\em benefit {\bf more} from such out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
202 examples, in particular do they benefit more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
203 examples that are perturbed versions of the examples from the task of interest?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 Similarly, does the feature learning step in deep learning algorithms benefit {\bf more}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 from training with moderately {\em different classes} (i.e. a multi-task learning scenario) than
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 a corresponding shallow and purely supervised architecture?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
209 We train on 62 classes and test on 10 (digits) or 26 (upper case or lower case)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
210 to answer this question.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 %\end{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
213 Our experimental results provide positive evidence towards all of these questions,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
214 as well as {\bf classifiers that reach human-level performance on 62-class isolated character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 recognition and beat previously published results on the NIST dataset (special database 19)}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 To achieve these results, we introduce in the next section a sophisticated system
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
217 for stochastically transforming character images and then explain the methodology,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
218 which is based on training with or without these transformed images and testing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
219 clean ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
220 Code for generating these transformations as well as for the deep learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
221 algorithms are made available at {\tt http://hg.assembla.com/ift6266}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
222
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
223 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
224 %\newpage
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
225 \section{Perturbed and Transformed Character Images}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
226 \label{s:perturbations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
227 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
228
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
229 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
230 \begin{wrapfigure}[8]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
231 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
232 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
233 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
234 \includegraphics[scale=.4]{images/Original.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
235 {\bf Original}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
236 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
237 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
238 %\vspace{0.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
239 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
240 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
241 This section describes the different transformations we used to stochastically
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
242 transform $32 \times 32$ source images (such as the one on the left)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
243 in order to obtain data from a larger distribution which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
244 covers a domain substantially larger than the clean characters distribution from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
245 which we start.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
246 Although character transformations have been used before to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
247 improve character recognizers, this effort is on a large scale both
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
248 in number of classes and in the complexity of the transformations, hence
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
249 in the complexity of the learning task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
250 More details can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
251 be found in this technical report~\citep{ARXIV-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
252 The code for these transformations (mostly python) is available at
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
253 {\tt http://hg.assembla.com/ift6266}. All the modules in the pipeline share
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
254 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
255 amount of deformation or noise introduced.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
256 There are two main parts in the pipeline. The first one,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
257 from thickness to pinch, performs transformations. The second
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
258 part, from blur to contrast, adds different kinds of noise.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
259 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
260
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
261 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
262 %\subsection{Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
263 {\large\bf 2.1 Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
264 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
265
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
266
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
267 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
268 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
269 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
270 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
271 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
272 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
273 \includegraphics[scale=.4]{images/Thick_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
274 {\bf Thickness}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
275 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
276 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
277 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
278 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
279 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
280 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
281 are applied. The neighborhood of each pixel is multiplied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
282 element-wise with a {\em structuring element} matrix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
283 The pixel value is replaced by the maximum or the minimum of the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
284 matrix, respectively for dilation or erosion. Ten different structural elements with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
285 increasing dimensions (largest is $5\times5$) were used. For each image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
286 randomly sample the operator type (dilation or erosion) with equal probability and one structural
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
287 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
288 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
289 A neutral element (no transformation)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
290 is always present in the set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
291 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
292 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
293 \vspace*{3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
294
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
295 \begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
296 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
297 \includegraphics[scale=.4]{images/Slant_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
298 {\bf Slant}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
299 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
300 \hspace{0.3cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
301 \begin{minipage}[b]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
302 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
303 To produce {\bf slant}, each row of the image is shifted
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
304 proportionally to its height: $shift = round(slant \times height)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
305 $slant \sim U[-complexity,complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
306 The shift is randomly chosen to be either to the left or to the right.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
307 \vspace{8mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
308 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
309 \vspace*{3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
310
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
311 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
312 \begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
313 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
314 \begin{wrapfigure}[8]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
315 \vspace*{-6mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
316 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
317 \includegraphics[scale=.4]{images/Affine_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
318 {\small {\bf Affine \mbox{Transformation}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
319 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
320 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
321 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
322 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
323 A $2 \times 3$ {\bf affine transform} matrix (with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
324 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
325 Output pixel $(x,y)$ takes the value of input pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
326 nearest to $(ax+by+c,dx+ey+f)$,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
327 producing scaling, translation, rotation and shearing.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
328 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
329 forbid large rotations (to avoid confusing classes) but to give good
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
330 variability of the transformation: $a$ and $d$ $\sim U[1-3
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
331 complexity,1+3\,complexity]$, $b$ and $e$ $\sim U[-3 \,complexity,3\,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
332 complexity]$, and $c$ and $f \sim U[-4 \,complexity, 4 \,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
333 complexity]$.\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
334 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
335 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
336
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
337 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
338 \vspace*{-4.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
339
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
340 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
341 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
342 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
343 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
344 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
345 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
346 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
347 {\bf Local Elastic Deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
348 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
349 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
350 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
351 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
352 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
353 The {\bf local elastic deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
354 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
355 which provides more details.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
356 The intensity of the displacement fields is given by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
357 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
358 convolved with a Gaussian 2D kernel (resulting in a blur) of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
359 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
360 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
361 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
362
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
363 \vspace*{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
364
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
365 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
366 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
367 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
368 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
369 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
370 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
371 \includegraphics[scale=.4]{images/Pinch_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
372 {\bf Pinch}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
373 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
374 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
375 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
376 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
377 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
378 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl set to 0.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
379 A pinch is ``similar to projecting the image onto an elastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
380 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
381 For a square input image, draw a radius-$r$ disk
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
382 around its center $C$. Any pixel $P$ belonging to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
383 that disk has its value replaced by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
384 the value of a ``source'' pixel in the original image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
385 on the line that goes through $C$ and $P$, but
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
386 at some other distance $d_2$. Define $d_1=distance(P,C)$
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
387 and $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
388 d_1$, where $pinch$ is a parameter of the filter.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
389 The actual value is given by bilinear interpolation considering the pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
390 around the (non-integer) source position thus found.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
391 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
392 %\vspace{1.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
393 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
394
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
395 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
396
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
397 {\large\bf 2.2 Injecting Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
398 %\subsection{Injecting Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
399 \vspace{2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
400
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
402 %\vspace*{-.2cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
405 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
406 \includegraphics[scale=.4]{images/Motionblur_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
407 {\bf Motion Blur}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
408 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
409 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
410 %\vspace*{.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
411 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
412 has parameters $length$ and $angle$. The value of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
413 a pixel in the final image is approximately the mean of the first $length$ pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
414 found by moving in the $angle$ direction,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
415 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
416 \vspace{5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
417 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
418 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
419
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
420 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
421
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
422 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
423 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
424 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
425 \includegraphics[scale=.4]{images/occlusion_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
426 {\bf Occlusion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
427 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
428 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
429 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
430 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
431 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
432 image and places it over the original {\em occluded}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
433 image. Pixels are combined by taking the max(occluder, occluded),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
434 i.e. keeping the lighter ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
435 The rectangle corners
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
436 are sampled so that larger complexity gives larger rectangles.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
437 The destination position in the occluded image are also sampled
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
438 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
439 This module is skipped with probability 60\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
440 %\vspace{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
441 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
442 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
443
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
444 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
445
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
446 \begin{wrapfigure}[8]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
447 \vspace*{-6mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
448 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
449 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
450 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
451 \includegraphics[scale=.4]{images/Bruitgauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
452 {\bf Gaussian Smoothing}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
453 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
454 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
455 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
456 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
457 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
458 With the {\bf Gaussian smoothing} module,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
459 different regions of the image are spatially smoothed.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
460 This is achieved by first convolving
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
461 the image with an isotropic Gaussian kernel of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
462 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
463 complexity]$ and $[2,2 + 6 \times complexity]$. This filtered image is normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
464 between $0$ and $1$. We also create an isotropic weighted averaging window, of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
465 kernel size, with maximum value at the center. For each image we sample
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
466 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
467 averaging centers between the original image and the filtered one. We
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
468 initialize to zero a mask matrix of the image size. For each selected pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
469 we add to the mask the averaging window centered on it. The final image is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
470 computed from the following element-wise operation: $\frac{image + filtered\_image
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
471 \times mask}{mask+1}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
472 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
473 %\end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
474
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
475 \newpage
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
476
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
477 \vspace*{-9mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
478
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
479 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
480 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
481 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
482 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
483 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
484 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
485 \includegraphics[scale=.4]{images/Permutpixel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
486 {\small\bf Permute Pixels}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
487 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
488 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
489 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
490 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
491 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
492 This module {\bf permutes neighbouring pixels}. It first selects a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
493 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
494 of these pixels is then sequentially exchanged with a random pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
495 among its four nearest neighbors (on its left, right, top or bottom).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
496 This module is skipped with probability 80\%.\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
497 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
498 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
499
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
500 \vspace{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
501
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
502 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
503 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
504 %\vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
505 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
506 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
507 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
508 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
509 \includegraphics[scale=.4]{images/Distorsiongauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
510 {\small \bf Gauss. Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
511 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
512 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
513 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
514 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
515 \vspace*{12mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
516 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
517 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
518 This module is skipped with probability 70\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
519 %\vspace{1.1cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
520 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
521
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
522 \vspace*{1.2cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
523
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
524 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
525 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
526 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
527 \includegraphics[scale=.4]{images/background_other_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
528 {\small \bf Bg Image}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
529 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
530 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
531 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
532 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
533 background image behind the letter, from a randomly chosen natural image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
534 with contrast adjustments depending on $complexity$, to preserve
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
535 more or less of the original character image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
536 %\vspace{.8cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
537 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
538 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
539 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
540
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
541 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
542 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
543 \includegraphics[scale=.4]{images/Poivresel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
544 {\small \bf Salt \& Pepper}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
545 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
546 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
547 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
548 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
549 The number of selected pixels is $0.2 \times complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
550 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
551 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
552 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
553 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
554
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
555 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
556
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
557 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
558 \begin{wrapfigure}[7]{l}{0.14\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
559 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
560 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
561 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
562 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
563 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
564 {\bf Scratches}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
565 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
566 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
567 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
568 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
569 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
570 The {\bf scratches} module places line-like white patches on the image. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
571 lines are heavily transformed images of the digit ``1'' (one), chosen
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
572 at random among 500 such 1 images,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
573 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
574 complexity)^2$ (in degrees), using bi-cubic interpolation.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
575 Two passes of a grey-scale morphological erosion filter
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
576 are applied, reducing the width of the line
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
577 by an amount controlled by $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
578 This module is skipped with probability 85\%. The probabilities
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
579 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
580 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
581
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
582 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
583
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
584 \begin{minipage}[t]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
585 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
586 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
587 {\bf Grey Level \& Contrast}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
588 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
589 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
590 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
591 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
592 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
593 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
594 polarity is inverted with probability 50\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
595 %\vspace{.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
596 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
597 \vspace{2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
598
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
599 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
600
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
601 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
602 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
603 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
604 \caption{Illustration of the pipeline of stochastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
605 transformations applied to the image of a lower-case \emph{t}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
606 (the upper left image). Each image in the pipeline (going from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
607 left to right, first top line, then bottom line) shows the result
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
608 of applying one of the modules in the pipeline. The last image
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
609 (bottom right) is used as training example.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
610 \label{fig:pipeline}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
611 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
612 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
613
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
614 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
615 \section{Experimental Setup}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
616 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
617
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
618 Much previous work on deep learning had been performed on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
619 the MNIST digits task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
620 with 60~000 examples, and variants involving 10~000
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
621 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
622 The focus here is on much larger training sets, from 10 times to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
623 to 1000 times larger, and 62 classes.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
624
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
625 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
626 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
627 and {\bf OCR data} (scanned machine printed characters). Once a character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
628 is sampled from one of these {\em data sources} (chosen randomly), the second step is to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
629 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
630
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
631 To provide a baseline of error rate comparison we also estimate human performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
632 on both the 62-class task and the 10-class digits task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
633 We compare the best Multi-Layer Perceptrons (MLP) against
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
634 the best Stacked Denoising Auto-encoders (SDA), when
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
635 both models' hyper-parameters are selected to minimize the validation set error.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
636 We also provide a comparison against a precise estimate
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
637 of human performance obtained via Amazon's Mechanical Turk (AMT)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
638 service ({\tt http://mturk.com}).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
639 AMT users are paid small amounts
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
640 of money to perform tasks for which human intelligence is required.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
641 An incentive for them to do the job right is that payment can be denied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
642 if the job is not properly done.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
643 Mechanical Turk has been used extensively in natural language processing and vision.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
644 %processing \citep{SnowEtAl2008} and vision
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
645 %\citep{SorokinAndForsyth2008,whitehill09}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
646 AMT users were presented
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
647 with 10 character images at a time (from a test set) and asked to choose 10 corresponding ASCII
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
648 characters. They were forced to choose a single character class (either among the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
649 62 or 10 character classes) for each image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
650 80 subjects classified 2500 images per (dataset,task) pair.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
651 Different humans labelers sometimes provided a different label for the same
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
652 example, and we were able to estimate the error variance due to this effect
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
653 because each image was classified by 3 different persons.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
654 The average error of humans on the 62-class task NIST test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
655 is 18.2\%, with a standard error of 0.1\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
656
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
657 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
658 \subsection{Data Sources}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
659 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
660
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
661 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
662 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
663 {\bf NIST.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
664 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
665 widely used for training and testing character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
666 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
667 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
668 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
669 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
670 The fourth partition (called $hsf_4$, 82587 examples),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
671 experimentally recognized to be the most difficult one, is the one recommended
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
672 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
673 for that purpose. We randomly split the remainder (731,668 examples) into a training set and a validation set for
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
674 model selection.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
675 The performances reported by previous work on that dataset mostly use only the digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
676 Here we use all the classes both in the training and testing phase. This is especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
677 useful to estimate the effect of a multi-task setting.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
678 The distribution of the classes in the NIST training and test sets differs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
679 substantially, with relatively many more digits in the test set, and a more uniform distribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
680 of letters in the test set (whereas in the training set they are distributed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
681 more like in natural text).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
682 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
683
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
684 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
685 {\bf Fonts.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
686 In order to have a good variety of sources we downloaded an important number of free fonts from:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
687 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
688 % TODO: pointless to anonymize, it's not pointing to our work
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
689 Including an operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
690 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
691 directly as input to our models.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
692 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
693
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
694 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
695 {\bf Captchas.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
696 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a Python-based captcha generator library) for
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
697 generating characters of the same format as the NIST dataset. This software is based on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
698 a random character class generator and various kinds of transformations similar to those described in the previous sections.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
699 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
700 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
701 depending on the value of the complexity parameter provided by the user of the data source.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
702 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
703 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
704
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
705 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
706 {\bf OCR data.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
707 A large set (2 million) of scanned, OCRed and manually verified machine-printed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
708 characters where included as an
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
709 additional source. This set is part of a larger corpus being collected by the Image Understanding
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
710 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
711 ({\tt http://www.iupr.com}), and which will be publicly released.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
712 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
713 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
714
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
715 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
716 \subsection{Data Sets}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
717 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
718
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
719 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
720 from one of the 62 character classes. They are obtained from the optional application of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
721 perturbation pipeline to iid samples from the datasources, and they are randomly split into
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
722 training set, validation set, and test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
723 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
724 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
725
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
726 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
727 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
728 \{651668 / 80000 / 82587\} \{training / validation / test\} examples, containing
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
729 upper case, lower case, and digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
730 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
731
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
732 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
733 {\bf P07.} This dataset of upper case, lower case and digit images
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
734 is obtained by taking raw characters from all four of the above sources
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
735 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
736 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
737 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
738 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
739 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
740 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
741
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
742 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
743 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
744 except that we only apply
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
745 transformations from slant to pinch. Therefore, the character is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
746 transformed but no additional noise is added to the image, giving images
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
747 closer to the NIST dataset.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
748 It has \{81,920,000 / 80,000 / 20,000\} \{training / validation / test\} examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
749 obtained from the corresponding NIST sets plus other sources.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
750 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
751
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
752 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
753 \subsection{Models and their Hyperparameters}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
754 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
755
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
756 The experiments are performed using MLPs (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
757 hidden layer) and deep SDAs.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
758 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
759
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
760 {\bf Multi-Layer Perceptrons (MLP).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
761 Whereas previous work had compared deep architectures to both shallow MLPs and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
762 SVMs, we only compared to MLPs here because of the very large datasets used
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
763 (making the use of SVMs computationally challenging because of their quadratic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
764 scaling behavior). Preliminary experiments on training SVMs (libSVM) with subsets of the training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
765 set allowing the program to fit in memory yielded substantially worse results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
766 than those obtained with MLPs. For training on nearly a hundred million examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
767 (with the perturbed data), the MLPs and SDA are much more convenient than
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
768 classifiers based on kernel methods.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
769 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
770 exponentials) on the output layer for estimating $P(class | image)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
771 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
772 Training examples are presented in minibatches of size 20. A constant learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
773 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
774 %through preliminary experiments (measuring performance on a validation set),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
775 %and $0.1$ (which was found to work best) was then selected for optimizing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
776 %the whole training sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
777 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
778
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
779
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
780 {\bf Stacked Denoising Auto-encoders (SDA).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
781 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
782 can be used to initialize the weights of each layer of a deep MLP (with many hidden
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
783 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
784 apparently setting parameters in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
785 basin of attraction of supervised gradient descent yielding better
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
786 generalization~\citep{Erhan+al-2010}. This initial {\em unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
787 pre-training phase} uses all of the training images but not the training labels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
788 Each layer is trained in turn to produce a new representation of its input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
789 (starting from the raw pixels).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
790 It is hypothesized that the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
791 advantage brought by this procedure stems from a better prior,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
792 on the one hand taking advantage of the link between the input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
793 distribution $P(x)$ and the conditional distribution of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
794 $P(y|x)$ (like in semi-supervised learning), and on the other hand
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
795 taking advantage of the expressive power and bias implicit in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
796 deep architecture (whereby complex concepts are expressed as
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
797 compositions of simpler ones through a deep hierarchy).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
798
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
799 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
800 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
801 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
802 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
803 \caption{Illustration of the computations and training criterion for the denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
804 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
805 the layer (i.e. raw input or output of previous layer)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
806 s corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
807 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
808 is compared to the uncorrupted input $x$ through the loss function
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
809 $L_H(x,z)$, whose expected value is approximately minimized during training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
810 by tuning $\theta$ and $\theta'$.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
811 \label{fig:da}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
812 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
813 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
814
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
815 Here we chose to use the Denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
816 Auto-encoder~\citep{VincentPLarochelleH2008} as the building block for
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
817 these deep hierarchies of features, as it is simple to train and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
818 explain (see Figure~\ref{fig:da}, as well as
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
819 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
820 provides efficient inference, and yielded results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
821 comparable or better than RBMs in series of experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
822 \citep{VincentPLarochelleH2008-very-small}. It really corresponds to a Gaussian
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
823 RBM trained by a Score Matching criterion~\cite{Vincent-SM-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
824 During training, a Denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
825 Auto-encoder is presented with a stochastically corrupted version
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
826 of the input and trained to reconstruct the uncorrupted input,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
827 forcing the hidden units to represent the leading regularities in
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
828 the data. Here we use the random binary masking corruption
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
829 (which sets to 0 a random subset of the inputs).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
830 Once it is trained, in a purely unsupervised way,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
831 its hidden units' activations can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
832 be used as inputs for training a second one, etc.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
833 After this unsupervised pre-training stage, the parameters
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
834 are used to initialize a deep MLP, which is fine-tuned by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
835 the same standard procedure used to train them (see previous section).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
836 The SDA hyper-parameters are the same as for the MLP, with the addition of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
837 amount of corruption noise (we used the masking noise process, whereby a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
838 fixed proportion of the input values, randomly selected, are zeroed), and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
839 separate learning rate for the unsupervised pre-training stage (selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
840 from the same above set). The fraction of inputs corrupted was selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
841 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
842 of hidden layers but it was fixed to 3 based on previous work with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
843 SDAs on MNIST~\citep{VincentPLarochelleH2008-very-small}. The size of the hidden
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
844 layers was kept constant across hidden layers, and the best results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
845 were obtained with the largest values that we could experiment
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
846 with given our patience, with 1000 hidden units.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
847
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
848 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
849
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
850 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
851 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
852 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
853 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
854 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
855 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
856 of all models, on NIST and NISTP test sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
857 Right: error rates on NIST test digits only, along with the previous results from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
858 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
859 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
860 \label{fig:error-rates-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
861 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
862 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
863
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
864
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
865 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
866 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
867 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
868 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
869 \caption{Relative improvement in error rate due to self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
870 Left: Improvement (or loss, when negative)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
871 induced by out-of-distribution examples (perturbed data).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
872 Right: Improvement (or loss, when negative) induced by multi-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
873 learning (training on all classes and testing only on either digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
874 upper case, or lower-case). The deep learner (SDA) benefits more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
875 both self-taught learning scenarios, compared to the shallow MLP.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
876 \label{fig:improvements-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
877 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
878 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
879
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
880 \section{Experimental Results}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
881 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
882
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
883 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
884 %\subsection{SDA vs MLP vs Humans}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
885 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
886 The models are either trained on NIST (MLP0 and SDA0),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
887 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
888 on either NIST, NISTP or P07 (regardless of the data set used for training),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
889 either on the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
890 or on the 10-digits task. Training time (including about half
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
891 for unsupervised pre-training, for DAs) on the larger
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
892 datasets is around one day on a GPU (GTX 285).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
893 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
894 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
895 SDA2), along with the previous results on the digits NIST special database
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
896 19 test set from the literature, respectively based on ARTMAP neural
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
897 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
898 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
899 ~\citep{Milgram+al-2005}.% More detailed and complete numerical results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
900 %(figures and tables, including standard errors on the error rates) can be
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
901 %found in Appendix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
902 The deep learner not only outperformed the shallow ones and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
903 previously published performance (in a statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
904 significant way) but when trained with perturbed data
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
905 reaches human performance on both the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
906 and the 10-class (digits) task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
907 17\% error (SDA1) or 18\% error (humans) may seem large but a large
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
908 majority of the errors from humans and from SDA1 are from out-of-context
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
909 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
910 ``c'' and a ``C'' are often indistinguishible).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
911
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
912 In addition, as shown in the left of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
913 Figure~\ref{fig:improvements-charts}, the relative improvement in error
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
914 rate brought by self-taught learning is greater for the SDA, and these
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
915 differences with the MLP are statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
916 significant.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
917 The left side of the figure shows the improvement to the clean
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
918 NIST test set error brought by the use of out-of-distribution examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
919 (i.e. the perturbed examples examples from NISTP or P07),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
920 over the models trained exclusively on NIST (respectively SDA0 and MLP0).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
921 Relative percent change is measured by taking
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
922 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
923 The right side of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
924 Figure~\ref{fig:improvements-charts} shows the relative improvement
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
925 brought by the use of a multi-task setting, in which the same model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
926 trained for more classes than the target classes of interest (i.e. training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
927 with all 62 classes when the target classes are respectively the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
928 lower-case, or upper-case characters). Again, whereas the gain from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
929 multi-task setting is marginal or negative for the MLP, it is substantial
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
930 for the SDA. Note that to simplify these multi-task experiments, only the original
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
931 NIST dataset is used. For example, the MLP-digits bar shows the relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
932 percent improvement in MLP error rate on the NIST digits test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
933 is $100\% \times$ (single-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
934 model's error / multi-task model's error - 1). The single-task model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
935 trained with only 10 outputs (one per digit), seeing only digit examples,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
936 whereas the multi-task model is trained with 62 outputs, with all 62
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
937 character classes as examples. Hence the hidden units are shared across
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
938 all tasks. For the multi-task model, the digit error rate is measured by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
939 comparing the correct digit class with the output class associated with the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
940 maximum conditional probability among only the digit classes outputs. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
941 setting is similar for the other two target classes (lower case characters
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
942 and upper case characters). Note however that some types of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
943 (NISTP) help more than others (P07) when testing on the clean images.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
944 %%\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
945 %\subsection{Perturbed Training Data More Helpful for SDA}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
946 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
947
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
948 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
949 %\subsection{Multi-Task Learning Effects}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
950 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
951
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
952 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
953 As previously seen, the SDA is better able to benefit from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
954 transformations applied to the data than the MLP. In this experiment we
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
955 define three tasks: recognizing digits (knowing that the input is a digit),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
956 recognizing upper case characters (knowing that the input is one), and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
957 recognizing lower case characters (knowing that the input is one). We
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
958 consider the digit classification task as the target task and we want to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
959 evaluate whether training with the other tasks can help or hurt, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
960 whether the effect is different for MLPs versus SDAs. The goal is to find
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
961 out if deep learning can benefit more (or less) from multiple related tasks
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
962 (i.e. the multi-task setting) compared to a corresponding purely supervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
963 shallow learner.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
964
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
965 We use a single hidden layer MLP with 1000 hidden units, and a SDA
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
966 with 3 hidden layers (1000 hidden units per layer), pre-trained and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
967 fine-tuned on NIST.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
968
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
969 Our results show that the MLP benefits marginally from the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
970 in the case of digits (5\% relative improvement) but is actually hurt in the case
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
971 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
972 On the other hand the SDA benefited from the multi-task setting, with relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
973 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
974 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
975 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
976
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
977
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
978 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
979 \section{Conclusions and Discussion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
980 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
981
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
982 We have found that the self-taught learning framework is more beneficial
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
983 to a deep learner than to a traditional shallow and purely
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
984 supervised learner. More precisely,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
985 the answers are positive for all the questions asked in the introduction.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
986 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
987
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
988 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
989 {\bf Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
990 MNIST digits generalize to a much larger and richer (but similar)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
991 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
992 Yes, the SDA {\em systematically outperformed the MLP and all the previously
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
993 published results on this dataset} (the ones that we are aware of), {\em in fact reaching human-level
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
994 performance} at around 17\% error on the 62-class task and 1.4\% on the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
995 and beating previously published results on the same data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
996
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
997 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
998 {\bf To what extent do self-taught learning scenarios help deep learners,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
999 and do they help them more than shallow supervised ones}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1000 We found that distorted training examples not only made the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1001 classifier better on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1002 the {\em original clean examples}, and more importantly and more novel,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1003 that deep architectures benefit more from such {\em out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1004 examples. MLPs were helped by perturbed training examples when tested on perturbed input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1005 images (65\% relative improvement on NISTP)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1006 but only marginally helped (5\% relative improvement on all classes)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1007 or even hurt (10\% relative loss on digits)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1008 with respect to clean examples. On the other hand, the deep SDAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1009 were significantly boosted by these out-of-distribution examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1010 Similarly, whereas the improvement due to the multi-task setting was marginal or
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1011 negative for the MLP (from +5.6\% to -3.6\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1012 it was quite significant for the SDA (from +13\% to +27\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1013 which may be explained by the arguments below.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1014 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1015
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1016 In the original self-taught learning framework~\citep{RainaR2007}, the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1017 out-of-sample examples were used as a source of unsupervised data, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1018 experiments showed its positive effects in a \emph{limited labeled data}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1019 scenario. However, many of the results by \citet{RainaR2007} (who used a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1020 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1021 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1022 We note instead that, for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1023 architectures, our experiments show that such a positive effect is accomplished
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1024 even in a scenario with a \emph{large number of labeled examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1025 i.e., here, the relative gain of self-taught learning and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1026 out-of-distribution examples is probably preserved
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1027 in the asymptotic regime. However, note that in our perturbation experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1028 (but not in our multi-task experiments),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1029 even the out-of-distribution examples are labeled, unlike in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1030 earlier self-taught learning experiments~\citep{RainaR2007}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1031
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1032 {\bf Why would deep learners benefit more from the self-taught learning framework}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1033 The key idea is that the lower layers of the predictor compute a hierarchy
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1034 of features that can be shared across tasks or across variants of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1035 input distribution. A theoretical analysis of generalization improvements
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1036 due to sharing of intermediate features across tasks already points
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1037 towards that explanation~\cite{baxter95a}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1038 Intermediate features that can be used in different
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1039 contexts can be estimated in a way that allows to share statistical
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1040 strength. Features extracted through many levels are more likely to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1041 be more abstract and more invariant to some of the factors of variation
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1042 in the underlying distribution (as the experiments in~\citet{Goodfellow2009} suggest),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1043 increasing the likelihood that they would be useful for a larger array
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1044 of tasks and input conditions.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1045 Therefore, we hypothesize that both depth and unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1046 pre-training play a part in explaining the advantages observed here, and future
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1047 experiments could attempt at teasing apart these factors.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1048 And why would deep learners benefit from the self-taught learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1049 scenarios even when the number of labeled examples is very large?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1050 We hypothesize that this is related to the hypotheses studied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1051 in~\citet{Erhan+al-2010}. In~\citet{Erhan+al-2010}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1052 it was found that online learning on a huge dataset did not make the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1053 advantage of the deep learning bias vanish, and a similar phenomenon
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1054 may be happening here. We hypothesize that unsupervised pre-training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1055 of a deep hierarchy with self-taught learning initializes the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1056 model in the basin of attraction of supervised gradient descent
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1057 that corresponds to better generalization. Furthermore, such good
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1058 basins of attraction are not discovered by pure supervised learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1059 (with or without self-taught settings), and more labeled examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1060 does not allow the model to go from the poorer basins of attraction discovered
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1061 by the purely supervised shallow models to the kind of better basins associated
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1062 with deep learning and self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1063
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1064 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1065 can be executed on-line at {\tt http://deep.host22.com}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1066
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1067 \newpage
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1068 {
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1069 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1070 %\bibliographystyle{plainnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1071 \bibliographystyle{unsrtnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1072 %\bibliographystyle{apalike}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1073 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1074
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1075
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1076 \end{document}