Mercurial > ift6266

annotate writeup/nips2010_cameraready.tex @ 609:7513b3a20f5f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
spotlight pdf
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Wed, 01 Dec 2010 18:04:19 -0500
parents d840139444fe
children
rev   line source
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass{article} % For LaTeX2e
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage{nips10submit_e,times}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 \usepackage{wrapfig}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4 \usepackage{amsthm,amsmath,bbm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 \usepackage[psamsfonts]{amssymb}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 \usepackage{algorithm,algorithmic}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \usepackage[utf8]{inputenc}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \usepackage{graphicx,subfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9 \usepackage[numbers]{natbib}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
11 \addtolength{\textwidth}{10mm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
12 \addtolength{\textheight}{10mm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
13 \addtolength{\topmargin}{-5mm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
14 \addtolength{\evensidemargin}{-5mm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
15 \addtolength{\oddsidemargin}{-5mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17 %\setlength\parindent{0mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 \author{
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21 Frédéric Bastien,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 Yoshua Bengio,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 Arnaud Bergeron,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 Nicolas Boulanger-Lewandowski,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 Thomas Breuel,\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 {\bf Youssouf Chherawala,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 Moustapha Cisse,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 Myriam Côté,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29 Dumitru Erhan,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 Jeremy Eustache,}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31 {\bf Xavier Glorot,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 Xavier Muller,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 Sylvain Pannetier Lebeuf,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 Razvan Pascanu,} \\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 {\bf Salah Rifai,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 Francois Savard,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 Guillaume Sicard}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 Dept. IRO, U. Montreal
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 \begin{document}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 %\makeanontitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44 \maketitle
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 \begin{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 Recent theoretical and empirical work in statistical machine learning has
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 demonstrated the importance of learning algorithms for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 architectures, i.e., function classes obtained by composing multiple
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 non-linear transformations. Self-taught learning (exploiting unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 examples or examples from other distributions) has already been applied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 to deep learners, but mostly to show the advantage of unlabeled
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 examples. Here we explore the advantage brought by {\em out-of-distribution examples}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 For this purpose we
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 developed a powerful generator of stochastic variations and noise
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
57 processes for character images, including not only affine transformations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 but also slant, local elastic deformations, changes in thickness,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59 background images, grey level changes, contrast, occlusion, and various
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 types of noise. The out-of-distribution examples are obtained from these
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 highly distorted images or by including examples of object classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 different from those in the target test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 We show that {\em deep learners benefit
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 more from them than a corresponding shallow learner}, at least in the area of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 handwritten character recognition. In fact, we show that they reach
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 human-level performance on both handwritten digit classification and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 62-class handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 \end{abstract}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 \section{Introduction}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 {\bf Deep Learning} has emerged as a promising new area of research in
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
75 statistical machine learning~\citep{Hinton06}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
76 (see \citet{Bengio-2009} for a review).
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 Learning algorithms for deep architectures are centered on the learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 of useful representations of data, which are better suited to the task at hand,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 and are organized in a hierarchy with multiple levels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 This is in part inspired by observations of the mammalian visual cortex,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 which consists of a chain of processing elements, each of which is associated with a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 different representation of the raw visual input. In fact,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 it was found recently that the features learnt in deep architectures resemble
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 those observed in the first two of these stages (in areas V1 and V2
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 of visual cortex)~\citep{HonglakL2008}, and that they become more and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86 more invariant to factors of variation (such as camera movement) in
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 higher layers~\citep{Goodfellow2009}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88 It has been hypothesized that learning a hierarchy of features increases the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89 ease and practicality of developing representations that are at once
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 tailored to specific tasks, yet are able to borrow statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
92 feature representation can lead to higher-level (more abstract, more
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
93 general) features that are more robust to unanticipated sources of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94 variance extant in real data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
95
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 {\bf Self-taught learning}~\citep{RainaR2007} is a paradigm that combines principles
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
98 that are unlabeled and possibly come from a distribution different from the target
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
99 distribution, e.g., from other classes than those of interest.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
100 It has already been shown that deep learners can clearly take advantage of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
101 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
102 but more needs to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 of {\em out-of-distribution} examples and of the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 (one exception is~\citep{CollobertR2008}, which uses a different kind
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 of learning algorithm). In particular the {\em relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106 advantage} of deep learning for these settings has not been evaluated.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 The hypothesis discussed in the conclusion is that a deep hierarchy of features
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108 may be better able to provide sharing of statistical strength
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
109 between different regions in input space or different tasks.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
111 Previous comparative experimental results with stacking of RBMs and DAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
112 to build deep supervised predictors had shown that they could outperform
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 shallow architectures in a variety of settings, especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 when the data involves complex interactions between many factors of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
115 variation~\citep{LarochelleH2007,Bengio-2009}. Other experiments have suggested
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116 that the unsupervised layer-wise pre-training acted as a useful
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
117 prior~\citep{Erhan+al-2010} that allows one to initialize a deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118 neural network in a relatively much smaller region of parameter space,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
119 corresponding to better generalization.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
120
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
121 To further the understanding of the reasons for the good performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122 observed with deep learners, we focus here on the following {\em hypothesis}:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
123 intermediate levels of representation, especially when there are
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124 more such levels, can be exploited to {\bf share
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
125 statistical strength across different but related types of examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126 such as examples coming from other tasks than the task of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 (the multi-task setting), or examples coming from an overlapping
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
128 but different distribution (images with different kinds of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
129 and noises, here). This is consistent with the hypotheses discussed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130 in~\citet{Bengio-2009} regarding the potential advantage
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
131 of deep learning and the idea that more levels of representation can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
132 give rise to more abstract, more general features of the raw input.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
134 This hypothesis is related to the
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
135 {\bf self-taught learning} setting~\citep{RainaR2007}, which combines principles
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 of semi-supervised and multi-task learning: the learner can exploit examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 that are unlabeled and possibly come from a distribution different from the target
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
138 distribution, e.g., from classes other than those of interest.
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
139 It has already been shown that deep learners can take advantage of
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 but more needed to be done to explore the impact
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
142 of {\em out-of-distribution} examples and of the {\em multi-task} setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143 (one exception is~\citep{CollobertR2008}, which shares and uses unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 pre-training only with the first layer). In particular the {\em relative
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
145 advantage of deep learning} for these settings had not been evaluated.
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
146
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
147
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
148 %
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
149 The {\bf main claim} of this paper is that deep learners (with several levels of representation) can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
150 {\bf benefit more from out-of-distribution examples than shallow learners} (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
151 level), both in the context of the multi-task setting and from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
152 perturbed examples. Because we are able to improve on state-of-the-art
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 performance and reach human-level performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
154 on a large-scale task, we consider that this paper is also a contribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
155 to advance the application of machine learning to handwritten character recognition.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
156 More precisely, we ask and answer the following questions:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
157
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
158 %\begin{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
159 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
160 Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
161 MNIST digit images generalize to the setting of a similar but much larger and richer
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162 dataset, the NIST special database 19, with 62 classes and around 800k examples?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 To what extent does the perturbation of input images (e.g. adding
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
166 noise, affine transformations, background images) make the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 classifiers better not only on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
168 the {\em original clean examples}? We study this question in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169 context of the 62-class and 10-class tasks of the NIST special database 19.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
170
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
172 Do deep architectures {\em benefit {\bf more} from such out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 examples, in particular do they benefit more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174 examples that are perturbed versions of the examples from the task of interest?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177 Similarly, does the feature learning step in deep learning algorithms benefit {\bf more}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178 from training with moderately {\em different classes} (i.e. a multi-task learning scenario) than
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 a corresponding shallow and purely supervised architecture?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 We train on 62 classes and test on 10 (digits) or 26 (upper case or lower case)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181 to answer this question.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 %\end{enumerate}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
184 Our experimental results provide positive evidence towards all of these questions,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185 as well as {\bf classifiers that reach human-level performance on 62-class isolated character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186 recognition and beat previously published results on the NIST dataset (special database 19)}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187 To achieve these results, we introduce in the next section a sophisticated system
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 for stochastically transforming character images and then explain the methodology,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
189 which is based on training with or without these transformed images and testing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190 clean ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
191 Code for generating these transformations as well as for the deep learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
192 algorithms are made available at {\tt http://hg.assembla.com/ift6266}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 \vspace*{-3mm}
606
bd7d11089a47 more fix to newpage command.
Frederic Bastien <nouiz@nouiz.org>
parents: 605
diff changeset
195 %%\newpage
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
196 \section{Perturbed and Transformed Character Images}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 \label{s:perturbations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
200 %\begin{minipage}[h]{\linewidth}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
201 \begin{wrapfigure}[8]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
202 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
203 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 \includegraphics[scale=.4]{images/Original.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 {\bf Original}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
209 %\vspace{0.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
210 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212 This section describes the different transformations we used to stochastically
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
213 transform $32 \times 32$ source images (such as the one on the left)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
214 in order to obtain data from a larger distribution which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 covers a domain substantially larger than the clean characters distribution from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 which we start.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
217 Although character transformations have been used before to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
218 improve character recognizers, this effort is on a large scale both
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
219 in number of classes and in the complexity of the transformations, hence
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
220 in the complexity of the learning task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
221 More details can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
222 be found in this technical report~\citep{ARXIV-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
223 The code for these transformations (mostly python) is available at
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
224 {\tt http://hg.assembla.com/ift6266}. All the modules in the pipeline share
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
225 a global control parameter ($0 \le complexity \le 1$) modulating the
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
226 amount of deformation or noise.
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
227 There are two main parts in the pipeline. The first one,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
228 from thickness to pinch, performs transformations. The second
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
229 part, from blur to contrast, adds different kinds of noise.
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
230 %\end{minipage}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
231
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
232 %\newpage
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
233 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
234 %\subsection{Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
235 {\large\bf 2.1 Transformations}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
236 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
237
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
238
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
239 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
240 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
241 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
242 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
243 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
244 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
245 \includegraphics[scale=.4]{images/Thick_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
246 {\bf Thickness}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
247 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
248 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
249 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
250 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
251 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
252 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
253 are applied. The neighborhood of each pixel is multiplied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
254 element-wise with a {\em structuring element} matrix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
255 The pixel value is replaced by the maximum or the minimum of the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
256 matrix, respectively for dilation or erosion. Ten different structural elements with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
257 increasing dimensions (largest is $5\times5$) were used. For each image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
258 randomly sample the operator type (dilation or erosion) with equal probability and one structural
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
259 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
260 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
261 A neutral element (no transformation)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
262 is always present in the set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
263 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
264 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
265 \vspace*{3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
266
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
267 \begin{minipage}[h]{\linewidth}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
268 \begin{wrapfigure}[7]{l}{0.15\textwidth}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
269 %\begin{minipage}[b]{0.14\linewidth}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
270 %\centering
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
271 \begin{center}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
272 \vspace*{-5mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
273 \includegraphics[scale=.4]{images/Slant_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
274 {\bf Slant}
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
275 \end{center}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
276 \end{wrapfigure}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
277
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
278 %\end{minipage}%
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
279 %\hspace{0.3cm}
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
280 %\begin{minipage}[b]{0.83\linewidth}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
281 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
282 To produce {\bf slant}, each row of the image is shifted
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
283 proportionally to its height: $shift = round(slant \times height)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
284 $slant \sim U[-complexity,complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
285 The shift is randomly chosen to be either to the left or to the right.
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
286 %\vspace{8mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
287 \end{minipage}
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
288 \vspace*{10mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
289
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
290 \begin{minipage}[h]{\linewidth}
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
291 %\begin{minipage}[b]{0.14\linewidth}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
292 %\centering
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
293 \begin{wrapfigure}[7]{l}{0.15\textwidth}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
294 \begin{center}
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
295 \vspace*{-5mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
296 \includegraphics[scale=.4]{images/Affine_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
297 {\small {\bf Affine \mbox{Transformation}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
298 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
299 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
300 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
301 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
302 A $2 \times 3$ {\bf affine transform} matrix (with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
303 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
304 Output pixel $(x,y)$ takes the value of input pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
305 nearest to $(ax+by+c,dx+ey+f)$,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
306 producing scaling, translation, rotation and shearing.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
307 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
308 forbid large rotations (to avoid confusing classes) but to give good
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
309 variability of the transformation: $a$ and $d$ $\sim U[1-3
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
310 complexity,1+3\,complexity]$, $b$ and $e$ $\sim U[-3 \,complexity,3\,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
311 complexity]$, and $c$ and $f \sim U[-4 \,complexity, 4 \,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
312 complexity]$.\\
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
313 %\end{minipage}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
314 \end{minipage}
605
63f838479510 fix transformation description printing.
Frederic Bastien <nouiz@nouiz.org>
parents: 604
diff changeset
315 \vspace*{3mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
316
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
317 \vspace*{-4.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
318
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
319 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
320 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
321 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
322 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
323 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
324 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
325 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
326 {\bf Local Elastic Deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
327 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
328 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
329 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
330 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
331 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
332 The {\bf local elastic deformation}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
333 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
334 which provides more details.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
335 The intensity of the displacement fields is given by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
336 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
337 convolved with a Gaussian 2D kernel (resulting in a blur) of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
338 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
339 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
340 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
341
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
342 \vspace*{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
343
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
344 %\begin{minipage}[b]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
345 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
346 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
347 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
348 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
349 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
350 \includegraphics[scale=.4]{images/Pinch_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
351 {\bf Pinch}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
352 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
353 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
354 %\vspace{.6cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
355 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
356 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
357 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl set to 0.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
358 A pinch is ``similar to projecting the image onto an elastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
359 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
360 For a square input image, draw a radius-$r$ disk
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
361 around its center $C$. Any pixel $P$ belonging to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
362 that disk has its value replaced by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
363 the value of a ``source'' pixel in the original image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
364 on the line that goes through $C$ and $P$, but
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
365 at some other distance $d_2$. Define $d_1=distance(P,C)$
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
366 and $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
367 d_1$, where $pinch$ is a parameter of the filter.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
368 The actual value is given by bilinear interpolation considering the pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
369 around the (non-integer) source position thus found.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
370 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
371 %\vspace{1.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
372 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
373
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
374 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
375
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
376 {\large\bf 2.2 Injecting Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
377 %\subsection{Injecting Noise}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
378 %\vspace{2mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
379
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
380 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
381 %\vspace*{-.2cm}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
382 %\begin{minipage}[t]{0.14\linewidth}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
383 \begin{wrapfigure}[8]{l}{0.15\textwidth}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
384 \begin{center}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
385 \vspace*{-5mm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
386 %\vspace*{-2mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
387 \includegraphics[scale=.4]{images/Motionblur_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
388 {\bf Motion Blur}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
389 %\end{minipage}%
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
390 \end{center}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
391 \end{wrapfigure}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
392 %\hspace{0.3cm}
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
393 %\begin{minipage}[t]{0.83\linewidth}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
394 %\vspace*{.5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
395 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
396 has parameters $length$ and $angle$. The value of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
397 a pixel in the final image is approximately the mean of the first $length$ pixels
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
398 found by moving in the $angle$ direction,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
399 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
400 \vspace{5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
401 \end{minipage}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
402 %\end{minipage}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
403
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
404 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
405
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
406 \begin{minipage}[h]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
407 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
408 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
409 \includegraphics[scale=.4]{images/occlusion_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
410 {\bf Occlusion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
411 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
412 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
413 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
414 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
415 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
416 image and places it over the original {\em occluded}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
417 image. Pixels are combined by taking the max(occluder, occluded),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
418 i.e. keeping the lighter ones.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
419 The rectangle corners
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
420 are sampled so that larger complexity gives larger rectangles.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
421 The destination position in the occluded image are also sampled
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
422 according to a normal distribution (more details in~\citet{ARXIV-2010}).
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
423 This module is skipped with probability 60\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
424 %\vspace{7mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
425 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
426 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
427
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
428 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
429
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
430 \begin{wrapfigure}[8]{l}{0.15\textwidth}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
431 \vspace*{-3mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
432 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
433 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
434 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
435 \includegraphics[scale=.4]{images/Bruitgauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
436 {\bf Gaussian Smoothing}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
437 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
438 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
439 %\vspace{.5cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
440 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
441 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
442 With the {\bf Gaussian smoothing} module,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
443 different regions of the image are spatially smoothed.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
444 This is achieved by first convolving
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
445 the image with an isotropic Gaussian kernel of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
446 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
447 complexity]$ and $[2,2 + 6 \times complexity]$. This filtered image is normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
448 between $0$ and $1$. We also create an isotropic weighted averaging window, of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
449 kernel size, with maximum value at the center. For each image we sample
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
450 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
451 averaging centers between the original image and the filtered one. We
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
452 initialize to zero a mask matrix of the image size. For each selected pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
453 we add to the mask the averaging window centered on it. The final image is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
454 computed from the following element-wise operation: $\frac{image + filtered\_image
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
455 \times mask}{mask+1}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
456 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
457 %\end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
458
606
bd7d11089a47 more fix to newpage command.
Frederic Bastien <nouiz@nouiz.org>
parents: 605
diff changeset
459 %\newpage
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
460
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
461 \vspace*{1mm}
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
462
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
463 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
464 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
465 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
466 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
467 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
468 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
469 \includegraphics[scale=.4]{images/Permutpixel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
470 {\small\bf Permute Pixels}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
471 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
472 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
473 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
474 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
475 %\vspace*{-20mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
476 This module {\bf permutes neighbouring pixels}. It first selects a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
477 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
478 of these pixels is then sequentially exchanged with a random pixel
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
479 among its four nearest neighbors (on its left, right, top or bottom).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
480 This module is skipped with probability 80\%.\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
481 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
482 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
483
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
484 \vspace{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
485
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
486 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
487 \begin{wrapfigure}[7]{l}{0.15\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
488 %\vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
489 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
490 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
491 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
492 \vspace*{-5mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
493 \includegraphics[scale=.4]{images/Distorsiongauss_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
494 {\small \bf Gauss. Noise}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
495 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
496 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
497 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
498 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
499 \vspace*{12mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
500 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
501 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
502 This module is skipped with probability 70\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
503 %\vspace{1.1cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
504 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
505
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
506 \vspace*{1.2cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
507
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
508 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
509 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
510 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
511 \includegraphics[scale=.4]{images/background_other_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
512 {\small \bf Bg Image}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
513 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
514 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
515 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
516 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
517 background image behind the letter, from a randomly chosen natural image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
518 with contrast adjustments depending on $complexity$, to preserve
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
519 more or less of the original character image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
520 %\vspace{.8cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
521 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
522 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
523 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
524
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
525 \begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
526 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
527 \includegraphics[scale=.4]{images/Poivresel_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
528 {\small \bf Salt \& Pepper}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
529 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
530 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
531 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
532 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
533 The number of selected pixels is $0.2 \times complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
534 This module is skipped with probability 75\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
535 %\vspace{.9cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
536 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
537 %\vspace{-.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
538
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
539 \vspace{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
540
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
541 \begin{minipage}[t]{\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
542 \begin{wrapfigure}[7]{l}{0.14\textwidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
543 %\begin{minipage}[t]{0.14\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
544 %\centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
545 \begin{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
546 \vspace*{-4mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
547 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
548 {\bf Scratches}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
549 %\end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
550 \end{center}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
551 \end{wrapfigure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
552 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
553 %\vspace{.4cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
554 The {\bf scratches} module places line-like white patches on the image. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
555 lines are heavily transformed images of the digit ``1'' (one), chosen
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
556 at random among 500 such 1 images,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
557 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
558 complexity)^2$ (in degrees), using bi-cubic interpolation.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
559 Two passes of a grey-scale morphological erosion filter
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
560 are applied, reducing the width of the line
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
561 by an amount controlled by $complexity$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
562 This module is skipped with probability 85\%. The probabilities
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
563 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
564 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
565
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
566 \vspace*{1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
567
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
568 \begin{minipage}[t]{0.25\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
569 \centering
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
570 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.png}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
571 {\bf Grey Level \& Contrast}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
572 \end{minipage}%
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
573 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
574 \vspace*{-18mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
575 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
576 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
577 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
578 polarity is inverted with probability 50\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
579 %\vspace{.7cm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
580 \end{minipage}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
581 \vspace{2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
582
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
583 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
584 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
585 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
586 \caption{Illustration of the pipeline of stochastic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
587 transformations applied to the image of a lower-case \emph{t}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
588 (the upper left image). Each image in the pipeline (going from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
589 left to right, first top line, then bottom line) shows the result
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
590 of applying one of the modules in the pipeline. The last image
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
591 (bottom right) is used as training example.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
592 \label{fig:pipeline}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
593 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
594 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
595
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
596 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
597 \section{Experimental Setup}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
598 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
599
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
600 Much previous work on deep learning had been performed on
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
601 the MNIST digits task
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
602 with 60~000 examples, and variants involving 10~000
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
603 examples~\citep{VincentPLarochelleH2008-very-small}.
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
604 The focus here is on much larger training sets, from 10 times to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
605 to 1000 times larger, and 62 classes.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
606
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
607 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
608 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
609 and {\bf OCR data} (scanned machine printed characters). Once a character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
610 is sampled from one of these {\em data sources} (chosen randomly), the second step is to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
611 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
612
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
613 To provide a baseline of error rate comparison we also estimate human performance
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
614 on both the 62-class task and the 10-class digits task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
615 We compare the best Multi-Layer Perceptrons (MLP) against
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
616 the best Stacked Denoising Auto-encoders (SDA), when
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
617 both models' hyper-parameters are selected to minimize the validation set error.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
618 We also provide a comparison against a precise estimate
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
619 of human performance obtained via Amazon's Mechanical Turk (AMT)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
620 service ({\tt http://mturk.com}).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
621 AMT users are paid small amounts
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
622 of money to perform tasks for which human intelligence is required.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
623 An incentive for them to do the job right is that payment can be denied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
624 if the job is not properly done.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
625 Mechanical Turk has been used extensively in natural language processing and vision.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
626 %processing \citep{SnowEtAl2008} and vision
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
627 %\citep{SorokinAndForsyth2008,whitehill09}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
628 AMT users were presented
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
629 with 10 character images at a time (from a test set) and asked to choose 10 corresponding ASCII
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
630 characters. They were forced to choose a single character class (either among the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
631 62 or 10 character classes) for each image.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
632 80 subjects classified 2500 images per (dataset,task) pair.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
633 Different humans labelers sometimes provided a different label for the same
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
634 example, and we were able to estimate the error variance due to this effect
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
635 because each image was classified by 3 different persons.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
636 The average error of humans on the 62-class task NIST test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
637 is 18.2\%, with a standard error of 0.1\%.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
638
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
639 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
640 \subsection{Data Sources}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
641 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
642
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
643 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
644 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
645 {\bf NIST.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
646 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
647 widely used for training and testing character
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
648 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
649 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
650 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
651 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
652 The fourth partition (called $hsf_4$, 82587 examples),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
653 experimentally recognized to be the most difficult one, is the one recommended
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
654 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
655 for that purpose. We randomly split the remainder (731,668 examples) into a training set and a validation set for
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
656 model selection.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
657 The performances reported by previous work on that dataset mostly use only the digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
658 Here we use all the classes both in the training and testing phase. This is especially
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
659 useful to estimate the effect of a multi-task setting.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
660 The distribution of the classes in the NIST training and test sets differs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
661 substantially, with relatively many more digits in the test set, and a more uniform distribution
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
662 of letters in the test set (whereas in the training set they are distributed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
663 more like in natural text).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
664 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
665
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
666 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
667 {\bf Fonts.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
668 In order to have a good variety of sources we downloaded an important number of free fonts from:
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
669 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
670 % TODO: pointless to anonymize, it's not pointing to our work
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
671 Including an operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
672 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
673 directly as input to our models.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
674 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
675
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
676 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
677 {\bf Captchas.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
678 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a Python-based captcha generator library) for
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
679 generating characters of the same format as the NIST dataset. This software is based on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
680 a random character class generator and various kinds of transformations similar to those described in the previous sections.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
681 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
682 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
683 depending on the value of the complexity parameter provided by the user of the data source.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
684 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
685 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
686
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
687 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
688 {\bf OCR data.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
689 A large set (2 million) of scanned, OCRed and manually verified machine-printed
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
690 characters where included as an
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
691 additional source. This set is part of a larger corpus being collected by the Image Understanding
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
692 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
693 ({\tt http://www.iupr.com}), and which will be publicly released.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
694 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
695 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
696
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
697 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
698 \subsection{Data Sets}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
699 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
700
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
701 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
702 from one of the 62 character classes. They are obtained from the optional application of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
703 perturbation pipeline to iid samples from the datasources, and they are randomly split into
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
704 training set, validation set, and test set.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
705 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
706 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
707
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
708 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
709 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
710 \{651668 / 80000 / 82587\} \{training / validation / test\} examples, containing
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
711 upper case, lower case, and digits.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
712 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
713
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
714 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
715 {\bf P07.} This dataset of upper case, lower case and digit images
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
716 is obtained by taking raw characters from all four of the above sources
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
717 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
718 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
719 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
720 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
721 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
722 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
723
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
724 %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
725 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
726 except that we only apply
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
727 transformations from slant to pinch. Therefore, the character is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
728 transformed but no additional noise is added to the image, giving images
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
729 closer to the NIST dataset.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
730 It has \{81,920,000 / 80,000 / 20,000\} \{training / validation / test\} examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
731 obtained from the corresponding NIST sets plus other sources.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
732 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
733
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
734 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
735 \subsection{Models and their Hyperparameters}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
736 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
737
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
738 The experiments are performed using MLPs (with a single
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
739 hidden layer) and deep SDAs.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
740 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
741
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
742 {\bf Multi-Layer Perceptrons (MLP).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
743 Whereas previous work had compared deep architectures to both shallow MLPs and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
744 SVMs, we only compared to MLPs here because of the very large datasets used
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
745 (making the use of SVMs computationally challenging because of their quadratic
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
746 scaling behavior). Preliminary experiments on training SVMs (libSVM) with subsets of the training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
747 set allowing the program to fit in memory yielded substantially worse results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
748 than those obtained with MLPs. For training on nearly a hundred million examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
749 (with the perturbed data), the MLPs and SDA are much more convenient than
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
750 classifiers based on kernel methods.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
751 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
752 exponentials) on the output layer for estimating $P(class | image)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
753 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
754 Training examples are presented in minibatches of size 20. A constant learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
755 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
756 %through preliminary experiments (measuring performance on a validation set),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
757 %and $0.1$ (which was found to work best) was then selected for optimizing on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
758 %the whole training sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
759 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
760
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
761
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
762 {\bf Stacked Denoising Auto-encoders (SDA).}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
763 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
764 can be used to initialize the weights of each layer of a deep MLP (with many hidden
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
765 layers)
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
766 apparently setting parameters in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
767 basin of attraction of supervised gradient descent yielding better
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
768 generalization~\citep{Erhan+al-2010}. This initial {\em unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
769 pre-training phase} uses all of the training images but not the training labels.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
770 Each layer is trained in turn to produce a new representation of its input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
771 (starting from the raw pixels).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
772 It is hypothesized that the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
773 advantage brought by this procedure stems from a better prior,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
774 on the one hand taking advantage of the link between the input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
775 distribution $P(x)$ and the conditional distribution of interest
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
776 $P(y|x)$ (like in semi-supervised learning), and on the other hand
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
777 taking advantage of the expressive power and bias implicit in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
778 deep architecture (whereby complex concepts are expressed as
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
779 compositions of simpler ones through a deep hierarchy).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
780
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
781 \iffalse
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
782 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
783 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
784 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
785 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
786 \caption{Illustration of the computations and training criterion for the denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
787 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
788 the layer (i.e. raw input or output of previous layer)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
789 s corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
790 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
791 is compared to the uncorrupted input $x$ through the loss function
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
792 $L_H(x,z)$, whose expected value is approximately minimized during training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
793 by tuning $\theta$ and $\theta'$.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
794 \label{fig:da}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
795 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
796 \end{figure}
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
797 \fi
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
798
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
799 Here we chose to use the Denoising
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
800 Auto-encoder~\citep{VincentPLarochelleH2008-very-small} as the building block for
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
801 these deep hierarchies of features, as it is simple to train and
607
d840139444fe NIPS workshop spotlight
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 606
diff changeset
802 explain (see % Figure~\ref{fig:da}, as well as
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
803 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
804 provides efficient inference, and yielded results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
805 comparable or better than RBMs in series of experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
806 \citep{VincentPLarochelleH2008-very-small}. It really corresponds to a Gaussian
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
807 RBM trained by a Score Matching criterion~\cite{Vincent-SM-2010}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
808 During training, a Denoising
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
809 Auto-encoder is presented with a stochastically corrupted version
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
810 of the input and trained to reconstruct the uncorrupted input,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
811 forcing the hidden units to represent the leading regularities in
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
812 the data. Here we use the random binary masking corruption
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
813 (which sets to 0 a random subset of the inputs).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
814 Once it is trained, in a purely unsupervised way,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
815 its hidden units' activations can
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
816 be used as inputs for training a second one, etc.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
817 After this unsupervised pre-training stage, the parameters
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
818 are used to initialize a deep MLP, which is fine-tuned by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
819 the same standard procedure used to train them (see previous section).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
820 The SDA hyper-parameters are the same as for the MLP, with the addition of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
821 amount of corruption noise (we used the masking noise process, whereby a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
822 fixed proportion of the input values, randomly selected, are zeroed), and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
823 separate learning rate for the unsupervised pre-training stage (selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
824 from the same above set). The fraction of inputs corrupted was selected
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
825 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
826 of hidden layers but it was fixed to 3 based on previous work with
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
827 SDAs on MNIST~\citep{VincentPLarochelleH2008-very-small}. The size of the hidden
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
828 layers was kept constant across hidden layers, and the best results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
829 were obtained with the largest values that we could experiment
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
830 with given our patience, with 1000 hidden units.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
831
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
832 \vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
833
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
834 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
835 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
836 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
837 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
838 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
839 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
840 of all models, on NIST and NISTP test sets.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
841 Right: error rates on NIST test digits only, along with the previous results from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
842 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
843 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
844 \label{fig:error-rates-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
845 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
846 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
847
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
848
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
849 \begin{figure}[ht]
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
850 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
851 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
852 \vspace*{-3mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
853 \caption{Relative improvement in error rate due to self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
854 Left: Improvement (or loss, when negative)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
855 induced by out-of-distribution examples (perturbed data).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
856 Right: Improvement (or loss, when negative) induced by multi-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
857 learning (training on all classes and testing only on either digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
858 upper case, or lower-case). The deep learner (SDA) benefits more from
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
859 both self-taught learning scenarios, compared to the shallow MLP.}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
860 \label{fig:improvements-charts}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
861 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
862 \end{figure}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
863
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
864 \section{Experimental Results}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
865 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
866
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
867 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
868 %\subsection{SDA vs MLP vs Humans}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
869 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
870 The models are either trained on NIST (MLP0 and SDA0),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
871 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
872 on either NIST, NISTP or P07 (regardless of the data set used for training),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
873 either on the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
874 or on the 10-digits task. Training time (including about half
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
875 for unsupervised pre-training, for DAs) on the larger
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
876 datasets is around one day on a GPU (GTX 285).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
877 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
878 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
879 SDA2), along with the previous results on the digits NIST special database
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
880 19 test set from the literature, respectively based on ARTMAP neural
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
881 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
882 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
883 ~\citep{Milgram+al-2005}.% More detailed and complete numerical results
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
884 %(figures and tables, including standard errors on the error rates) can be
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
885 %found in Appendix.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
886 The deep learner not only outperformed the shallow ones and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
887 previously published performance (in a statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
888 significant way) but when trained with perturbed data
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
889 reaches human performance on both the 62-class task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
890 and the 10-class (digits) task.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
891 17\% error (SDA1) or 18\% error (humans) may seem large but a large
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
892 majority of the errors from humans and from SDA1 are from out-of-context
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
893 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
894 ``c'' and a ``C'' are often indistinguishible).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
895
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
896 In addition, as shown in the left of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
897 Figure~\ref{fig:improvements-charts}, the relative improvement in error
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
898 rate brought by self-taught learning is greater for the SDA, and these
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
899 differences with the MLP are statistically and qualitatively
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
900 significant.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
901 The left side of the figure shows the improvement to the clean
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
902 NIST test set error brought by the use of out-of-distribution examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
903 (i.e. the perturbed examples examples from NISTP or P07),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
904 over the models trained exclusively on NIST (respectively SDA0 and MLP0).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
905 Relative percent change is measured by taking
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
906 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
907 The right side of
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
908 Figure~\ref{fig:improvements-charts} shows the relative improvement
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
909 brought by the use of a multi-task setting, in which the same model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
910 trained for more classes than the target classes of interest (i.e. training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
911 with all 62 classes when the target classes are respectively the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
912 lower-case, or upper-case characters). Again, whereas the gain from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
913 multi-task setting is marginal or negative for the MLP, it is substantial
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
914 for the SDA. Note that to simplify these multi-task experiments, only the original
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
915 NIST dataset is used. For example, the MLP-digits bar shows the relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
916 percent improvement in MLP error rate on the NIST digits test set
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
917 is $100\% \times$ (single-task
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
918 model's error / multi-task model's error - 1). The single-task model is
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
919 trained with only 10 outputs (one per digit), seeing only digit examples,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
920 whereas the multi-task model is trained with 62 outputs, with all 62
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
921 character classes as examples. Hence the hidden units are shared across
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
922 all tasks. For the multi-task model, the digit error rate is measured by
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
923 comparing the correct digit class with the output class associated with the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
924 maximum conditional probability among only the digit classes outputs. The
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
925 setting is similar for the other two target classes (lower case characters
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
926 and upper case characters). Note however that some types of perturbations
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
927 (NISTP) help more than others (P07) when testing on the clean images.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
928 %%\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
929 %\subsection{Perturbed Training Data More Helpful for SDA}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
930 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
931
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
932 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
933 %\subsection{Multi-Task Learning Effects}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
934 %\vspace*{-1mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
935
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
936 \iffalse
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
937 As previously seen, the SDA is better able to benefit from the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
938 transformations applied to the data than the MLP. In this experiment we
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
939 define three tasks: recognizing digits (knowing that the input is a digit),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
940 recognizing upper case characters (knowing that the input is one), and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
941 recognizing lower case characters (knowing that the input is one). We
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
942 consider the digit classification task as the target task and we want to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
943 evaluate whether training with the other tasks can help or hurt, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
944 whether the effect is different for MLPs versus SDAs. The goal is to find
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
945 out if deep learning can benefit more (or less) from multiple related tasks
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
946 (i.e. the multi-task setting) compared to a corresponding purely supervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
947 shallow learner.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
948
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
949 We use a single hidden layer MLP with 1000 hidden units, and a SDA
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
950 with 3 hidden layers (1000 hidden units per layer), pre-trained and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
951 fine-tuned on NIST.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
952
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
953 Our results show that the MLP benefits marginally from the multi-task setting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
954 in the case of digits (5\% relative improvement) but is actually hurt in the case
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
955 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
956 On the other hand the SDA benefited from the multi-task setting, with relative
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
957 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
958 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
959 \fi
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
960
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
961
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
962 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
963 \section{Conclusions and Discussion}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
964 \vspace*{-2mm}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
965
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
966 We have found that the self-taught learning framework is more beneficial
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
967 to a deep learner than to a traditional shallow and purely
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
968 supervised learner. More precisely,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
969 the answers are positive for all the questions asked in the introduction.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
970 %\begin{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
971
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
972 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
973 {\bf Do the good results previously obtained with deep architectures on the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
974 MNIST digits generalize to a much larger and richer (but similar)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
975 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
976 Yes, the SDA {\em systematically outperformed the MLP and all the previously
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
977 published results on this dataset} (the ones that we are aware of), {\em in fact reaching human-level
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
978 performance} at around 17\% error on the 62-class task and 1.4\% on the digits,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
979 and beating previously published results on the same data.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
980
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
981 $\bullet$ %\item
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
982 {\bf To what extent do self-taught learning scenarios help deep learners,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
983 and do they help them more than shallow supervised ones}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
984 We found that distorted training examples not only made the resulting
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
985 classifier better on similarly perturbed images but also on
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
986 the {\em original clean examples}, and more importantly and more novel,
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
987 that deep architectures benefit more from such {\em out-of-distribution}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
988 examples. MLPs were helped by perturbed training examples when tested on perturbed input
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
989 images (65\% relative improvement on NISTP)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
990 but only marginally helped (5\% relative improvement on all classes)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
991 or even hurt (10\% relative loss on digits)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
992 with respect to clean examples. On the other hand, the deep SDAs
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
993 were significantly boosted by these out-of-distribution examples.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
994 Similarly, whereas the improvement due to the multi-task setting was marginal or
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
995 negative for the MLP (from +5.6\% to -3.6\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
996 it was quite significant for the SDA (from +13\% to +27\% relative change),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
997 which may be explained by the arguments below.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
998 %\end{itemize}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
999
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1000 In the original self-taught learning framework~\citep{RainaR2007}, the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1001 out-of-sample examples were used as a source of unsupervised data, and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1002 experiments showed its positive effects in a \emph{limited labeled data}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1003 scenario. However, many of the results by \citet{RainaR2007} (who used a
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1004 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1005 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1006 We note instead that, for deep
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1007 architectures, our experiments show that such a positive effect is accomplished
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1008 even in a scenario with a \emph{large number of labeled examples},
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1009 i.e., here, the relative gain of self-taught learning and
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1010 out-of-distribution examples is probably preserved
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1011 in the asymptotic regime. However, note that in our perturbation experiments
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1012 (but not in our multi-task experiments),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1013 even the out-of-distribution examples are labeled, unlike in the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1014 earlier self-taught learning experiments~\citep{RainaR2007}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1015
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1016 {\bf Why would deep learners benefit more from the self-taught learning framework}?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1017 The key idea is that the lower layers of the predictor compute a hierarchy
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1018 of features that can be shared across tasks or across variants of the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1019 input distribution. A theoretical analysis of generalization improvements
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1020 due to sharing of intermediate features across tasks already points
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1021 towards that explanation~\cite{baxter95a}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1022 Intermediate features that can be used in different
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1023 contexts can be estimated in a way that allows to share statistical
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1024 strength. Features extracted through many levels are more likely to
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1025 be more abstract and more invariant to some of the factors of variation
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1026 in the underlying distribution (as the experiments in~\citet{Goodfellow2009} suggest),
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1027 increasing the likelihood that they would be useful for a larger array
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1028 of tasks and input conditions.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1029 Therefore, we hypothesize that both depth and unsupervised
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1030 pre-training play a part in explaining the advantages observed here, and future
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1031 experiments could attempt at teasing apart these factors.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1032 And why would deep learners benefit from the self-taught learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1033 scenarios even when the number of labeled examples is very large?
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1034 We hypothesize that this is related to the hypotheses studied
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1035 in~\citet{Erhan+al-2010}. In~\citet{Erhan+al-2010}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1036 it was found that online learning on a huge dataset did not make the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1037 advantage of the deep learning bias vanish, and a similar phenomenon
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1038 may be happening here. We hypothesize that unsupervised pre-training
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1039 of a deep hierarchy with self-taught learning initializes the
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1040 model in the basin of attraction of supervised gradient descent
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1041 that corresponds to better generalization. Furthermore, such good
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1042 basins of attraction are not discovered by pure supervised learning
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1043 (with or without self-taught settings), and more labeled examples
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1044 does not allow the model to go from the poorer basins of attraction discovered
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1045 by the purely supervised shallow models to the kind of better basins associated
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1046 with deep learning and self-taught learning.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1047
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1048 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1049 can be executed on-line at {\tt http://deep.host22.com}.
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1050
606
bd7d11089a47 more fix to newpage command.
Frederic Bastien <nouiz@nouiz.org>
parents: 605
diff changeset
1051 %\newpage
604
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1052 {
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1053 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1054 %\bibliographystyle{plainnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1055 \bibliographystyle{unsrtnat}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1056 %\bibliographystyle{apalike}
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1057 }
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1058
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1059
51213beaed8b draft of NIPS 2010 workshop camera-ready version
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1060 \end{document}