ift6266: writeup/nips2010_submission.tex annotate

annotate writeup/nips2010_submission.tex @ 535:caf7769ca19c

typo

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Tue, 01 Jun 2010 22:25:35 -0400
parents	4d6493d171f6
children	84f42fe05594

rev	line source
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	5 \usepackage[psamsfonts]{amssymb}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 \usepackage{graphicx,subfigure}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	9 \usepackage[numbers]{natbib}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	12
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	14 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	15
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16 \begin{document}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 \maketitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	21 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 \begin{abstract}
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	23 Recent theoretical and empirical work in statistical machine learning has
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	24 demonstrated the importance of learning algorithms for deep
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	25 architectures, i.e., function classes obtained by composing multiple
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	26 non-linear transformations. Self-taught learning (exploiting unlabeled
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	27 examples or examples from other distributions) has already been applied
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	28 to deep learners, but mostly to show the advantage of unlabeled
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	29 examples. Here we explore the advantage brought by {\em out-of-distribution
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	30 examples} and show that {\em deep learners benefit more from them than a
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	31 corresponding shallow learner}, in the area
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	32 of handwritten character recognition. In fact, we show that they reach
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	33 human-level performance on both handwritten digit classification and
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	34 62-class handwritten character recognition. For this purpose we
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	35 developed a powerful generator of stochastic variations and noise
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	36 processes character images, including not only affine transformations but
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	37 also slant, local elastic deformations, changes in thickness, background
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	38 images, grey level changes, contrast, occlusion, and various types of pixel and
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	39 spatially correlated noise. The out-of-distribution examples are
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	40 obtained by training with these highly distorted images or
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	41 by including object classes different from those in the target test set.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 \end{abstract}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	43 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 \section{Introduction}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	46 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 Deep Learning has emerged as a promising new area of research in
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	49 statistical machine learning (see~\citet{Bengio-2009} for a review).
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 which consists of a chain of processing elements, each of which is associated with a
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	54 different representation of the raw visual input. In fact,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	56 those observed in the first two of these stages (in areas V1 and V2
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	57 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	58 more invariant to factors of variation (such as camera movement) in
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	59 higher layers~\citep{Goodfellow2009}.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	60 Learning a hierarchy of features increases the
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73 learning, often in an greedy layer-wise ``unsupervised pre-training''
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	74 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 applied here, is the Denoising
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	76 Auto-Encoder~(DEA)~\citep{VincentPLarochelleH2008-very-small}, which
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 useful for classification. The principle is that each layer starting from
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	80 the bottom is trained to encode its input (the output of the previous
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	81 layer) and to reconstruct it from a corrupted version. After this
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 unsupervised initialization, the stack of denoising auto-encoders can be
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	83 converted into a deep supervised feedforward neural network and fine-tuned by
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	86 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	87 of semi-supervised and multi-task learning: the learner can exploit examples
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	88 that are unlabeled and/or come from a distribution different from the target
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	89 distribution, e.g., from other classes that those of interest.
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	90 It has already been shown that deep learners can clearly take advantage of
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	91 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	92 but more needs to be done to explore the impact
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	93 of {\em out-of-distribution} examples and of the multi-task setting
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	94 (one exception is~\citep{CollobertR2008}, but using very different kinds
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	95 of learning algorithms). In particular the {\em relative
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	96 advantage} of deep learning for this settings has not been evaluated.
512 6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	97 The hypothesis explored here is that a deep hierarchy of features
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	98 may be better able to provide sharing of statistical strength
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	99 between different regions in input space or different tasks,
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	100 as discussed in the conclusion.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	101
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	102 In this paper we ask the following questions:
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	103
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	104 %\begin{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	105 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	106 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	107 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	108 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	109
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	110 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	111 To what extent does the perturbation of input images (e.g. adding
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	112 noise, affine transformations, background images) make the resulting
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	113 classifiers better not only on similarly perturbed images but also on
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	114 the {\em original clean examples}?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	115
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	116 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	117 Do deep architectures {\em benefit more from such out-of-distribution}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	118 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	119
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	120 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	121 Similarly, does the feature learning step in deep learning algorithms benefit more
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	122 training with similar but different classes (i.e. a multi-task learning scenario) than
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	123 a corresponding shallow and purely supervised architecture?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	124 %\end{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	125
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	126 Our experimental results provide positive evidence towards all of these questions.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	128 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	129 \section{Perturbation and Transformation of Character Images}
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	130 \label{s:perturbations}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	131 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	133 This section describes the different transformations we used to stochastically
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	134 transform source images in order to obtain data. More details can
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	135 be found in this technical report~\citep{ift6266-tr-anonymous}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	138 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	139 amount of deformation or noise introduced.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	141 There are two main parts in the pipeline. The first one,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	142 from slant to pinch below, performs transformations. The second
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	143 part, from blur to contrast, adds different kinds of noise.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	145 \begin{figure}[ht]
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	146 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/transfo.png}}}
506 8bf07979b8ba desiderata Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 503 diff changeset	147 % TODO: METTRE LE NOM DE LA TRANSFO A COTE DE CHAQUE IMAGE
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	148 \caption{Illustration of each transformation applied alone to the same image
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	149 of an upper-case h (top left). First row (from left to right) : original image, slant,
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	150 thickness, affine transformation (translation, rotation, shear),
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	151 local elastic deformation; second row (from left to right) :
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	152 pinch, motion blur, occlusion, pixel permutation, Gaussian noise; third row (from left to right) :
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	153 background image, salt and pepper noise, spatially Gaussian noise, scratches,
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	154 grey level and contrast changes.}
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	155 \label{fig:transfo}
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	156 \end{figure}
5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	157
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	158 {\large\bf Transformations}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	159
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	160 \vspace*{2mm}
483 b9cdb464de5f pointeur a la demo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 482 diff changeset	161
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	162 {\bf Slant.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	163 We mimic slant by shifting each row of the image
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	164 proportionally to its height: $shift = round(slant \times height)$.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	165 The $slant$ coefficient can be negative or positive with equal probability
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	166 and its value is randomly sampled according to the complexity level:
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	167 $slant \sim U[0,complexity]$, so the
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168 maximum displacement for the lowest or highest pixel line is of
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	169 $round(complexity \times 32)$.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	170 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	171
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	172 {\bf Thickness.}
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	173 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	174 are applied. The neighborhood of each pixel is multiplied
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	175 element-wise with a {\em structuring element} matrix.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	176 The pixel value is replaced by the maximum or the minimum of the resulting
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	177 matrix, respectively for dilation or erosion. Ten different structural elements with
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	178 increasing dimensions (largest is $5\times5$) were used. For each image,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	179 randomly sample the operator type (dilation or erosion) with equal probability and one structural
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 element from a subset of the $n$ smallest structuring elements where $n$ is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	182 for erosion. A neutral element is always present in the set, and if it is
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	183 chosen no transformation is applied. Erosion allows only the six
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	184 smallest structural elements because when the character is too thin it may
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	185 be completely erased.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	186 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	187
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	188 {\bf Affine Transformations.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	189 A $2 \times 3$ affine transform matrix (with
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	190 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	191 Each pixel $(x,y)$ of the output image takes the value of the pixel
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	192 nearest to $(ax+by+c,dx+ey+f)$ in the input image. This
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	193 produces scaling, translation, rotation and shearing.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	194 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	195 forbid important rotations (not to confuse classes) but to give good
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	196 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	197 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	198 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	199 complexity]$.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	200 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	201
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	202 {\bf Local Elastic Deformations.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	203 This filter induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	204 which provides more details.
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	205 Two ``displacements'' fields are generated and applied, for horizontal
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	206 and vertical displacements of pixels.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207 To generate a pixel in either field, first a value between -1 and 1 is
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	208 chosen from a uniform distribution. Then all the pixels, in both fields, are
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209 multiplied by a constant $\alpha$ which controls the intensity of the
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	210 displacements (larger $\alpha$ translates into larger wiggles).
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	211 Each field is convoluted with a Gaussian 2D kernel of
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	212 standard deviation $\sigma$. Visually, this results in a blur.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	213 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	214 \sqrt[3]{complexity}$.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	215 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	216
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	217 {\bf Pinch.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	218 This is a GIMP filter called ``Whirl and
0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	219 pinch'', but whirl was set to 0. A pinch is ``similar to projecting the image onto an elastic
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	220 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	221 For a square input image, this is akin to drawing a circle of
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	222 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	223 that disk (region inside circle) will have its value recalculated by taking
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	224 the value of another ``source'' pixel in the original image. The position of
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	225 that source pixel is found on the line that goes through $C$ and $P$, but
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	226 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	227 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	228 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	229 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	230 around the (non-integer) source position thus found.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	231 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	232
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	233 \vspace*{1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	234
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	235 {\large\bf Injecting Noise}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	236
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	237 \vspace*{1mm}
483 b9cdb464de5f pointeur a la demo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 482 diff changeset	238
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	239 {\bf Motion Blur.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	240 This is a ``linear motion blur'' in GIMP
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	241 terminology, with two parameters, $length$ and $angle$. The value of
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	242 a pixel in the final image is approximately the mean value of the $length$ first pixels
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	243 found by moving in the $angle$ direction.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	244 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	245 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	246
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	247 {\bf Occlusion.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	248 Selects a random rectangle from an {\em occluder} character
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	249 images and places it over the original {\em occluded} character
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	250 image. Pixels are combined by taking the max(occluder,occluded),
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	251 closer to black. The rectangle corners
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	252 are sampled so that larger complexity gives larger rectangles.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	253 The destination position in the occluded image are also sampled
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	254 according to a normal distribution (see more details in~\citet{ift6266-tr-anonymous}).
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	255 This filter has a probability of 60\% of not being applied.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	256 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	257
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	258 {\bf Pixel Permutation.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	259 This filter permutes neighbouring pixels. It selects first
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	260 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	261 sequentially exchanged with one other pixel in its $V4$ neighbourhood. The number
0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	262 of exchanges to the left, right, top, bottom is equal or does not differ
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	263 from more than 1 if the number of selected pixels is not a multiple of 4.
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	264 % TODO: The previous sentence is hard to parse
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	265 This filter has a probability of 80\% of not being applied.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	266 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	267
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	268 {\bf Gaussian Noise.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	269 This filter simply adds, to each pixel of the image independently, a
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	270 noise $\sim Normal(0(\frac{complexity}{10})^2)$.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	271 It has a probability of 70\% of not being applied.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	272 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	273
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	274 {\bf Background Images.}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	275 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	276 background behind the letter. The background is chosen by first selecting,
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	277 at random, an image from a set of images. Then a 32$\times$32 sub-region
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	278 of that image is chosen as the background image (by sampling position
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	279 uniformly while making sure not to cross image borders).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	280 To combine the original letter image and the background image, contrast
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	281 adjustments are made. We first get the maximal values (i.e. maximal
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	282 intensity) for both the original image and the background image, $maximage$
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	283 and $maxbg$. We also have a parameter $contrast \sim U[complexity, 1]$.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	284 Each background pixel value is multiplied by $\frac{max(maximage -
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	285 contrast, 0)}{maxbg}$ (higher contrast yield darker
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	286 background). The output image pixels are max(background,original).
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	287 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	288
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	289 {\bf Salt and Pepper Noise.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	290 This filter adds noise $\sim U[0,1]$ to random subsets of pixels.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	291 The number of selected pixels is $0.2 \times complexity$.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	292 This filter has a probability of not being applied at all of 75\%.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	293 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	294
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	295 {\bf Spatially Gaussian Noise.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	296 Different regions of the image are spatially smoothed.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	297 The image is convolved with a symmetric Gaussian kernel of
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	298 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	299 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	300 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	301 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	302 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	303 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	304 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	305 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	306 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	307 image \times mask}{mask+1}$.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	308 This filter has a probability of not being applied at all of 75\%.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	309 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	310
474 bcf024e6ab23 fits now, but still now graphics Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 472 diff changeset	311 {\bf Scratches.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	312 The scratches module places line-like white patches on the image. The
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	313 lines are heavily transformed images of the digit ``1'' (one), chosen
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	314 at random among five thousands such 1 images. The 1 image is
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	315 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	316 complexity)^2$, using bi-cubic interpolation,
5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	317 Two passes of a grey-scale morphological erosion filter
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	318 are applied, reducing the width of the line
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	319 by an amount controlled by $complexity$.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	320 This filter is only applied only 15\% of the time. When it is applied, 50\%
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	321 of the time, only one patch image is generated and applied. In 30\% of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	322 cases, two patches are generated, and otherwise three patches are
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	323 generated. The patch is applied by taking the maximal value on any given
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	324 patch or the original image, for each of the 32x32 pixel locations.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	325 \vspace*{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	326
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	327 {\bf Grey Level and Contrast Changes.}
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	328 This filter changes the contrast and may invert the image polarity (white
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	329 on black to black on white). The contrast $C$ is defined here as the
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	330 difference between the maximum and the minimum pixel value of the image.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	331 Contrast $\sim U[1-0.85 \times complexity,1]$ (so contrast $\geq 0.15$).
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	332 The image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	333 polarity is inverted with $0.5$ probability.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	334
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	335 \iffalse
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	336 \begin{figure}[ht]
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	337 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	338 \caption{Illustration of the pipeline of stochastic
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	339 transformations applied to the image of a lower-case \emph{t}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	340 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	341 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	342 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	343 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	344 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	345 \end{figure}
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	346 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	347
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	348
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	349 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	350 \section{Experimental Setup}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	351 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	352
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	353 Whereas much previous work on deep learning algorithms had been performed on
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	354 the MNIST digits classification task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	355 with 60~000 examples, and variants involving 10~000
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	356 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}, we want
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	357 to focus here on the case of much larger training sets, from 10 times to
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	358 to 1000 times larger.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	359
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	360 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	361 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	362 and {\bf OCR data} (scanned machine printed characters). Once a character
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	363 is sampled from one of these sources (chosen randomly), the pipeline of
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	364 the transformations and/or noise processes described in section \ref{s:perturbations}
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	365 is applied to the image.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	366
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	367 We compare the best MLPs against
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	368 the best SDAs (both models' hyper-parameters are selected to minimize the validation set error),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	369 along with a comparison against a precise estimate
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	370 of human performance obtained via Amazon's Mechanical Turk (AMT)
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	371 service (http://mturk.com).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	372 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	373 of money to perform tasks for which human intelligence is required.
522 d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	374 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	375 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	376 %\citep{SorokinAndForsyth2008,whitehill09}.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	377 AMT users were presented
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	378 with 10 character images and asked to choose 10 corresponding ASCII
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	379 characters. They were forced to make a hard choice among the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	380 62 or 10 character classes (all classes or digits only).
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	381 Three users classified each image, allowing
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	382 to estimate inter-human variability. A total 2500 images/dataset were classified.
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	383
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	384 \vspace*{-1mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	385 \subsection{Data Sources}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	386 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	387
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	388 %\begin{itemize}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	389 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	390 {\bf NIST.}
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	391 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	392 widely used for training and testing character
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	393 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	394 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	395 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	396 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	397 The fourth partition (called $hsf_4$, 82587 examples),
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	398 experimentally recognized to be the most difficult one, is the one recommended
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	399 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	400 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	401 model selection.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	402 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	403 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	404 useful to estimate the effect of a multi-task setting.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	405 Note that the distribution of the classes in the NIST training and test sets differs
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	406 substantially, with relatively many more digits in the test set, and more uniform distribution
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	407 of letters in the test set, compared to the training set (in the latter, the letters are distributed
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	408 more like the natural distribution of letters in text).
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	409
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	410 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	411 {\bf Fonts.}
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	412 In order to have a good variety of sources we downloaded an important number of free fonts from:
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	413 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	414 % TODO: pointless to anonymize, it's not pointing to our work
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	415 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	416 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	417 directly as input to our models.
6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	418
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	419 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	420 {\bf Captchas.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	421 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	422 generating characters of the same format as the NIST dataset. This software is based on
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	423 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	424 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	425 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	426 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	427 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	428
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	429 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	430 {\bf OCR data.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	431 A large set (2 million) of scanned, OCRed and manually verified machine-printed
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	432 characters (from various documents and books) where included as an
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	433 additional source. This set is part of a larger corpus being collected by the Image Understanding
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	434 Pattern Recognition Research group lead by Thomas Breuel at University of Kaiserslautern
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	435 ({\tt http://www.iupr.com}), and which will be publicly released.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	436 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	437 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	438
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	439 \vspace*{-1mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	440 \subsection{Data Sets}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	441 \vspace*{-1mm}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	442
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	443 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	444 from one of the 62 character classes.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	445 %\begin{itemize}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	446
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	447 %\item
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	448 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	449 \{651668 / 80000 / 82587\} \{training / validation / test\} examples.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	450
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	451 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	452 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	453 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	454 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	455 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	456 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	457 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	458
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	459 %\item
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	460 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	461 except that we only apply
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	462 transformations from slant to pinch. Therefore, the character is
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	463 transformed but no additional noise is added to the image, giving images
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	464 closer to the NIST dataset.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	465 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	466 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	467
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	468 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	469 \subsection{Models and their Hyperparameters}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	470 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	471
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	472 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	473 hidden layer and with Stacked Denoising Auto-Encoders (SDA).
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	474 \emph{Note that all hyper-parameters are selected based on performance on the {\bf NISTP} validation set.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	475
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	476 {\bf Multi-Layer Perceptrons (MLP).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	477 Whereas previous work had compared deep architectures to both shallow MLPs and
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	478 SVMs, we only compared to MLPs here because of the very large datasets used
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	479 (making the use of SVMs computationally challenging because of their quadratic
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	480 scaling behavior).
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	481 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	482 exponentials) on the output layer for estimating $P(class \| image)$.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	483 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	484 Training examples are presented in minibatches of size 20. A constant learning
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	485 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	486 through preliminary experiments (measuring performance on a validation set),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	487 and $0.1$ was then selected for optimizing on the whole training sets.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	488
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	489 \begin{figure}[ht]
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	490 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	491 \caption{Illustration of the computations and training criterion for the denoising
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	492 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	493 the layer (i.e. raw input or output of previous layer)
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	494 is corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	495 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	496 is compared to the uncorrupted input $x$ through the loss function
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	497 $L_H(x,z)$, whose expected value is approximately minimized during training
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	498 by tuning $\theta$ and $\theta'$.}
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	499 \label{fig:da}
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	500 \end{figure}
13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	501
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	502 {\bf Stacked Denoising Auto-Encoders (SDA).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	503 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	504 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	505 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	506 apparently setting parameters in the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	507 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	508 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	509 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	510 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	511 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	512 $P(y\|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	513 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	514 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	515 compositions of simpler ones through a deep hierarchy).
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	516
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	517 Here we chose to use the Denoising
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	518 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	519 these deep hierarchies of features, as it is very simple to train and
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	520 explain (see Figure~\ref{fig:da}, as well as
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	521 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	522 provides immediate and efficient inference, and yielded results
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	523 comparable or better than RBMs in series of experiments
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	524 \citep{VincentPLarochelleH2008}. During training, a Denoising
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	525 Auto-Encoder is presented with a stochastically corrupted version
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	526 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	527 forcing the hidden units to represent the leading regularities in
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	528 the data. Once it is trained, in a purely unsupervised way,
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	529 its hidden units activations can
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	530 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	531 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	532 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	533 the same standard procedure used to train them (see previous section).
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	534 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	535 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	536 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	537 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	538 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	539 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	540 of hidden layers but it was fixed to 3 based on previous work with
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	541 stacked denoising auto-encoders on MNIST~\citep{VincentPLarochelleH2008}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	542
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	543 \vspace*{-1mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	544
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	545 \begin{figure}[ht]
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	546 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	547 \caption{Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	548 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	549 of all models, on 3 different test sets (NIST, NISTP, P07).
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	550 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	551 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	552 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	553
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	554 \label{fig:error-rates-charts}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	555 \vspace*{-1mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	556 \end{figure}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	557
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	558
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	559 \section{Experimental Results}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	560
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	561 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	562 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	563 %\vspace*{-1mm}
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	564 The models are either trained on NIST (MLP0 and SDA0),
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	565 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	566 on either NIST, NISTP or P07, either on all 62 classes
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	567 or only on the digits (considering only the outputs
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	568 associated with digit classes).
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	569 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	570 comparing Humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	571 SDA2), along with the previous results on the digits NIST special database
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	572 19 test set from the literature respectively based on ARTMAP neural
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	573 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	574 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	575 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	576 (figures and tables, including standard errors on the error rates) can be
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	577 found in Appendix I of the supplementary material.
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	578 The deep learner not only outperformed the shallow ones and
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	579 previously published performance (in a statistically and qualitatively
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	580 significant way) but when trained with perturbed data
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	581 reaches human performance on both the 62-class task
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	582 and the 10-class (digits) task.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	583
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	584 \begin{figure}[ht]
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	585 \vspace*{-2mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	586 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	587 \caption{Relative improvement in error rate due to self-taught learning.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	588 Left: Improvement (or loss, when negative)
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	589 induced by out-of-distribution examples (perturbed data).
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	590 Right: Improvement (or loss, when negative) induced by multi-task
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	591 learning (training on all classes and testing only on either digits,
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	592 upper case, or lower-case). The deep learner (SDA) benefits more from
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	593 both self-taught learning scenarios, compared to the shallow MLP.}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	594 \label{fig:improvements-charts}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	595 \vspace*{-2mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	596 \end{figure}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	597
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	598 In addition, as shown in the left of
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	599 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	600 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	601 differences with the MLP are statistically and qualitatively
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	602 significant.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	603 The left side of the figure shows the improvement to the clean
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	604 NIST test set error brought by the use of out-of-distribution examples
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	605 (i.e. the perturbed examples examples from NISTP or P07).
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	606 Relative change is measured by taking
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	607 (original model's error / perturbed-data model's error - 1).
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	608 The right side of
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	609 Figure~\ref{fig:improvements-charts} shows the relative improvement
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	610 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	611 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	612 with all 62 classes when the target classes are respectively the digits,
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	613 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	614 multi-task setting is marginal or negative for the MLP, it is substantial
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	615 for the SDA. Note that for these multi-task experiment, only the original
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	616 NIST dataset is used. For example, the MLP-digits bar shows the relative
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	617 improvement in MLP error rate on the NIST digits test set (1 - single-task
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	618 model's error / multi-task model's error). The single-task model is
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	619 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	620 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	621 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	622 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	623 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	624 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	625 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	626 and upper case characters).
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	627
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	628 %\vspace*{-1mm}
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	629 %\subsection{Perturbed Training Data More Helpful for SDA}
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	630 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	631
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	632 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	633 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	634 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	635
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	636 \iffalse
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	637 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	638 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	639 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	640 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	641 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	642 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	643 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	644 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	645 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	646 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	647 shallow learner.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	648
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	649 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	650 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	651 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	652
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	653 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	654 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	655 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	656 On the other hand the SDA benefited from the multi-task setting, with relative
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	657 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	658 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	659 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	660
475 ead3085c1c66 Added charts to nips2010_submission.tex fsavard parents: 469 diff changeset	661
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	662 \vspace*{-1mm}
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	663 \section{Conclusions and Discussion}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	664 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	665
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	666 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	667 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	668 supervised learner. More precisely,
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	669 the answers are positive for all the questions asked in the introduction.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	670 %\begin{itemize}
487 21787ac4e5a0 - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 486 diff changeset	671
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	672 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	673 Do the good results previously obtained with deep architectures on the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	674 MNIST digits generalize to the setting of a much larger and richer (but similar)
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	675 dataset, the NIST special database 19, with 62 classes and around 800k examples?
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	676 Yes, the SDA {\bf systematically outperformed the MLP and all the previously
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	677 published results on this dataset} (the ones that we are aware of), {\bf in fact reaching human-level
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	678 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	679
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	680 $\bullet$ %\item
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	681 To what extent do self-taught learning scenarios help deep learners,
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	682 and do they help them more than shallow supervised ones?
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	683 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	684 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	685 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	686 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	687 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	688 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	689 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	690 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	691 with respect to clean examples . On the other hand, the deep SDAs
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	692 were very significantly boosted by these out-of-distribution examples.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	693 Similarly, whereas the improvement due to the multi-task setting was marginal or
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	694 negative for the MLP (from +5.6\% to -3.6\% relative change),
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	695 it was very significant for the SDA (from +13\% to +27\% relative change).
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	696 %\end{itemize}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	697
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	698 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	699 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	700 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	701 scenario. However, many of the results by \citet{RainaR2007} (who used a
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	702 shallow, sparse coding approach) suggest that the relative gain of self-taught
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	703 learning diminishes as the number of labeled examples increases, (essentially,
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	704 a ``diminishing returns'' scenario occurs). We note instead that, for deep
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	705 architectures, our experiments show that such a positive effect is accomplished
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	706 even in a scenario with a \emph{very large number of labeled examples}.
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	707
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	708 Why would deep learners benefit more from the self-taught learning framework?
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	709 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	710 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	711 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	712 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	713 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	714 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	715 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	716 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	717 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	718 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	719 experiments could attempt at teasing apart these factors.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	720 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	721 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	722 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	723 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	724 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	725 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	726 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	727 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	728 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	729 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	730 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	731 (with or without self-taught settings), and more labeled examples
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	732 does not allow to go from the poorer basins of attraction discovered
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	733 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	734 with deep learning and self-taught learning.
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	735
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	736 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	737 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	738
498 7ff00c27c976 add missing file for bibtex and make it smaller. Frederic Bastien <nouiz@nouiz.org> parents: 496 diff changeset	739 \newpage
496 e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	740 {
e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	741 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	742 %\bibliographystyle{plainnat}
d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	743 \bibliographystyle{unsrtnat}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	744 %\bibliographystyle{apalike}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	745 }
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	746
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	747
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	748 \end{document}

Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 535:caf7769ca19c