ift6266: writeup/nips2010_submission.tex annotate

annotate writeup/nips2010_submission.tex @ 557:17d16700e0c8

encore du visuel de Myriam

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Thu, 03 Jun 2010 09:17:01 -0400
parents	b6dfba0a110c
children	cf5a7ee2d892

rev	line source
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 \usepackage{nips10submit_e,times}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	3 \usepackage{wrapfig}
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	5 \usepackage[psamsfonts]{amssymb}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 \usepackage{graphicx,subfigure}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	9 \usepackage[numbers]{natbib}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	12
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	14 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	15
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16 \begin{document}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 \maketitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	21 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 \begin{abstract}
554 e95395f51d72 minor Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 553 diff changeset	23 Recent theoretical and empirical work in statistical machine learning has demonstrated the importance of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple non-linear transformations. Self-taught learning (exploiting unlabeled examples or examples from other distributions) has already been applied to deep learners, but mostly to show the advantage of unlabeled examples. Here we explore the advantage brought by {\em out-of-distribution examples} and show that {\em deep learners benefit more from them than a corresponding shallow learner}, in the area of handwritten character recognition. In fact, we show that they reach human-level performance on both handwritten digit classification and 62-class handwritten character recognition. For this purpose we developed a powerful generator of stochastic variations and noise processes for character images, including not only affine transformations but also slant, local elastic deformations, changes in thickness, background images, grey level changes, contrast, occlusion, and various types of noise. The out-of-distribution examples are obtained from these highly distorted images or by including examples of object classes different from those in the target test set.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 \end{abstract}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	25 \vspace*{-3mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 \section{Introduction}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	28 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 Deep Learning has emerged as a promising new area of research in
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	31 statistical machine learning (see~\citet{Bengio-2009} for a review).
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 which consists of a chain of processing elements, each of which is associated with a
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	36 different representation of the raw visual input. In fact,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 those observed in the first two of these stages (in areas V1 and V2
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	39 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	40 more invariant to factors of variation (such as camera movement) in
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	41 higher layers~\citep{Goodfellow2009}.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	42 Learning a hierarchy of features increases the
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 learning, often in an greedy layer-wise ``unsupervised pre-training''
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	56 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 applied here, is the Denoising
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	58 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see Figure~\ref{fig:da}),
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	59 which
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 useful for classification. The principle is that each layer starting from
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	63 the bottom is trained to encode its input (the output of the previous
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	64 layer) and to reconstruct it from a corrupted version. After this
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	65 unsupervised initialization, the stack of DAs can be
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	66 converted into a deep supervised feedforward neural network and fine-tuned by
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	69 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	70 of semi-supervised and multi-task learning: the learner can exploit examples
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	71 that are unlabeled and/or come from a distribution different from the target
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	72 distribution, e.g., from other classes than those of interest.
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	73 It has already been shown that deep learners can clearly take advantage of
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	74 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	75 but more needs to be done to explore the impact
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	76 of {\em out-of-distribution} examples and of the multi-task setting
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	77 (one exception is~\citep{CollobertR2008}, which uses very different kinds
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	78 of learning algorithms). In particular the {\em relative
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	79 advantage} of deep learning for these settings has not been evaluated.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	80 The hypothesis discussed in the conclusion is that a deep hierarchy of features
512 6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	81 may be better able to provide sharing of statistical strength
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	82 between different regions in input space or different tasks.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	83 %
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	84 In this paper we ask the following questions:
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	85
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	86 %\begin{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	87 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	88 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	89 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	90 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	91
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	92 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	93 To what extent does the perturbation of input images (e.g. adding
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	94 noise, affine transformations, background images) make the resulting
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	95 classifiers better not only on similarly perturbed images but also on
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	96 the {\em original clean examples}?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	97
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	98 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	99 Do deep architectures {\em benefit more from such out-of-distribution}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	100 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	101
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	102 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	103 Similarly, does the feature learning step in deep learning algorithms benefit more
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	104 from training with moderately different classes (i.e. a multi-task learning scenario) than
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	105 a corresponding shallow and purely supervised architecture?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	106 %\end{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	107
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	108 Our experimental results provide positive evidence towards all of these questions.
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	109 To achieve these results, we introduce in the next section a sophisticated system
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	110 for stochastically transforming character images and then explain the methodology.
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	111 The conclusion discusses
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	112 the more general question of why deep learners may benefit so much from
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	113 the self-taught learning framework.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	114
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	115 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116 \section{Perturbation and Transformation of Character Images}
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	117 \label{s:perturbations}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	118 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	120 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	121 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	122 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	123 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	124 \includegraphics[scale=.4]{images/Original.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	125 {\bf Original}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	126 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	127 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	128 %\vspace{0.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	129 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	130 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	131 This section describes the different transformations we used to stochastically
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	132 transform source images such as the one on the left
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	133 in order to obtain data from a larger distribution which
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	134 covers a domain substantially larger than the clean characters distribution from
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	135 which we start.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	136 Although character transformations have been used before to
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	137 improve character recognizers, this effort is on a large scale both
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	138 in number of classes and in the complexity of the transformations, hence
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	139 in the complexity of the learning task.
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	140 More details can
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	141 be found in this technical report~\citep{ift6266-tr-anonymous}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	142 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	143 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	145 amount of deformation or noise introduced.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	146 There are two main parts in the pipeline. The first one,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	147 from slant to pinch below, performs transformations. The second
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	148 part, from blur to contrast, adds different kinds of noise.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	149 %\end{minipage}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	151 \vspace*{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	152 %\subsection{Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	153 {\large\bf 2.1 Transformations}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	154 \vspace*{1mm}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	155
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	156
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	157 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	158 %\begin{minipage}[b]{0.14\linewidth}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	159 %\centering
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	160 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	161 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	162 \includegraphics[scale=.4]{images/Thick_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	163 {\bf Thickness}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	164 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	165 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	166 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	167 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	168 \end{wrapfigure}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	169 To change character {\bf thickness}, morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	170 are applied. The neighborhood of each pixel is multiplied
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	171 element-wise with a {\em structuring element} matrix.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	172 The pixel value is replaced by the maximum or the minimum of the resulting
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	173 matrix, respectively for dilation or erosion. Ten different structural elements with
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	174 increasing dimensions (largest is $5\times5$) were used. For each image,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	175 randomly sample the operator type (dilation or erosion) with equal probability and one structural
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	176 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	177 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	178 A neutral element (no transformation)
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	179 is always present in the set. is applied.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	180 %\vspace{.4cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	181 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	182 %\vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	183
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	184 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	185 \centering
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	186 \includegraphics[scale=.4]{images/Slant_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	187 {\bf Slant}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	188 \end{minipage}%
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	189 \hspace{0.3cm}\begin{minipage}[b]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	190 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	191 %\vspace*{-15mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	192 To produce {\bf slant}, each row of the image is shifted
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	193 proportionally to its height: $shift = round(slant \times height)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	194 $slant \sim U[-complexity,complexity]$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	195 \vspace{1.5cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	196 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	197 %\vspace*{-4mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	198
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	199 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	200 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	201 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	202 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	203 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	204 \includegraphics[scale=.4]{images/Affine_only.PNG}\\
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	205 {\bf Affine Transformation}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	206 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	207 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	208 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	209 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	210 A $2 \times 3$ {\bf affine transform} matrix (with
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	211 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	212 Output pixel $(x,y)$ takes the value of input pixel
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	213 nearest to $(ax+by+c,dx+ey+f)$,
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	214 producing scaling, translation, rotation and shearing.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	215 Marginal distributions of $(a,b,c,d,e,f)$ have been tuned to
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	216 forbid large rotations (not to confuse classes) but to give good
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	217 variability of the transformation: $a$ and $d$ $\sim U[1-3
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	218 complexity,1+3\,complexity]$, $b$ and $e$ $\sim[-3 \,complexity,3\,
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	219 complexity]$ and $c$ and $f$ $\sim U[-4 \,complexity, 4 \,
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	220 complexity]$.\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	221 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	222
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	223 \vspace*{-4.5mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	224
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	225 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	226 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	227 %\hspace*{-8mm}\begin{minipage}[b]{0.25\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	228 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	229 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	230 \vspace*{-4mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	231 \includegraphics[scale=.4]{images/Localelasticdistorsions_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	232 {\bf Local Elastic}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	233 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	234 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	235 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	236 %\hspace{-3mm}\begin{minipage}[b]{0.85\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	237 %\vspace*{-20mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	238 The {\bf local elastic} deformation
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	239 module induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	240 which provides more details.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	241 The intensity of the displacement fields is given by
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	242 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	243 convolved with a Gaussian 2D kernel (resulting in a blur) of
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	244 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	245 %\vspace{.9cm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	246 \end{minipage}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	247
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	248 \vspace*{5mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	249
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	250 %\begin{minipage}[b]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	251 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	252 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	253 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	254 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	255 \includegraphics[scale=.4]{images/Pinch_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	256 {\bf Pinch}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	257 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	258 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	259 %\vspace{.6cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	260 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	261 %\hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	262 The {\bf pinch} module applies the ``Whirl and pinch'' GIMP filter with whirl was set to 0.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	263 A pinch is ``similar to projecting the image onto an elastic
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	264 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	265 For a square input image, draw a radius-$r$ disk
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	266 around $C$. Any pixel $P$ belonging to
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	267 that disk has its value replaced by
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	268 the value of a ``source'' pixel in the original image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	269 on the line that goes through $C$ and $P$, but
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	270 at some other distance $d_2$. Define $d_1=distance(P,C) = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	271 d_1$, where $pinch$ is a parameter of the filter.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	272 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	273 around the (non-integer) source position thus found.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	274 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	275 %\vspace{1.5cm}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	276 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	277
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	278 \vspace{2mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	279
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	280 {\large\bf 2.2 Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	281 %\subsection{Injecting Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	282 \vspace{2mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	283
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	284 %\vspace*{-.2cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	285 \begin{minipage}[t]{0.14\linewidth}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	286 \centering
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	287 \vspace*{-2mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	288 \includegraphics[scale=.4]{images/Motionblur_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	289 {\bf Motion Blur}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	290 \end{minipage}%
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	291 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	292 %\vspace*{.5mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	293 The {\bf motion blur} module is GIMP's ``linear motion blur'', which
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	294 has parameters $length$ and $angle$. The value of
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	295 a pixel in the final image is approximately the mean of the first $length$ pixels
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	296 found by moving in the $angle$ direction,
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	297 $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	298 \vspace{5mm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	299 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	300
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	301 \vspace*{1mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	302
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	303 \begin{minipage}[t]{0.14\linewidth}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	304 \centering
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	305 \includegraphics[scale=.4]{images/occlusion_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	306 {\bf Occlusion}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	307 %\vspace{.5cm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	308 \end{minipage}%
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	309 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	310 \vspace*{-18mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	311 The {\bf occlusion} module selects a random rectangle from an {\em occluder} character
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	312 image and places it over the original {\em occluded}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	313 image. Pixels are combined by taking the max(occluder,occluded),
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	314 closer to black. The rectangle corners
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	315 are sampled so that larger complexity gives larger rectangles.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	316 The destination position in the occluded image are also sampled
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	317 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}).
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	318 This module is skipped with probability 60\%.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	319 %\vspace{7mm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	320 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	321
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	322 \vspace*{1mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	323
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	324 \begin{wrapfigure}[8]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	325 \vspace*{-6mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	326 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	327 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	328 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	329 \includegraphics[scale=.4]{images/Bruitgauss_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	330 {\bf Gaussian Smoothing}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	331 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	332 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	333 %\vspace{.5cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	334 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	335 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	336 With the {\bf Gaussian smoothing} module,
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	337 different regions of the image are spatially smoothed by convolving
554 e95395f51d72 minor Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 553 diff changeset	338 the image with a symmetric Gaussian kernel of
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	339 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	340 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
554 e95395f51d72 minor Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 553 diff changeset	341 between $0$ and $1$. We also create a symmetric weighted averaging window, of the
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	342 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	343 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	344 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	345 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	346 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	347 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	348 image \times mask}{mask+1}$.
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	349 This module is skipped with probability 75\%.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	350 %\end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	351
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	352 \newpage
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	353
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	354 \vspace*{-9mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	355
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	356 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	357 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	358 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	359 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	360 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	361 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	362 \includegraphics[scale=.4]{images/Permutpixel_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	363 {\small\bf Permute Pixels}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	364 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	365 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	366 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	367 %\hspace{-0cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	368 %\vspace*{-20mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	369 This module {\bf permutes neighbouring pixels}. It first selects
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	370 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each of them are then
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	371 sequentially exchanged with one other in as $V4$ neighbourhood.
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	372 This module is skipped with probability 80\%.\\
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	373 \vspace*{1mm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	374 \end{minipage}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	375
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	376 \vspace{-3mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	377
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	378 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	379 \begin{wrapfigure}[7]{l}{0.15\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	380 %\vspace*{-3mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	381 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	382 %\hspace*{-3mm}\begin{minipage}[t]{0.18\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	383 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	384 \vspace*{-5mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	385 \includegraphics[scale=.4]{images/Distorsiongauss_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	386 {\small \bf Gauss. Noise}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	387 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	388 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	389 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	390 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	391 \vspace*{12mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	392 The {\bf Gaussian noise} module simply adds, to each pixel of the image independently, a
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	393 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	394 This module is skipped with probability 70\%.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	395 %\vspace{1.1cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	396 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	397
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	398 \vspace*{1.2cm}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	399
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	400 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	401 \begin{minipage}[t]{0.14\linewidth}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	402 \centering
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	403 \includegraphics[scale=.4]{images/background_other_only.png}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	404 {\small \bf Bg Image}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	405 \end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	406 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	407 \vspace*{-18mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	408 Following~\citet{Larochelle-jmlr-2009}, the {\bf background image} module adds a random
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	409 background image behind the letter, from a randomly chosen natural image,
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	410 with contrast adjustments depending on $complexity$, to preserve
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	411 more or less of the original character image.
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	412 %\vspace{.8cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	413 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	414 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	415 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	416
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	417 \begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	418 \centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	419 \includegraphics[scale=.4]{images/Poivresel_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	420 {\small \bf Salt \& Pepper}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	421 \end{minipage}%
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	422 \hspace{0.3cm}\begin{minipage}[t]{0.83\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	423 \vspace*{-18mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	424 The {\bf salt and pepper noise} module adds noise $\sim U[0,1]$ to random subsets of pixels.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	425 The number of selected pixels is $0.2 \times complexity$.
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	426 This module is skipped with probability 75\%.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	427 %\vspace{.9cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	428 \end{minipage}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	429 %\vspace{-.7cm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	430
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	431 \vspace{1mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	432
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	433 \begin{minipage}[t]{\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	434 \begin{wrapfigure}[7]{l}{0.14\textwidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	435 %\begin{minipage}[t]{0.14\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	436 %\centering
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	437 \begin{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	438 \vspace*{-4mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	439 \hspace*{-1mm}\includegraphics[scale=.4]{images/Rature_only.PNG}\\
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	440 {\bf Scratches}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	441 %\end{minipage}%
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	442 \end{center}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	443 \end{wrapfigure}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	444 %\hspace{0.3cm}\begin{minipage}[t]{0.86\linewidth}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	445 %\vspace{.4cm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	446 The {\bf scratches} module places line-like white patches on the image. The
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	447 lines are heavily transformed images of the digit ``1'' (one), chosen
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	448 at random among 500 such 1 images,
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	449 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
554 e95395f51d72 minor Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 553 diff changeset	450 complexity)^2$ (in degrees), using bi-cubic interpolation.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	451 Two passes of a grey-scale morphological erosion filter
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	452 are applied, reducing the width of the line
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	453 by an amount controlled by $complexity$.
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	454 This module is skipped with probability 85\%. The probabilities
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	455 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	456 \end{minipage}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	457
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	458 \vspace*{2mm}
b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	459
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	460 \begin{minipage}[t]{0.25\linewidth}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	461 \centering
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	462 \hspace*{-16mm}\includegraphics[scale=.4]{images/Contrast_only.PNG}\\
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	463 {\bf Grey Level \& Contrast}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	464 \end{minipage}%
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	465 \hspace{-12mm}\begin{minipage}[t]{0.82\linewidth}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	466 \vspace*{-18mm}
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	467 The {\bf grey level and contrast} module changes the contrast by changing grey levels, and may invert the image polarity (white
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	468 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	469 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	470 polarity is inverted with probability 50\%.
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	471 %\vspace{.7cm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	472 \end{minipage}
555 b6dfba0a110c ameliorer l'aspect visuel, Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 554 diff changeset	473 \vspace{2mm}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	474
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	475
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	476 \iffalse
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	477 \begin{figure}[ht]
538 f0ee2212ea7c typos and stuff Dumitru Erhan <dumitru.erhan@gmail.com> parents: 537 534 diff changeset	478 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	479 \caption{Illustration of the pipeline of stochastic
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	480 transformations applied to the image of a lower-case \emph{t}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	481 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	482 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	483 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	484 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	485 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	486 \end{figure}
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	487 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	488
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	489
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	490 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	491 \section{Experimental Setup}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	492 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	493
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	494 Much previous work on deep learning had been performed on
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	495 the MNIST digits task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	496 with 60~000 examples, and variants involving 10~000
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	497 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}.
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	498 The focus here is on much larger training sets, from 10 times to
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	499 to 1000 times larger, and 62 classes.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	500
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	501 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	502 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	503 and {\bf OCR data} (scanned machine printed characters). Once a character
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	504 is sampled from one of these sources (chosen randomly), the second step is to
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	505 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	506
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	507 To provide a baseline of error rate comparison we also estimate human performance
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	508 on both the 62-class task and the 10-class digits task.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	509 We compare the best MLPs against
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	510 the best SDAs (both models' hyper-parameters are selected to minimize the validation set error),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	511 along with a comparison against a precise estimate
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	512 of human performance obtained via Amazon's Mechanical Turk (AMT)
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	513 service (http://mturk.com).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	514 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	515 of money to perform tasks for which human intelligence is required.
522 d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	516 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	517 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	518 %\citep{SorokinAndForsyth2008,whitehill09}.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	519 AMT users were presented
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	520 with 10 character images (from a test set) and asked to choose 10 corresponding ASCII
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	521 characters. They were forced to make a hard choice among the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	522 62 or 10 character classes (all classes or digits only).
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	523 80 subjects classified 2500 images per (dataset,task) pair,
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	524 with the guarantee that 3 different subjects classified each image, allowing
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	525 us to estimate inter-human variability (e.g a standard error of 0.1\%
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	526 on the average 18.2\% error done by humans on the 62-class task NIST test set).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	527
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	528 \vspace*{-3mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	529 \subsection{Data Sources}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	530 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	531
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	532 %\begin{itemize}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	533 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	534 {\bf NIST.}
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	535 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	536 widely used for training and testing character
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	537 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	538 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	539 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	540 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	541 The fourth partition (called $hsf_4$, 82587 examples),
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	542 experimentally recognized to be the most difficult one, is the one recommended
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	543 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	544 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	545 model selection.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	546 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	547 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	548 useful to estimate the effect of a multi-task setting.
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	549 The distribution of the classes in the NIST training and test sets differs
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	550 substantially, with relatively many more digits in the test set, and a more uniform distribution
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	551 of letters in the test set (where the letters are distributed
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	552 more like in natural text).
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	553 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	554
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	555 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	556 {\bf Fonts.}
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	557 In order to have a good variety of sources we downloaded an important number of free fonts from:
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	558 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	559 % TODO: pointless to anonymize, it's not pointing to our work
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	560 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	561 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	562 directly as input to our models.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	563 \vspace*{-1mm}
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	564
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	565 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	566 {\bf Captchas.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	567 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	568 generating characters of the same format as the NIST dataset. This software is based on
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	569 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	570 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	571 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	572 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	573 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	574 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	575
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	576 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	577 {\bf OCR data.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	578 A large set (2 million) of scanned, OCRed and manually verified machine-printed
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	579 characters where included as an
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	580 additional source. This set is part of a larger corpus being collected by the Image Understanding
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	581 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	582 ({\tt http://www.iupr.com}), and which will be publicly released.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	583 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	584 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	585
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	586 \vspace*{-3mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	587 \subsection{Data Sets}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	588 \vspace*{-2mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	589
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	590 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	591 from one of the 62 character classes.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	592 %\begin{itemize}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	593 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	594
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	595 %\item
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	596 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	597 \{651668 / 80000 / 82587\} \{training / validation / test\} examples.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	598 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	599
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	600 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	601 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	602 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	603 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	604 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	605 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	606 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	607 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	608
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	609 %\item
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	610 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	611 except that we only apply
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	612 transformations from slant to pinch. Therefore, the character is
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	613 transformed but no additional noise is added to the image, giving images
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	614 closer to the NIST dataset.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	615 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	616 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	617
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	618 \vspace*{-3mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	619 \subsection{Models and their Hyperparameters}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	620 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	621
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	622 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	623 hidden layer and with Stacked Denoising Auto-Encoders (SDA).
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	624 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	625
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	626 {\bf Multi-Layer Perceptrons (MLP).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	627 Whereas previous work had compared deep architectures to both shallow MLPs and
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	628 SVMs, we only compared to MLPs here because of the very large datasets used
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	629 (making the use of SVMs computationally challenging because of their quadratic
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	630 scaling behavior).
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	631 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	632 exponentials) on the output layer for estimating $P(class \| image)$.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	633 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	634 Training examples are presented in minibatches of size 20. A constant learning
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	635 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	636 through preliminary experiments (measuring performance on a validation set),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	637 and $0.1$ was then selected for optimizing on the whole training sets.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	638 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	639
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	640
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	641 {\bf Stacked Denoising Auto-Encoders (SDA).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	642 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	643 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	644 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	645 apparently setting parameters in the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	646 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	647 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	648 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	649 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	650 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	651 $P(y\|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	652 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	653 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	654 compositions of simpler ones through a deep hierarchy).
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	655
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	656 \begin{figure}[ht]
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	657 \vspace*{-2mm}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	658 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	659 \vspace*{-2mm}
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	660 \caption{Illustration of the computations and training criterion for the denoising
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	661 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	662 the layer (i.e. raw input or output of previous layer)
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	663 is corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	664 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	665 is compared to the uncorrupted input $x$ through the loss function
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	666 $L_H(x,z)$, whose expected value is approximately minimized during training
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	667 by tuning $\theta$ and $\theta'$.}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	668 \label{fig:da}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	669 \vspace*{-2mm}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	670 \end{figure}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	671
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	672 Here we chose to use the Denoising
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	673 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	674 these deep hierarchies of features, as it is very simple to train and
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	675 explain (see Figure~\ref{fig:da}, as well as
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	676 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	677 provides immediate and efficient inference, and yielded results
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	678 comparable or better than RBMs in series of experiments
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	679 \citep{VincentPLarochelleH2008}. During training, a Denoising
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	680 Auto-Encoder is presented with a stochastically corrupted version
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	681 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	682 forcing the hidden units to represent the leading regularities in
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	683 the data. Once it is trained, in a purely unsupervised way,
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	684 its hidden units' activations can
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	685 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	686 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	687 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	688 the same standard procedure used to train them (see previous section).
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	689 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	690 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	691 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	692 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	693 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	694 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	695 of hidden layers but it was fixed to 3 based on previous work with
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	696 SDAs on MNIST~\citep{VincentPLarochelleH2008}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	697
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	698 \vspace*{-1mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	699
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	700 \begin{figure}[ht]
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	701 \vspace*{-2mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	702 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	703 \vspace*{-3mm}
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	704 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	705 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	706 of all models, on NIST and NISTP test sets.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	707 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	708 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	709 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	710 \label{fig:error-rates-charts}
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	711 \vspace*{-2mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	712 \end{figure}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	713
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	714
557 17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	715 \begin{figure}[ht]
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	716 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	717 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	718 \vspace*{-3mm}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	719 \caption{Relative improvement in error rate due to self-taught learning.
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	720 Left: Improvement (or loss, when negative)
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	721 induced by out-of-distribution examples (perturbed data).
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	722 Right: Improvement (or loss, when negative) induced by multi-task
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	723 learning (training on all classes and testing only on either digits,
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	724 upper case, or lower-case). The deep learner (SDA) benefits more from
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	725 both self-taught learning scenarios, compared to the shallow MLP.}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	726 \label{fig:improvements-charts}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	727 \vspace*{-2mm}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	728 \end{figure}
17d16700e0c8 encore du visuel de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 555 diff changeset	729
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	730 \section{Experimental Results}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	731 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	732
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	733 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	734 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	735 %\vspace*{-1mm}
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	736 The models are either trained on NIST (MLP0 and SDA0),
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	737 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	738 on either NIST, NISTP or P07, either on the 62-class task
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	739 or on the 10-digits task.
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	740 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	741 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	742 SDA2), along with the previous results on the digits NIST special database
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	743 19 test set from the literature respectively based on ARTMAP neural
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	744 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	745 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	746 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	747 (figures and tables, including standard errors on the error rates) can be
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	748 found in Appendix I of the supplementary material.
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	749 The deep learner not only outperformed the shallow ones and
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	750 previously published performance (in a statistically and qualitatively
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	751 significant way) but when trained with perturbed data
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	752 reaches human performance on both the 62-class task
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	753 and the 10-class (digits) task.
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	754 17\% error (SDA1) or 18\% error (humans) may seem large but a large
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	755 majority of the errors from humans and from SDA1 are from out-of-context
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	756 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	757 ``c'' and a ``C'' are often indistinguishible).
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	758
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	759 In addition, as shown in the left of
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	760 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	761 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	762 differences with the MLP are statistically and qualitatively
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	763 significant.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	764 The left side of the figure shows the improvement to the clean
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	765 NIST test set error brought by the use of out-of-distribution examples
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	766 (i.e. the perturbed examples examples from NISTP or P07).
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	767 Relative percent change is measured by taking
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	768 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	769 The right side of
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	770 Figure~\ref{fig:improvements-charts} shows the relative improvement
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	771 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	772 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	773 with all 62 classes when the target classes are respectively the digits,
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	774 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	775 multi-task setting is marginal or negative for the MLP, it is substantial
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	776 for the SDA. Note that to simplify these multi-task experiments, only the original
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	777 NIST dataset is used. For example, the MLP-digits bar shows the relative
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	778 percent improvement in MLP error rate on the NIST digits test set
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	779 is $100\% \times$ (1 - single-task
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	780 model's error / multi-task model's error). The single-task model is
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	781 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	782 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	783 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	784 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	785 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	786 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	787 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	788 and upper case characters).
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	789 %\vspace*{-1mm}
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	790 %\subsection{Perturbed Training Data More Helpful for SDA}
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	791 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	792
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	793 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	794 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	795 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	796
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	797 \iffalse
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	798 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	799 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	800 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	801 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	802 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	803 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	804 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	805 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	806 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	807 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	808 shallow learner.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	809
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	810 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	811 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	812 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	813
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	814 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	815 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	816 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	817 On the other hand the SDA benefited from the multi-task setting, with relative
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	818 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	819 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	820 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	821
475 ead3085c1c66 Added charts to nips2010_submission.tex fsavard parents: 469 diff changeset	822
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	823 \vspace*{-2mm}
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	824 \section{Conclusions and Discussion}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	825 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	826
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	827 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	828 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	829 supervised learner. More precisely,
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	830 the answers are positive for all the questions asked in the introduction.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	831 %\begin{itemize}
487 21787ac4e5a0 - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 486 diff changeset	832
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	833 $\bullet$ %\item
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	834 {\bf Do the good results previously obtained with deep architectures on the
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	835 MNIST digits generalize to a much larger and richer (but similar)
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	836 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	837 Yes, the SDA {\bf systematically outperformed the MLP and all the previously
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	838 published results on this dataset} (the ones that we are aware of), {\bf in fact reaching human-level
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	839 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	840
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	841 $\bullet$ %\item
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	842 {\bf To what extent do self-taught learning scenarios help deep learners,
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	843 and do they help them more than shallow supervised ones}?
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	844 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	845 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	846 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	847 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	848 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	849 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	850 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	851 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	852 with respect to clean examples . On the other hand, the deep SDAs
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	853 were very significantly boosted by these out-of-distribution examples.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	854 Similarly, whereas the improvement due to the multi-task setting was marginal or
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	855 negative for the MLP (from +5.6\% to -3.6\% relative change),
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	856 it was very significant for the SDA (from +13\% to +27\% relative change),
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	857 which may be explained by the arguments below.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	858 %\end{itemize}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	859
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	860 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	861 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	862 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	863 scenario. However, many of the results by \citet{RainaR2007} (who used a
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	864 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	865 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	866 We note instead that, for deep
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	867 architectures, our experiments show that such a positive effect is accomplished
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	868 even in a scenario with a \emph{very large number of labeled examples},
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	869 i.e., here, the relative gain of self-taught learning is probably preserved
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	870 in the asymptotic regime.
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	871
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	872 {\bf Why would deep learners benefit more from the self-taught learning framework}?
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	873 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	874 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	875 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	876 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	877 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	878 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	879 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	880 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	881 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	882 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	883 experiments could attempt at teasing apart these factors.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	884 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	885 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	886 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	887 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	888 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	889 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	890 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	891 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	892 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	893 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	894 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	895 (with or without self-taught settings), and more labeled examples
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	896 does not allow the model to go from the poorer basins of attraction discovered
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	897 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	898 with deep learning and self-taught learning.
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	899
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	900 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	901 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	902
498 7ff00c27c976 add missing file for bibtex and make it smaller. Frederic Bastien <nouiz@nouiz.org> parents: 496 diff changeset	903 \newpage
496 e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	904 {
e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	905 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	906 %\bibliographystyle{plainnat}
d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	907 \bibliographystyle{unsrtnat}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	908 %\bibliographystyle{apalike}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	909 }
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	910
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	911
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	912 \end{document}

Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 557:17d16700e0c8