ift6266: writeup/nips2010_submission.tex annotate

annotate writeup/nips2010_submission.tex @ 553:8f6c09d1140f

ca fitte de nouveau

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Wed, 02 Jun 2010 17:40:43 -0400
parents	35c611363291
children	e95395f51d72

rev	line source
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	4 \usepackage{amsthm,amsmath,bbm}
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	5 \usepackage[psamsfonts]{amssymb}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 \usepackage{graphicx,subfigure}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	9 \usepackage[numbers]{natbib}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	11 %\setlength\parindent{0mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	12
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	13 \title{Deep Self-Taught Learning for Handwritten Character Recognition}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	14 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	15
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16 \begin{document}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 \maketitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	21 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 \begin{abstract}
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	23 Recent theoretical and empirical work in statistical machine learning has
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	24 demonstrated the importance of learning algorithms for deep
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	25 architectures, i.e., function classes obtained by composing multiple
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	26 non-linear transformations. Self-taught learning (exploiting unlabeled
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	27 examples or examples from other distributions) has already been applied
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	28 to deep learners, but mostly to show the advantage of unlabeled
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	29 examples. Here we explore the advantage brought by {\em out-of-distribution
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	30 examples} and show that {\em deep learners benefit more from them than a
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	31 corresponding shallow learner}, in the area
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	32 of handwritten character recognition. In fact, we show that they reach
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	33 human-level performance on both handwritten digit classification and
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	34 62-class handwritten character recognition. For this purpose we
ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	35 developed a powerful generator of stochastic variations and noise
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	36 processes for character images, including not only affine transformations but
482 ce69aa9204d8 changement au titre et reecriture abstract Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 480 diff changeset	37 also slant, local elastic deformations, changes in thickness, background
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	38 images, grey level changes, contrast, occlusion, and various types of
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	39 noise. The out-of-distribution examples are
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	40 obtained from these highly distorted images or
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	41 by including examples of object classes different from those in the target test set.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 \end{abstract}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	43 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 \section{Introduction}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	46 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 Deep Learning has emerged as a promising new area of research in
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	49 statistical machine learning (see~\citet{Bengio-2009} for a review).
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 which consists of a chain of processing elements, each of which is associated with a
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	54 different representation of the raw visual input. In fact,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	56 those observed in the first two of these stages (in areas V1 and V2
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	57 of visual cortex)~\citep{HonglakL2008}, and that they become more and
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	58 more invariant to factors of variation (such as camera movement) in
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	59 higher layers~\citep{Goodfellow2009}.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	60 Learning a hierarchy of features increases the
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73 learning, often in an greedy layer-wise ``unsupervised pre-training''
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	74 stage~\citep{Bengio-2009}. One of these layer initialization techniques,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 applied here, is the Denoising
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	76 Auto-encoder~(DA)~\citep{VincentPLarochelleH2008-very-small} (see Figure~\ref{fig:da}),
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	77 which
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 useful for classification. The principle is that each layer starting from
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	81 the bottom is trained to encode its input (the output of the previous
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	82 layer) and to reconstruct it from a corrupted version. After this
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	83 unsupervised initialization, the stack of DAs can be
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	84 converted into a deep supervised feedforward neural network and fine-tuned by
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	86
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	87 Self-taught learning~\citep{RainaR2007} is a paradigm that combines principles
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	88 of semi-supervised and multi-task learning: the learner can exploit examples
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	89 that are unlabeled and/or come from a distribution different from the target
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	90 distribution, e.g., from other classes than those of interest.
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	91 It has already been shown that deep learners can clearly take advantage of
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	92 unsupervised learning and unlabeled examples~\citep{Bengio-2009,WestonJ2008-small},
2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	93 but more needs to be done to explore the impact
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	94 of {\em out-of-distribution} examples and of the multi-task setting
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	95 (one exception is~\citep{CollobertR2008}, which uses very different kinds
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	96 of learning algorithms). In particular the {\em relative
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	97 advantage} of deep learning for these settings has not been evaluated.
512 6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	98 The hypothesis explored here is that a deep hierarchy of features
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	99 may be better able to provide sharing of statistical strength
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	100 between different regions in input space or different tasks,
6f042a71be23 todo done Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 507 diff changeset	101 as discussed in the conclusion.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	102
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	103 In this paper we ask the following questions:
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	104
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	105 %\begin{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	106 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	107 Do the good results previously obtained with deep architectures on the
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	108 MNIST digit images generalize to the setting of a much larger and richer (but similar)
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	109 dataset, the NIST special database 19, with 62 classes and around 800k examples?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	110
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	111 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	112 To what extent does the perturbation of input images (e.g. adding
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	113 noise, affine transformations, background images) make the resulting
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	114 classifiers better not only on similarly perturbed images but also on
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	115 the {\em original clean examples}?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	116
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	117 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	118 Do deep architectures {\em benefit more from such out-of-distribution}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	119 examples, i.e. do they benefit more from the self-taught learning~\citep{RainaR2007} framework?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	120
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	121 $\bullet$ %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	122 Similarly, does the feature learning step in deep learning algorithms benefit more
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	123 from training with moderately different classes (i.e. a multi-task learning scenario) than
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	124 a corresponding shallow and purely supervised architecture?
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	125 %\end{enumerate}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	126
511 d057941417ed a few changes in the first section Dumitru Erhan <dumitru.erhan@gmail.com> parents: 510 diff changeset	127 Our experimental results provide positive evidence towards all of these questions.
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	128 To achieve these results, we introduce in the next section a sophisticated system
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	129 for stochastically transforming character images. The conclusion discusses
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	130 the more general question of why deep learners may benefit so much from
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	131 the self-taught learning framework.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	133 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	134 \section{Perturbation and Transformation of Character Images}
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	135 \label{s:perturbations}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	136 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	138 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	139 \centering
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	140 \includegraphics[scale=.45]{images/Original.PNG}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	141 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	142 \vspace{1.2cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	143 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	144 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	145 {\bf Original.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	146 This section describes the different transformations we used to stochastically
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	147 transform source images such as the one on the left
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	148 in order to obtain data from a larger distribution which
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	149 covers a domain substantially larger than the clean characters distribution from
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	150 which we start. Although character transformations have been used before to
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	151 improve character recognizers, this effort is on a large scale both
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	152 in number of classes and in the complexity of the transformations, hence
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	153 in the complexity of the learning task.
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	154 More details can
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	155 be found in this technical report~\citep{ift6266-tr-anonymous}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	157 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	158 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	159 amount of deformation or noise introduced.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	160 There are two main parts in the pipeline. The first one,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	161 from slant to pinch below, performs transformations. The second
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	162 part, from blur to contrast, adds different kinds of noise.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	163 \end{minipage}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	165 {\large\bf Transformations}
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	166
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	167
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	168 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	169 \centering
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	170 \includegraphics[scale=.45]{images/Slant_only.PNG}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	171 \label{fig:Slant}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	172 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	173 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	174 %\centering
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	175 {\bf Slant.}
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	176 Each row of the image is shifted
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	177 proportionally to its height: $shift = round(slant \times height)$.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	178 $slant \sim U[-complexity,complexity]$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	179 \vspace{1.2cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	180 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	181
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	182
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	183 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	184 \centering
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	185 \includegraphics[scale=.45]{images/Thick_only.PNG}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	186 \label{fig:Think}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	187 \vspace{.9cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	188 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	189 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	190 {\bf Thinkness.}
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	191 Morphological operators of dilation and erosion~\citep{Haralick87,Serra82}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	192 are applied. The neighborhood of each pixel is multiplied
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	193 element-wise with a {\em structuring element} matrix.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	194 The pixel value is replaced by the maximum or the minimum of the resulting
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	195 matrix, respectively for dilation or erosion. Ten different structural elements with
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	196 increasing dimensions (largest is $5\times5$) were used. For each image,
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	197 randomly sample the operator type (dilation or erosion) with equal probability and one structural
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	198 element from a subset of the $n=round(m \times complexity)$ smallest structuring elements
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	199 where $m=10$ for dilation and $m=6$ for erosion (to avoid completely erasing thin characters).
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	200 A neutral element (no transformation)
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	201 is always present in the set. is applied.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	202 \vspace{.4cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	203 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	204 \vspace{-.7cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	205
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	206
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	207 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	208 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	209 \includegraphics[scale=.45]{images/Affine_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	210 \label{fig:Affine}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	211 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	212 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	213 {\bf Affine Transformations.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	214 A $2 \times 3$ affine transform matrix (with
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	215 6 parameters $(a,b,c,d,e,f)$) is sampled according to the $complexity$ level.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	216 Output pixel $(x,y)$ takes the value of input pixel
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	217 nearest to $(ax+by+c,dx+ey+f)$,
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	218 producing scaling, translation, rotation and shearing.
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	219 The marginal distributions of $(a,b,c,d,e,f)$ have been tuned by hand to
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	220 forbid large rotations (not to confuse classes) but to give good
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	221 variability of the transformation: $a$ and $d$ $\sim U[1-3 \times
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	222 complexity,1+3 \times complexity]$, $b$ and $e$ $\sim[-3 \times complexity,3
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	223 \times complexity]$ and $c$ and $f$ $\sim U[-4 \times complexity, 4 \times
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	224 complexity]$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	225 \end{minipage}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	226
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	227 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	228 \centering
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	229 \includegraphics[scale=.45]{images/Localelasticdistorsions_only.PNG}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	230 \label{fig:Elastic}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	231 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	232 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	233 {\bf Local Elastic Deformations.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	234 This filter induces a ``wiggly'' effect in the image, following~\citet{SimardSP03-short},
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	235 which provides more details.
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	236 The intensity of the displacement fields is given by
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	237 $\alpha = \sqrt[3]{complexity} \times 10.0$, which are
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	238 convolved with a Gaussian 2D kernel (resulting in a blur) of
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	239 standard deviation $\sigma = 10 - 7 \times\sqrt[3]{complexity}$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	240 \vspace{.4cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	241 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	242 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	243
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	244 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	245 \centering
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	246 \includegraphics[scale=.45]{images/Pinch_only.PNG}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	247 \label{fig:Pinch}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	248 \vspace{.6cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	249 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	250 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	251 {\bf Pinch.}
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	252 This is the ``Whirl and pinch'' GIMP filter but with whirl was set to 0.
8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	253 A pinch is ``similar to projecting the image onto an elastic
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	254 surface and pressing or pulling on the center of the surface'' (GIMP documentation manual).
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	255 For a square input image, this is akin to drawing a circle of
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	256 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	257 that disk (region inside circle) will have its value recalculated by taking
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	258 the value of another ``source'' pixel in the original image. The position of
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	259 that source pixel is found on the line that goes through $C$ and $P$, but
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	260 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	261 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	262 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	263 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	264 around the (non-integer) source position thus found.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	265 Here $pinch \sim U[-complexity, 0.7 \times complexity]$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	266 %\vspace{1.5cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	267 \end{minipage}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	268
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	269 \vspace{.1cm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	270
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	271 {\large\bf Injecting Noise}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	272
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	273 \vspace*{-.2cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	274 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	275 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	276 \includegraphics[scale=.45]{images/Motionblur_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	277 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	278 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	279 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	280 {\bf Motion Blur.}
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	281 This is GIMP's ``linear motion blur''
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	282 with parameters $length$ and $angle$. The value of
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	283 a pixel in the final image is approximately the mean value of the first $length$ pixels
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	284 found by moving in the $angle$ direction.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	285 Here $angle \sim U[0,360]$ degrees, and $length \sim {\rm Normal}(0,(3 \times complexity)^2)$.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	286 \vspace{.7cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	287 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	288
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	289 \vspace*{-5mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	290
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	291 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	292 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	293 \includegraphics[scale=.45]{images/occlusion_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	294 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	295 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	296 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	297 {\bf Occlusion.}
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	298 Selects a random rectangle from an {\em occluder} character
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	299 image and places it over the original {\em occluded}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	300 image. Pixels are combined by taking the max(occluder,occluded),
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	301 closer to black. The rectangle corners
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	302 are sampled so that larger complexity gives larger rectangles.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	303 The destination position in the occluded image are also sampled
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	304 according to a normal distribution (more details in~\citet{ift6266-tr-anonymous}).
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	305 This filter is skipped with probability 60\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	306 \vspace{.4cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	307 \end{minipage}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	308
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	309 \vspace*{-5mm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	310 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	311 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	312 \includegraphics[scale=.45]{images/Permutpixel_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	313 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	314 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	315 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	316 {\bf Pixel Permutation.}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	317 This filter permutes neighbouring pixels. It first selects
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	318 fraction $\frac{complexity}{3}$ of pixels randomly in the image. Each of them are then
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	319 sequentially exchanged with one other in as $V4$ neighbourhood.
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	320 This filter is skipped with probability 80\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	321 \vspace{.8cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	322 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	323
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	324
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	325 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	326 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	327 \includegraphics[scale=.45]{images/Distorsiongauss_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	328 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	329 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	330 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	331 {\bf Gaussian Noise.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	332 This filter simply adds, to each pixel of the image independently, a
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	333 noise $\sim Normal(0,(\frac{complexity}{10})^2)$.
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	334 This filter is skipped with probability 70\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	335 \vspace{1.1cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	336 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	337 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	338
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	339 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	340 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	341 \includegraphics[scale=.45]{images/background_other_only.png}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	342 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	343 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	344 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	345 {\bf Background Images.}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	346 Following~\citet{Larochelle-jmlr-2009}, this transformation adds a random
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	347 background behind the letter, from a randomly chosen natural image,
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	348 with contrast adjustments depending on $complexity$, to preserve
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	349 more or less of the original character image.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	350 \vspace{.8cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	351 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	352 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	353
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	354 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	355 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	356 \includegraphics[scale=.45]{images/Poivresel_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	357 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	358 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	359 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	360 {\bf Salt and Pepper Noise.}
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	361 This filter adds noise $\sim U[0,1]$ to random subsets of pixels.
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	362 The number of selected pixels is $0.2 \times complexity$.
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	363 This filter is skipped with probability 75\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	364 \vspace{.9cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	365 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	366 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	367
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	368 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	369 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	370 \includegraphics[scale=.45]{images/Bruitgauss_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	371 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	372 \vspace{.5cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	373 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	374 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	375 {\bf Spatially Gaussian Noise.}
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	376 Different regions of the image are spatially smoothed by convolving
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	377 the image is convolved with a symmetric Gaussian kernel of
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	378 size and variance chosen uniformly in the ranges $[12,12 + 20 \times
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	379 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	380 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	381 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	382 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	383 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	384 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	385 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	386 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	387 image \times mask}{mask+1}$.
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	388 This filter is skipped with probability 75\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	389 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	390 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	391
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	392 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	393 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	394 \includegraphics[scale=.45]{images/Rature_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	395 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	396 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	397 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	398 \vspace{.4cm}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	399 {\bf Scratches.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	400 The scratches module places line-like white patches on the image. The
517 0a5945249f2b section 2, quick first pass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 511 diff changeset	401 lines are heavily transformed images of the digit ``1'' (one), chosen
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	402 at random among 500 such 1 images,
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	403 randomly cropped and rotated by an angle $\sim Normal(0,(100 \times
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	404 complexity)^2$, using bi-cubic interpolation.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	405 Two passes of a grey-scale morphological erosion filter
467 e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	406 are applied, reducing the width of the line
e0e57270b2af refs Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 466 diff changeset	407 by an amount controlled by $complexity$.
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	408 This filter is skipped with probability 85\%. The probabilities
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	409 of applying 1, 2, or 3 patches are (50\%,30\%,20\%).
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	410 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	411 \vspace{-.7cm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	412
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	413 \begin{minipage}[b]{0.14\linewidth}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	414 \centering
552 35c611363291 added the real picture Frederic Bastien <nouiz@nouiz.org> parents: 551 diff changeset	415 \includegraphics[scale=.45]{images/Contrast_only.PNG}
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	416 \label{fig:Original}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	417 \end{minipage}%
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	418 \hspace{0.3cm}\begin{minipage}[b]{0.86\linewidth}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	419 {\bf Grey Level and Contrast Changes.}
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	420 This filter changes the contrast and may invert the image polarity (white
544 1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	421 to black and black to white). The contrast is $C \sim U[1-0.85 \times complexity,1]$
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	422 so the image is normalized into $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
1cdfc17e890f ca fitte maintenant Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 541 diff changeset	423 polarity is inverted with probability 50\%.
551 8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	424 \vspace{.7cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	425 \end{minipage}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	426 \vspace{-.7cm}
8f365abf171d separete the transmo image Frederic Bastien <nouiz@nouiz.org> parents: 550 diff changeset	427
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	428
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	429 \iffalse
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	430 \begin{figure}[ht]
538 f0ee2212ea7c typos and stuff Dumitru Erhan <dumitru.erhan@gmail.com> parents: 537 534 diff changeset	431 \centerline{\resizebox{.9\textwidth}{!}{\includegraphics{images/example_t.png}}}\\
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	432 \caption{Illustration of the pipeline of stochastic
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	433 transformations applied to the image of a lower-case \emph{t}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	434 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	435 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	436 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	437 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	438 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	439 \end{figure}
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	440 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	441
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	442
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	443 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	444 \section{Experimental Setup}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	445 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	446
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	447 Much previous work on deep learning had been performed on
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	448 the MNIST digits task~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006,Salakhutdinov+Hinton-2009},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	449 with 60~000 examples, and variants involving 10~000
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	450 examples~\citep{Larochelle-jmlr-toappear-2008,VincentPLarochelleH2008}.
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	451 The focus here is on much larger training sets, from 10 times to
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	452 to 1000 times larger, and 62 classes.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	453
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	454 The first step in constructing the larger datasets (called NISTP and P07) is to sample from
499 2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	455 a {\em data source}: {\bf NIST} (NIST database 19), {\bf Fonts}, {\bf Captchas},
2b58eda9fc08 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 495 diff changeset	456 and {\bf OCR data} (scanned machine printed characters). Once a character
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	457 is sampled from one of these sources (chosen randomly), the second step is to
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	458 apply a pipeline of transformations and/or noise processes described in section \ref{s:perturbations}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	459
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	460 To provide a baseline of error rate comparison we also estimate human performance
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	461 on both the 62-class task and the 10-class digits task.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	462 We compare the best MLPs against
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	463 the best SDAs (both models' hyper-parameters are selected to minimize the validation set error),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	464 along with a comparison against a precise estimate
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	465 of human performance obtained via Amazon's Mechanical Turk (AMT)
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	466 service (http://mturk.com).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	467 AMT users are paid small amounts
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	468 of money to perform tasks for which human intelligence is required.
522 d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	469 Mechanical Turk has been used extensively in natural language processing and vision.
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	470 %processing \citep{SnowEtAl2008} and vision
d41926a68993 remis les choses qui avaient disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 521 diff changeset	471 %\citep{SorokinAndForsyth2008,whitehill09}.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	472 AMT users were presented
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	473 with 10 character images (from a test set) and asked to choose 10 corresponding ASCII
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	474 characters. They were forced to make a hard choice among the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	475 62 or 10 character classes (all classes or digits only).
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	476 80 subjects classified 2500 images per (dataset,task) pair,
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	477 with the guarantee that 3 different subjects classified each image, allowing
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	478 us to estimate inter-human variability (e.g a standard error of 0.1\%
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	479 on the average 18.2\% error done by humans on the 62-class task NIST test set).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	480
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	481 \vspace*{-3mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	482 \subsection{Data Sources}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	483 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	484
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	485 %\begin{itemize}
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	486 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	487 {\bf NIST.}
501 5927432d8b8d - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 500 diff changeset	488 Our main source of characters is the NIST Special Database 19~\citep{Grother-1995},
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	489 widely used for training and testing character
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	490 recognition systems~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	491 The dataset is composed of 814255 digits and characters (upper and lower cases), with hand checked classifications,
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	492 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	493 corresponding to ``0''-``9'',``A''-``Z'' and ``a''-``z''. The dataset contains 8 parts (partitions) of varying complexity.
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	494 The fourth partition (called $hsf_4$, 82587 examples),
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	495 experimentally recognized to be the most difficult one, is the one recommended
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	496 by NIST as a testing set and is used in our work as well as some previous work~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	497 for that purpose. We randomly split the remainder (731668 examples) into a training set and a validation set for
4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	498 model selection.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	499 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	500 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	501 useful to estimate the effect of a multi-task setting.
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	502 The distribution of the classes in the NIST training and test sets differs
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	503 substantially, with relatively many more digits in the test set, and a more uniform distribution
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	504 of letters in the test set (where the letters are distributed
8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	505 more like in natural text).
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	506 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	507
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	508 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	509 {\bf Fonts.}
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	510 In order to have a good variety of sources we downloaded an important number of free fonts from:
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	511 {\tt http://cg.scs.carleton.ca/\textasciitilde luc/freefonts.html}.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	512 % TODO: pointless to anonymize, it's not pointing to our work
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	513 Including the operating system's (Windows 7) fonts, there is a total of $9817$ different fonts that we can choose uniformly from.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	514 The chosen {\tt ttf} file is either used as input of the Captcha generator (see next item) or, by producing a corresponding image,
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	515 directly as input to our models.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	516 \vspace*{-1mm}
479 6593e67381a3 Added transformation figure Xavier Glorot <glorotxa@iro.umontreal.ca> parents: 476 diff changeset	517
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	518 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	519 {\bf Captchas.}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	520 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	521 generating characters of the same format as the NIST dataset. This software is based on
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	522 a random character class generator and various kinds of transformations similar to those described in the previous sections.
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	523 In order to increase the variability of the data generated, many different fonts are used for generating the characters.
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	524 Transformations (slant, distortions, rotation, translation) are applied to each randomly generated character with a complexity
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	525 depending on the value of the complexity parameter provided by the user of the data source.
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	526 %Two levels of complexity are allowed and can be controlled via an easy to use facade class. %TODO: what's a facade class?
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	527 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	528
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	529 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	530 {\bf OCR data.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	531 A large set (2 million) of scanned, OCRed and manually verified machine-printed
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	532 characters where included as an
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	533 additional source. This set is part of a larger corpus being collected by the Image Understanding
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	534 Pattern Recognition Research group led by Thomas Breuel at University of Kaiserslautern
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	535 ({\tt http://www.iupr.com}), and which will be publicly released.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	536 %TODO: let's hope that Thomas is not a reviewer! :) Seriously though, maybe we should anonymize this
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	537 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	538
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	539 \vspace*{-3mm}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	540 \subsection{Data Sets}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	541 \vspace*{-2mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	542
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	543 All data sets contain 32$\times$32 grey-level images (values in $[0,1]$) associated with a label
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	544 from one of the 62 character classes.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	545 %\begin{itemize}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	546 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	547
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	548 %\item
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	549 {\bf NIST.} This is the raw NIST special database 19~\citep{Grother-1995}. It has
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	550 \{651668 / 80000 / 82587\} \{training / validation / test\} examples.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	551 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	552
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	553 %\item
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	554 {\bf P07.} This dataset is obtained by taking raw characters from all four of the above sources
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	555 and sending them through the transformation pipeline described in section \ref{s:perturbations}.
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	556 For each new example to generate, a data source is selected with probability $10\%$ from the fonts,
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	557 $25\%$ from the captchas, $25\%$ from the OCR data and $40\%$ from NIST. We apply all the transformations in the
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	558 order given above, and for each of them we sample uniformly a \emph{complexity} in the range $[0,0.7]$.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	559 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	560 \vspace*{-1mm}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	561
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	562 %\item
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	563 {\bf NISTP.} This one is equivalent to P07 (complexity parameter of $0.7$ with the same proportions of data sources)
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	564 except that we only apply
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	565 transformations from slant to pinch. Therefore, the character is
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	566 transformed but no additional noise is added to the image, giving images
534 4d6493d171f6 added all sizes Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 533 diff changeset	567 closer to the NIST dataset.
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	568 It has \{81920000 / 80000 / 20000\} \{training / validation / test\} examples.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	569 %\end{itemize}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	570
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	571 \vspace*{-3mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	572 \subsection{Models and their Hyperparameters}
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	573 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	574
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	575 The experiments are performed with Multi-Layer Perceptrons (MLP) with a single
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	576 hidden layer and with Stacked Denoising Auto-Encoders (SDA).
553 8f6c09d1140f ca fitte de nouveau Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 552 diff changeset	577 \emph{Hyper-parameters are selected based on the {\bf NISTP} validation set error.}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	578
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	579 {\bf Multi-Layer Perceptrons (MLP).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	580 Whereas previous work had compared deep architectures to both shallow MLPs and
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	581 SVMs, we only compared to MLPs here because of the very large datasets used
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	582 (making the use of SVMs computationally challenging because of their quadratic
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	583 scaling behavior).
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	584 The MLP has a single hidden layer with $\tanh$ activation functions, and softmax (normalized
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	585 exponentials) on the output layer for estimating $P(class \| image)$.
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	586 The number of hidden units is taken in $\{300,500,800,1000,1500\}$.
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	587 Training examples are presented in minibatches of size 20. A constant learning
8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	588 rate was chosen among $\{0.001, 0.01, 0.025, 0.075, 0.1, 0.5\}$
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	589 through preliminary experiments (measuring performance on a validation set),
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	590 and $0.1$ was then selected for optimizing on the whole training sets.
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	591 \vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	592
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	593
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	594 {\bf Stacked Denoising Auto-Encoders (SDA).}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	595 Various auto-encoder variants and Restricted Boltzmann Machines (RBMs)
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	596 can be used to initialize the weights of each layer of a deep MLP (with many hidden
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	597 layers)~\citep{Hinton06,ranzato-07-small,Bengio-nips-2006},
18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	598 apparently setting parameters in the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	599 basin of attraction of supervised gradient descent yielding better
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	600 generalization~\citep{Erhan+al-2010}. It is hypothesized that the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	601 advantage brought by this procedure stems from a better prior,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	602 on the one hand taking advantage of the link between the input
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	603 distribution $P(x)$ and the conditional distribution of interest
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	604 $P(y\|x)$ (like in semi-supervised learning), and on the other hand
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	605 taking advantage of the expressive power and bias implicit in the
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	606 deep architecture (whereby complex concepts are expressed as
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	607 compositions of simpler ones through a deep hierarchy).
530 8fe77eac344f Clarifying the experimental setup, typos here and there Dumitru Erhan <dumitru.erhan@gmail.com> parents: 524 diff changeset	608
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	609 \begin{figure}[ht]
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	610 \vspace*{-2mm}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	611 \centerline{\resizebox{0.8\textwidth}{!}{\includegraphics{images/denoising_autoencoder_small.pdf}}}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	612 \vspace*{-2mm}
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	613 \caption{Illustration of the computations and training criterion for the denoising
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	614 auto-encoder used to pre-train each layer of the deep architecture. Input $x$ of
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	615 the layer (i.e. raw input or output of previous layer)
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	616 is corrupted into $\tilde{x}$ and encoded into code $y$ by the encoder $f_\theta(\cdot)$.
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	617 The decoder $g_{\theta'}(\cdot)$ maps $y$ to reconstruction $z$, which
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	618 is compared to the uncorrupted input $x$ through the loss function
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	619 $L_H(x,z)$, whose expected value is approximately minimized during training
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	620 by tuning $\theta$ and $\theta'$.}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	621 \label{fig:da}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	622 \vspace*{-2mm}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	623 \end{figure}
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	624
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	625 Here we chose to use the Denoising
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	626 Auto-Encoder~\citep{VincentPLarochelleH2008} as the building block for
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	627 these deep hierarchies of features, as it is very simple to train and
532 2e33885730cf changements aux charts.ods Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 529 diff changeset	628 explain (see Figure~\ref{fig:da}, as well as
521 13816dbef6ed des choses ont disparu Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 520 diff changeset	629 tutorial and code there: {\tt http://deeplearning.net/tutorial}),
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	630 provides immediate and efficient inference, and yielded results
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	631 comparable or better than RBMs in series of experiments
519 eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	632 \citep{VincentPLarochelleH2008}. During training, a Denoising
eaa595ea2402 section 3 quickpass Dumitru Erhan <dumitru.erhan@gmail.com> parents: 518 diff changeset	633 Auto-Encoder is presented with a stochastically corrupted version
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	634 of the input and trained to reconstruct the uncorrupted input,
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	635 forcing the hidden units to represent the leading regularities in
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	636 the data. Once it is trained, in a purely unsupervised way,
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	637 its hidden units' activations can
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	638 be used as inputs for training a second one, etc.
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	639 After this unsupervised pre-training stage, the parameters
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	640 are used to initialize a deep MLP, which is fine-tuned by
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	641 the same standard procedure used to train them (see previous section).
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	642 The SDA hyper-parameters are the same as for the MLP, with the addition of the
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	643 amount of corruption noise (we used the masking noise process, whereby a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	644 fixed proportion of the input values, randomly selected, are zeroed), and a
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	645 separate learning rate for the unsupervised pre-training stage (selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	646 from the same above set). The fraction of inputs corrupted was selected
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	647 among $\{10\%, 20\%, 50\%\}$. Another hyper-parameter is the number
2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	648 of hidden layers but it was fixed to 3 based on previous work with
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	649 SDAs on MNIST~\citep{VincentPLarochelleH2008}.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	650
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	651 \vspace*{-1mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	652
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	653 \begin{figure}[ht]
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	654 \vspace*{-2mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	655 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/error_rates_charts.pdf}}}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	656 \vspace*{-3mm}
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	657 \caption{SDAx are the {\bf deep} models. Error bars indicate a 95\% confidence interval. 0 indicates that the model was trained
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	658 on NIST, 1 on NISTP, and 2 on P07. Left: overall results
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	659 of all models, on NIST and NISTP test sets.
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	660 Right: error rates on NIST test digits only, along with the previous results from
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	661 literature~\citep{Granger+al-2007,Cortes+al-2000,Oliveira+al-2002-short,Milgram+al-2005}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	662 respectively based on ART, nearest neighbors, MLPs, and SVMs.}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	663 \label{fig:error-rates-charts}
541 8aad1c6ec39a reduction espace Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 539 diff changeset	664 \vspace*{-2mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	665 \end{figure}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	666
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	667
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	668 \section{Experimental Results}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	669 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	670
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	671 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	672 %\subsection{SDA vs MLP vs Humans}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	673 %\vspace*{-1mm}
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	674 The models are either trained on NIST (MLP0 and SDA0),
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	675 NISTP (MLP1 and SDA1), or P07 (MLP2 and SDA2), and tested
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	676 on either NIST, NISTP or P07, either on the 62-class task
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	677 or on the 10-digits task.
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	678 Figure~\ref{fig:error-rates-charts} summarizes the results obtained,
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	679 comparing humans, the three MLPs (MLP0, MLP1, MLP2) and the three SDAs (SDA0, SDA1,
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	680 SDA2), along with the previous results on the digits NIST special database
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	681 19 test set from the literature respectively based on ARTMAP neural
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	682 networks ~\citep{Granger+al-2007}, fast nearest-neighbor search
516 092dae9a5040 make the reference more compact. Frederic Bastien <nouiz@nouiz.org> parents: 514 diff changeset	683 ~\citep{Cortes+al-2000}, MLPs ~\citep{Oliveira+al-2002-short}, and SVMs
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	684 ~\citep{Milgram+al-2005}. More detailed and complete numerical results
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	685 (figures and tables, including standard errors on the error rates) can be
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	686 found in Appendix I of the supplementary material.
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	687 The deep learner not only outperformed the shallow ones and
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	688 previously published performance (in a statistically and qualitatively
535 caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	689 significant way) but when trained with perturbed data
caf7769ca19c typo Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 534 diff changeset	690 reaches human performance on both the 62-class task
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	691 and the 10-class (digits) task.
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	692 17\% error (SDA1) or 18\% error (humans) may seem large but a large
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	693 majority of the errors from humans and from SDA1 are from out-of-context
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	694 confusions (e.g. a vertical bar can be a ``1'', an ``l'' or an ``L'', and a
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	695 ``c'' and a ``C'' are often indistinguishible).
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	696
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	697 \begin{figure}[ht]
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	698 \vspace*{-3mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	699 \centerline{\resizebox{.99\textwidth}{!}{\includegraphics{images/improvements_charts.pdf}}}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	700 \vspace*{-3mm}
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	701 \caption{Relative improvement in error rate due to self-taught learning.
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	702 Left: Improvement (or loss, when negative)
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	703 induced by out-of-distribution examples (perturbed data).
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	704 Right: Improvement (or loss, when negative) induced by multi-task
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	705 learning (training on all classes and testing only on either digits,
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	706 upper case, or lower-case). The deep learner (SDA) benefits more from
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	707 both self-taught learning scenarios, compared to the shallow MLP.}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	708 \label{fig:improvements-charts}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	709 \vspace*{-2mm}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	710 \end{figure}
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	711
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	712 In addition, as shown in the left of
c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	713 Figure~\ref{fig:improvements-charts}, the relative improvement in error
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	714 rate brought by self-taught learning is greater for the SDA, and these
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	715 differences with the MLP are statistically and qualitatively
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	716 significant.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	717 The left side of the figure shows the improvement to the clean
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	718 NIST test set error brought by the use of out-of-distribution examples
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	719 (i.e. the perturbed examples examples from NISTP or P07).
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	720 Relative percent change is measured by taking
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	721 $100 \% \times$ (original model's error / perturbed-data model's error - 1).
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	722 The right side of
523 c778d20ab6f8 space adjustments Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 522 diff changeset	723 Figure~\ref{fig:improvements-charts} shows the relative improvement
486 877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	724 brought by the use of a multi-task setting, in which the same model is
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	725 trained for more classes than the target classes of interest (i.e. training
877af97ee193 section resultats et appendice Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 485 diff changeset	726 with all 62 classes when the target classes are respectively the digits,
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	727 lower-case, or upper-case characters). Again, whereas the gain from the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	728 multi-task setting is marginal or negative for the MLP, it is substantial
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	729 for the SDA. Note that to simplify these multi-task experiments, only the original
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	730 NIST dataset is used. For example, the MLP-digits bar shows the relative
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	731 percent improvement in MLP error rate on the NIST digits test set
548 34cb28249de0 suggestions de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 547 diff changeset	732 is $100\% \times$ (1 - single-task
493 a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	733 model's error / multi-task model's error). The single-task model is
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	734 trained with only 10 outputs (one per digit), seeing only digit examples,
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	735 whereas the multi-task model is trained with 62 outputs, with all 62
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	736 character classes as examples. Hence the hidden units are shared across
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	737 all tasks. For the multi-task model, the digit error rate is measured by
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	738 comparing the correct digit class with the output class associated with the
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	739 maximum conditional probability among only the digit classes outputs. The
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	740 setting is similar for the other two target classes (lower case characters
a194ce5a4249 difference stat. sign. Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 491 diff changeset	741 and upper case characters).
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	742 %\vspace*{-1mm}
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	743 %\subsection{Perturbed Training Data More Helpful for SDA}
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	744 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	745
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	746 %\vspace*{-1mm}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	747 %\subsection{Multi-Task Learning Effects}
6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	748 %\vspace*{-1mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	749
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	750 \iffalse
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	751 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	752 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	753 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	754 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	755 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	756 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	757 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	758 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	759 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	760 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	761 shallow learner.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	762
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	763 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	764 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	765 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	766
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	767 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	768 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	769 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
495 5764a2ae1fb5 typos Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 493 diff changeset	770 On the other hand the SDA benefited from the multi-task setting, with relative
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	771 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	772 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	773 \fi
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	774
475 ead3085c1c66 Added charts to nips2010_submission.tex fsavard parents: 469 diff changeset	775
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	776 \vspace*{-2mm}
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	777 \section{Conclusions and Discussion}
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	778 \vspace*{-2mm}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	779
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	780 We have found that the self-taught learning framework is more beneficial
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	781 to a deep learner than to a traditional shallow and purely
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	782 supervised learner. More precisely,
520 18a6379999fd more after lunch :) Dumitru Erhan <dumitru.erhan@gmail.com> parents: 519 diff changeset	783 the answers are positive for all the questions asked in the introduction.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	784 %\begin{itemize}
487 21787ac4e5a0 - Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 486 diff changeset	785
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	786 $\bullet$ %\item
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	787 {\bf Do the good results previously obtained with deep architectures on the
549 ef172f4a322a ca fitte Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 548 diff changeset	788 MNIST digits generalize to a much larger and richer (but similar)
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	789 dataset, the NIST special database 19, with 62 classes and around 800k examples}?
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	790 Yes, the SDA {\bf systematically outperformed the MLP and all the previously
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	791 published results on this dataset} (the ones that we are aware of), {\bf in fact reaching human-level
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	792 performance} at around 17\% error on the 62-class task and 1.4\% on the digits.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	793
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	794 $\bullet$ %\item
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	795 {\bf To what extent do self-taught learning scenarios help deep learners,
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	796 and do they help them more than shallow supervised ones}?
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	797 We found that distorted training examples not only made the resulting
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	798 classifier better on similarly perturbed images but also on
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	799 the {\em original clean examples}, and more importantly and more novel,
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	800 that deep architectures benefit more from such {\em out-of-distribution}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	801 examples. MLPs were helped by perturbed training examples when tested on perturbed input
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	802 images (65\% relative improvement on NISTP)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	803 but only marginally helped (5\% relative improvement on all classes)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	804 or even hurt (10\% relative loss on digits)
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	805 with respect to clean examples . On the other hand, the deep SDAs
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	806 were very significantly boosted by these out-of-distribution examples.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	807 Similarly, whereas the improvement due to the multi-task setting was marginal or
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	808 negative for the MLP (from +5.6\% to -3.6\% relative change),
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	809 it was very significant for the SDA (from +13\% to +27\% relative change),
316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	810 which may be explained by the arguments below.
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	811 %\end{itemize}
472 2dd6e8962df1 conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 469 diff changeset	812
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	813 In the original self-taught learning framework~\citep{RainaR2007}, the
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	814 out-of-sample examples were used as a source of unsupervised data, and
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	815 experiments showed its positive effects in a \emph{limited labeled data}
07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	816 scenario. However, many of the results by \citet{RainaR2007} (who used a
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	817 shallow, sparse coding approach) suggest that the {\em relative gain of self-taught
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	818 learning vs ordinary supervised learning} diminishes as the number of labeled examples increases.
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	819 We note instead that, for deep
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	820 architectures, our experiments show that such a positive effect is accomplished
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	821 even in a scenario with a \emph{very large number of labeled examples},
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	822 i.e., here, the relative gain of self-taught learning is probably preserved
662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	823 in the asymptotic regime.
524 07bc0ca8d246 added paragraph comparing "our" self-taught learning with "theirs" Dumitru Erhan <dumitru.erhan@gmail.com> parents: 523 diff changeset	824
547 316c7bdad5ad charts Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 544 diff changeset	825 {\bf Why would deep learners benefit more from the self-taught learning framework}?
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	826 The key idea is that the lower layers of the predictor compute a hierarchy
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	827 of features that can be shared across tasks or across variants of the
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	828 input distribution. Intermediate features that can be used in different
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	829 contexts can be estimated in a way that allows to share statistical
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	830 strength. Features extracted through many levels are more likely to
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	831 be more abstract (as the experiments in~\citet{Goodfellow2009} suggest),
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	832 increasing the likelihood that they would be useful for a larger array
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	833 of tasks and input conditions.
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	834 Therefore, we hypothesize that both depth and unsupervised
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	835 pre-training play a part in explaining the advantages observed here, and future
2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	836 experiments could attempt at teasing apart these factors.
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	837 And why would deep learners benefit from the self-taught learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	838 scenarios even when the number of labeled examples is very large?
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	839 We hypothesize that this is related to the hypotheses studied
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	840 in~\citet{Erhan+al-2010}. Whereas in~\citet{Erhan+al-2010}
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	841 it was found that online learning on a huge dataset did not make the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	842 advantage of the deep learning bias vanish, a similar phenomenon
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	843 may be happening here. We hypothesize that unsupervised pre-training
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	844 of a deep hierarchy with self-taught learning initializes the
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	845 model in the basin of attraction of supervised gradient descent
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	846 that corresponds to better generalization. Furthermore, such good
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	847 basins of attraction are not discovered by pure supervised learning
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	848 (with or without self-taught settings), and more labeled examples
550 662299f265ab suggestions from Ian Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 549 diff changeset	849 does not allow the model to go from the poorer basins of attraction discovered
529 4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	850 by the purely supervised shallow models to the kind of better basins associated
4354c3c8f49c longer conclusion Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 524 diff changeset	851 with deep learning and self-taught learning.
502 2b35a6e5ece4 changements de Myriam Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 501 diff changeset	852
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	853 A Flash demo of the recognizer (where both the MLP and the SDA can be compared)
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	854 can be executed on-line at {\tt http://deep.host22.com}.
9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	855
498 7ff00c27c976 add missing file for bibtex and make it smaller. Frederic Bastien <nouiz@nouiz.org> parents: 496 diff changeset	856 \newpage
496 e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	857 {
e41007dd40e9 make the reference shorter. Frederic Bastien <nouiz@nouiz.org> parents: 495 diff changeset	858 \bibliography{strings,strings-short,strings-shorter,ift6266_ml,aigaion-shorter,specials}
469 d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	859 %\bibliographystyle{plainnat}
d02d288257bf redone bib style Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 467 diff changeset	860 \bibliographystyle{unsrtnat}
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	861 %\bibliographystyle{apalike}
484 9a757d565e46 reduction de taille Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 483 diff changeset	862 }
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	863
485 6beaf3328521 les tables enlevées Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 484 diff changeset	864
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	865 \end{document}

Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 553:8f6c09d1140f