ift6266: writeup/nips2010_submission.tex annotate

annotate writeup/nips2010_submission.tex @ 466:6205481bf33f

asking the questions

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Fri, 28 May 2010 17:39:22 -0600
parents	24f4a8b53fcc
children	e0e57270b2af

rev	line source
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 \documentclass{article} % For LaTeX2e
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 \usepackage{nips10submit_e,times}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	4 \usepackage{amsthm,amsmath,amssymb,bbold,bbm}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	5 \usepackage{algorithm,algorithmic}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 \usepackage[utf8]{inputenc}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 \usepackage{graphicx,subfigure}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 \usepackage{mlapa}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	9
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10 \title{Generating and Exploiting Perturbed and Multi-Task Handwritten Training Data for Deep Architectures}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	11 \author{The IFT6266 Gang}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	12
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	13 \begin{document}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	14
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	15 %\makeanontitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16 \maketitle
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 \begin{abstract}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 Recent theoretical and empirical work in statistical machine learning has
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20 demonstrated the importance of learning algorithms for deep
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	21 architectures, i.e., function classes obtained by composing multiple
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 non-linear transformations. In the area of handwriting recognition,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	23 deep learning algorithms
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 had been evaluated on rather small datasets with a few tens of thousands
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	25 of examples. Here we propose a powerful generator of variations
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26 of examples for character images based on a pipeline of stochastic
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 transformations that include not only the usual affine transformations
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	28 but also the addition of slant, local elastic deformations, changes
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29 in thickness, background images, color, contrast, occlusion, and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 various types of pixel and spatially correlated noise.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	31 We evaluate a deep learning algorithm (Stacked Denoising Autoencoders)
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 on the task of learning to classify digits and letters transformed
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 with this pipeline, using the hundreds of millions of generated examples
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 and testing on the full 62-class NIST test set.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 We find that the SDA outperforms its
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	36 shallow counterpart, an ordinary Multi-Layer Perceptron,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 and that it is better able to take advantage of the additional
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 generated data, as well as better able to take advantage of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	39 the multi-task setting, i.e.,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	40 training from more classes than those of interest in the end.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	41 In fact, we find that the SDA reaches human performance as
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 estimated by the Amazon Mechanical Turk on the 62-class NIST test characters.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 \end{abstract}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 \section{Introduction}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47 Deep Learning has emerged as a promising new area of research in
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 statistical machine learning (see~\emcite{Bengio-2009} for a review).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49 Learning algorithms for deep architectures are centered on the learning
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 of useful representations of data, which are better suited to the task at hand.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 This is in great part inspired by observations of the mammalian visual cortex,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 which consists of a chain of processing elements, each of which is associated with a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 different representation. In fact,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 it was found recently that the features learnt in deep architectures resemble
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 those observed in the first two of these stages (in areas V1 and V2
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	56 of visual cortex)~\cite{HonglakL2008}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 Processing images typically involves transforming the raw pixel data into
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	58 new {\bf representations} that can be used for analysis or classification.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	59 For example, a principal component analysis representation linearly projects
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60 the input image into a lower-dimensional feature space.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 Why learn a representation? Current practice in the computer vision
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 literature converts the raw pixels into a hand-crafted representation
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63 (e.g.\ SIFT features~\cite{Lowe04}), but deep learning algorithms
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64 tend to discover similar features in their first few
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65 levels~\cite{HonglakL2008,ranzato-08,Koray-08,VincentPLarochelleH2008-very-small}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 Learning increases the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 ease and practicality of developing representations that are at once
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 tailored to specific tasks, yet are able to borrow statistical strength
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 feature representation can lead to higher-level (more abstract, more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 general) features that are more robust to unanticipated sources of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 variance extant in real data.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	74 Whereas a deep architecture can in principle be more powerful than a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 shallow one in terms of representation, depth appears to render the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	76 training problem more difficult in terms of optimization and local minima.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 It is also only recently that successful algorithms were proposed to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78 overcome some of these difficulties. All are based on unsupervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 learning, often in an greedy layer-wise ``unsupervised pre-training''
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 stage~\cite{Bengio-2009}. One of these layer initialization techniques,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	81 applied here, is the Denoising
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 Auto-Encoder~(DEA)~\cite{VincentPLarochelleH2008-very-small}, which
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	83 performed similarly or better than previously proposed Restricted Boltzmann
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 Machines in terms of unsupervised extraction of a hierarchy of features
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85 useful for classification. The principle is that each layer starting from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	86 the bottom is trained to encode their input (the output of the previous
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	87 layer) and try to reconstruct it from a corrupted version of it. After this
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	88 unsupervised initialization, the stack of denoising auto-encoders can be
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	89 converted into a deep supervised feedforward neural network and trained by
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	90 stochastic gradient descent.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	91
466 6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	92 In this paper we ask the following questions:
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	93 \begin{enumerate}
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	94 \item Do the good results previously obtained with deep architectures on the
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	95 MNIST digits generalize to the setting of a much larger and richer (but similar)
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	96 dataset, the NIST special database 19, with 62 classes and around 800k examples?
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	97 \item To what extent does the perturbation of input images (e.g. adding
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	98 noise, affine transformations, background images) make the resulting
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	99 classifier better not only on similarly perturbed images but also on
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	100 the {\em original clean examples}?
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	101 \item Do deep architectures benefit more from such {\em out-of-distribution}
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	102 examples, i.e. do they benefit more from the self-taught learning~\cite{RainaR2007} framework?
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	103 \item Similarly, does the feature learning step in deep learning algorithms benefit more
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	104 training with similar but different classes (i.e. a multi-task learning scenario) than
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	105 a corresponding shallow and purely supervised architecture?
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	106 \end{enumerate}
6205481bf33f asking the questions Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 464 diff changeset	107 The experimental results presented here provide positive evidence towards all of these questions.
464 24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	108
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	109 \section{Perturbation and Transformation of Character Images}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	110
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	111 This section describes the different transformations we used to generate data, in their order.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	112 The code for these transformations (mostly python) is available at
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	113 {\tt http://anonymous.url.net}. All the modules in the pipeline share
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	114 a global control parameter ($0 \le complexity \le 1$) that allows one to modulate the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	115 amount of deformation or noise introduced.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	117 We can differentiate two important parts in the pipeline. The first one,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	118 from slant to pinch, performs transformations of the character. The second
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119 part, from blur to contrast, adds noise to the image.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	120
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	121 \subsection{Slant}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	122
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	123 In order to mimic a slant effect, we simply shift each row of the image
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	124 proportionnaly to its height: $shift = round(slant \times height)$. We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	125 round the shift in order to have a discret displacement. We do not use a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	126 filter to smooth the result in order to save computing time and also
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127 because latter transformations have similar effects.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	128
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	129 The $slant$ coefficient can be negative or positive with equal probability
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	130 and its value is randomly sampled according to the complexity level. In
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	131 our case we take uniformly a number in the range $[0,complexity]$, so the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132 maximum displacement for the lowest or highest pixel line is of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	133 $round(complexity \times 32)$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	134
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	135
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136 \subsection{Thickness}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	138 To change the thickness of the characters we used morpholigical operators:
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	139 dilation and erosion~\cite{Haralick87,Serra82}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	141 The basic idea of such transform is, for each pixel, to multiply in the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	142 element-wise manner its neighbourhood with a matrix called the structuring
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	143 element. Then for dilation we remplace the pixel value by the maximum of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144 the result, or the minimum for erosion. This will dilate or erode objects
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	145 in the image and the strength of the transform only depends on the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	146 structuring element.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	147
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	148 We used ten different structural elements with increasing dimensions (the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	149 biggest is $5\times5$). for each image, we radomly sample the operator
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150 type (dilation or erosion) with equal probability and one structural
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	151 element from a subset of the $n$ smallest structuring elements where $n$ is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	152 $round(10 \times complexity)$ for dilation and $round(6 \times complexity)$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	153 for erosion. A neutral element is always present in the set, if it is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	154 chosen the transformation is not applied. Erosion allows only the six
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	155 smallest structural elements because when the character is too thin it may
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156 erase it completly.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	157
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	158 \subsection{Affine Transformations}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	159
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	160 We generate an affine transform matrix according to the complexity level,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	161 then we apply it directly to the image. The matrix is of size $2 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	162 3$, so we can represent it by six parameters $(a,b,c,d,e,f)$. Formally,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	163 for each pixel $(x,y)$ of the output image, we give the value of the pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164 nearest to : $(ax+by+c,dx+ey+f)$, in the input image. This allows to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	165 produce scaling, translation, rotation and shearing variances.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	166
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	167 The sampling of the parameters $(a,b,c,d,e,f)$ have been tuned by hand to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168 forbid important rotations (not to confuse classes) but to give good
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	169 variability of the transformation. For each image we sample uniformly the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	170 parameters in the following ranges: $a$ and $d$ in $[1-3 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	171 complexity,1+3 \times complexity]$, $b$ and $e$ in $[-3 \times complexity,3
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	172 \times complexity]$ and $c$ and $f$ in $[-4 \times complexity, 4 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	173 complexity]$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	174
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	175
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	176 \subsection{Local Elastic Deformations}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	177
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	178 This filter induces a "wiggly" effect in the image. The description here
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	179 will be brief, as the algorithm follows precisely what is described in
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 \cite{SimardSP03}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	182 The general idea is to generate two "displacements" fields, for horizontal
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	183 and vertical displacements of pixels. Each of these fields has the same
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	184 size as the original image.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	185
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	186 When generating the transformed image, we'll loop over the x and y
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	187 positions in the fields and select, as a value, the value of the pixel in
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	188 the original image at the (relative) position given by the displacement
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	189 fields for this x and y. If the position we'd retrieve is outside the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	190 borders of the image, we use a 0 value instead.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	191
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	192 To generate a pixel in either field, first a value between -1 and 1 is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	193 chosen from a uniform distribution. Then all the pixels, in both fields, is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	194 multiplied by a constant $\alpha$ which controls the intensity of the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	195 displacements (bigger $\alpha$ translates into larger wiggles).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	196
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	197 As a final step, each field is convoluted with a Gaussian 2D kernel of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	198 standard deviation $\sigma$. Visually, this results in a "blur"
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	199 filter. This has the effect of making values next to each other in the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	200 displacement fields similar. In effect, this makes the wiggles more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	201 coherent, less noisy.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	202
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	203 As displacement fields were long to compute, 50 pairs of fields were
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	204 generated per complexity in increments of 0.1 (50 pairs for 0.1, 50 pairs
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	205 for 0.2, etc.), and afterwards, given a complexity, we selected randomly
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	206 among the 50 corresponding pairs.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	208 $\sigma$ and $\alpha$ were linked to complexity through the formulas
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209 $\alpha = \sqrt[3]{complexity} \times 10.0$ and $\sigma = 10 - 7 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	210 \sqrt[3]{complexity}$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	211
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	212
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	213 \subsection{Pinch}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	214
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	215 This is another GIMP filter we used. The filter is in fact named "Whirl and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	216 pinch", but we don't use the "whirl" part (whirl is set to 0). As described
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	217 in GIMP, a pinch is "similar to projecting the image onto an elastic
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	218 surface and pressing or pulling on the center of the surface".
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	219
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	220 Mathematically, for a square input image, think of drawing a circle of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	221 radius $r$ around a center point $C$. Any point (pixel) $P$ belonging to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	222 that disk (region inside circle) will have its value recalculated by taking
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	223 the value of another "source" pixel in the original image. The position of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	224 that source pixel is found on the line thats goes through $C$ and $P$, but
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	225 at some other distance $d_2$. Define $d_1$ to be the distance between $P$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	226 and $C$. $d_2$ is given by $d_2 = sin(\frac{\pi{}d_1}{2r})^{-pinch} \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	227 d_1$, where $pinch$ is a parameter to the filter.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	228
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	229 If the region considered is not square then, before computing $d_2$, the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	230 smallest dimension (x or y) is stretched such that we may consider the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	231 region as if it was square. Then, after $d_2$ has been computed and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	232 corresponding components $d_2\_x$ and $d_2\_y$ have been found, the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	233 component corresponding to the stretched dimension is compressed back by an
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	234 inverse ratio.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	235
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	236 The actual value is given by bilinear interpolation considering the pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	237 around the (non-integer) source position thus found.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	238
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	239 The value for $pinch$ in our case was given by sampling from an uniform
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	240 distribution over the range $[-complexity, 0.7 \times complexity]$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	241
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	242 \subsection{Motion Blur}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	243
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	244 This is a GIMP filter we applied, a "linear motion blur" in GIMP
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	245 terminology. The description will be brief as it is a well-known filter.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	246
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	247 This algorithm has two input parameters, $length$ and $angle$. The value of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	248 a pixel in the final image is the mean value of the $length$ first pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	249 found by moving in the $angle$ direction. An approximation of this idea is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	250 used, as we won't fall onto precise pixels by following that
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	251 direction. This is done using the Bresenham line algorithm.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	252
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	253 The angle, in our case, is chosen from a uniform distribution over
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	254 $[0,360]$ degrees. The length, though, depends on the complexity; it's
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	255 sampled from a Gaussian distribution of mean 0 and standard deviation
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	256 $\sigma = 3 \times complexity$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	257
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	258 \subsection{Occlusion}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	259
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	260 This filter selects random parts of other (hereafter "occlusive") letter
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	261 images and places them over the original letter (hereafter "occluded")
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	262 image. To be more precise, having selected a subregion of the occlusive
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	263 image and a desination position in the occluded image, to determine the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	264 final value for a given overlapping pixel, it selects whichever pixel is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	265 the lightest. As a reminder, the background value is 0, black, so the value
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	266 nearest to 1 is selected.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	267
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	268 To select a subpart of the occlusive image, four numbers are generated. For
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	269 compability with the code, we'll call them "haut", "bas", "gauche" and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	270 "droite" (respectively meaning top, bottom, left and right). Each of these
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	271 numbers is selected according to a Gaussian distribution of mean $8 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	272 complexity$ and standard deviation $2$. This means the largest the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	273 complexity is, the biggest the occlusion will be. The absolute value is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	274 taken, as the numbers must be positive, and the maximum value is capped at
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	275 15.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	276
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	277 These four sizes collectively define a window centered on the middle pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	278 of the occlusive image. This is the part that will be extracted as the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	279 occlusion.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	280
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	281 The next step is to select a destination position in the occluded
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	282 image. Vertical and horizontal displacements $y\_arrivee$ and $x\_arrivee$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	283 are selected according to Gaussian distributions of mean 0 and of standard
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	284 deviations of, respectively, 3 and 2. Then an horizontal placement mode,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	285 $place$, is selected to be of three values meaning
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	286 left, middle or right.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	287
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	288 If $place$ is "middle", the occlusion will be horizontally centered
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	289 around the horizontal middle of the occluded image, then shifted according
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	290 to $x\_arrivee$. If $place$ is "left", it will be placed on the left of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	291 the occluded image, then displaced right according to $x\_arrivee$. The
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	292 contrary happens if $place$ is $right$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	293
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	294 In both the horizontal and vertical positionning, the maximum position in
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	295 either direction is such that the selected occlusion won't go beyond the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	296 borders of the occluded image.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	297
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	298 This filter has a probability of not being applied, at all, of 60\%.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	299
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	300
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	301 \subsection{Pixel Permutation}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	302
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	303 This filter permuts neighbouring pixels. It selects first
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	304 $\frac{complexity}{3}$ pixels randomly in the image. Each of them are then
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	305 sequentially exchanged to one other pixel in its $V4$ neighbourhood. Number
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	306 of exchanges to the left, right, top, bottom are equal or does not differ
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	307 from more than 1 if the number of selected pixels is not a multiple of 4.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	308
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	309 It has has a probability of not being applied, at all, of 80\%.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	310
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	311
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	312 \subsection{Gaussian Noise}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	313
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	314 This filter simply adds, to each pixel of the image independently, a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	315 Gaussian noise of mean $0$ and standard deviation $\frac{complexity}{10}$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	316
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	317 It has has a probability of not being applied, at all, of 70\%.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	318
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	319
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	320 \subsection{Background Images}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	321
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	322 Following~\cite{Larochelle-jmlr-2009}, this transformation adds a random
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	323 background behind the letter. The background is chosen by first selecting,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	324 at random, an image from a set of images. Then we choose a 32x32 subregion
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	325 of that image as the background image (by sampling x and y positions
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	326 uniformly while making sure not to cross image borders).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	327
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	328 To combine the original letter image and the background image, contrast
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	329 adjustments are made. We first get the maximal values (i.e. maximal
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	330 intensity) for both the original image and the background image, $maximage$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	331 and $maxbg$. We also have a parameter, $contrast$, given by sampling from a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	332 uniform distribution over $[complexity, 1]$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	333
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	334 Once we have all these numbers, we first adjust the values for the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	335 background image. Each pixel value is multiplied by $\frac{max(maximage -
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	336 contrast, 0)}{maxbg}$. Therefore the higher the contrast, the darkest the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	337 background will be.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	338
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	339 The final image is found by taking the brightest (i.e. value nearest to 1)
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	340 pixel from either the background image or the corresponding pixel in the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	341 original image.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	342
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	343 \subsection{Salt and Pepper Noise}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	344
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	345 This filter adds noise to the image by randomly selecting a certain number
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	346 of them and, for those selected pixels, assign a random value according to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	347 a uniform distribution over the $[0,1]$ ranges. This last distribution does
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	348 not change according to complexity. Instead, the number of selected pixels
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	349 does: the proportion of changed pixels corresponds to $complexity / 5$,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	350 which means, as a maximum, 20\% of the pixels will be randomized. On the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	351 lowest extreme, no pixel is changed.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	352
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	353 This filter also has a probability of not being applied, at all, of 75\%.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	354
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	355 \subsection{Spatially Gaussian Noise}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	356
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	357 The aim of this transformation is to filter, with a gaussian kernel,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	358 different regions of the image. In order to save computing time we decided
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	359 to convolve the whole image only once with a symmetric gaussian kernel of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	360 size and variance choosen uniformly in the ranges: $[12,12 + 20 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	361 complexity]$ and $[2,2 + 6 \times complexity]$. The result is normalized
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	362 between $0$ and $1$. We also create a symmetric averaging window, of the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	363 kernel size, with maximum value at the center. For each image we sample
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	364 uniformly from $3$ to $3 + 10 \times complexity$ pixels that will be
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	365 averaging centers between the original image and the filtered one. We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	366 initialize to zero a mask matrix of the image size. For each selected pixel
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	367 we add to the mask the averaging window centered to it. The final image is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	368 computed from the following element-wise operation: $\frac{image + filtered
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	369 image \times mask}{mask+1}$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	370
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	371 This filter has a probability of not being applied, at all, of 75\%.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	372
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	373 \subsection{Scratches}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	374
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	375 The scratches module places line-like white patches on the image. The
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	376 lines are in fact heavily transformed images of the digit "1" (one), chosen
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	377 at random among five thousands such start images of this digit.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	378
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	379 Once the image is selected, the transformation begins by finding the first
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	380 $top$, $bottom$, $right$ and $left$ non-zero pixels in the image. It is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	381 then cropped to the region thus delimited, then this cropped version is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	382 expanded to $32\times32$ again. It is then rotated by a random angle having a
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	383 Gaussian distribution of mean 90 and standard deviation $100 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	384 complexity$ (in degrees). The rotation is done with bicubic interpolation.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	385
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	386 The rotated image is then resized to $50\times50$, with anti-aliasing. In
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	387 that image, we crop the image again by selecting a region delimited
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	388 horizontally to $left$ to $left+32$ and vertically by $top$ to $top+32$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	389
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	390 Once this is done, two passes of a greyscale morphological erosion filter
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	391 are applied. Put briefly, this erosion filter reduces the width of the line
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	392 by a certain $smoothing$ amount. For small complexities (< 0.5),
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	393 $smoothing$ is 6, so the line is very small. For complexities ranging from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	394 0.25 to 0.5, $smoothing$ is 5. It is 4 for complexities 0.5 to 0.75, and 3
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	395 for higher complexities.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	396
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	397 To compensate for border effects, the image is then cropped to 28x28 by
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	398 removing two pixels everywhere on the borders, then expanded to 32x32
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	399 again. The pixel values are then linearly expanded such that the minimum
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	400 value is 0 and the maximal one is 1. Then, 50\% of the time, the image is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	401 vertically flipped.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	402
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	403 This filter is only applied only 15\% of the time. When it is applied, 50\%
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	404 of the time, only one patch image is generated and applied. In 30\% of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	405 cases, two patches are generated, and otherwise three patches are
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	406 generated. The patch is applied by taking the maximal value on any given
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	407 patch or the original image, for each of the 32x32 pixel locations.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	408
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	409 \subsection{Color and Contrast Changes}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	410
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	411 This filter changes the constrast and may invert the image polarity (white
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	412 on black to black on white). The contrast $C$ is defined here as the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	413 difference between the maximum and the minimum pixel value of the image. A
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	414 contrast value is sampled uniformly between $1$ and $1-0.85 \times
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	415 complexity$ (this insure a minimum constrast of $0.15$). We then simply
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	416 normalize the image to the range $[\frac{1-C}{2},1-\frac{1-C}{2}]$. The
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	417 polarity is inverted with $0.5$ probability.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	418
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	419
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	420 \begin{figure}[h]
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	421 \resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}\\
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	422 \caption{Illustration of the pipeline of stochastic
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	423 transformations applied to the image of a lower-case t
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	424 (the upper left image). Each image in the pipeline (going from
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	425 left to right, first top line, then bottom line) shows the result
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	426 of applying one of the modules in the pipeline. The last image
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	427 (bottom right) is used as training example.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	428 \label{fig:pipeline}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	429 \end{figure}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	430
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	431
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	432 \section{Experimental Setup}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	433
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	434 \subsection{Training Datasets}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	435
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	436 \subsubsection{Data Sources}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	437
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	438 \begin{itemize}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	439 \item {\bf NIST}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	440 The NIST Special Database 19 (NIST19) is a very widely used dataset for training and testing OCR systems.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	441 The dataset is composed with 8????? digits and characters (upper and lower cases), with hand checked classifications,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	442 extracted from handwritten sample forms of 3600 writers. The characters are labelled by one of the 62 classes
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	443 corresponding to "0"-"9","A"-"Z" and "a"-"z". The dataset contains 8 series of different complexity.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	444 The fourth series, $hsf_4$, experimentally recognized to be the most difficult one for classification task is recommended
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	445 by NIST as testing set and is used in our work for that purpose. It contains 82600 examples,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	446 while the training and validation sets (which have the same distribution) contain XXXXX and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	447 XXXXX examples respectively.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	448 The performances reported by previous work on that dataset mostly use only the digits.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	449 Here we use all the classes both in the training and testing phase. This is especially
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	450 useful to estimate the effect of a multi-task setting.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	451 Note that the distribution of the classes in the NIST training and test sets differs
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	452 substantially, with relatively many more digits in the test set, and uniform distribution
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	453 of letters in the test set, not in the training set (more like the natural distribution
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	454 of letters in text).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	455
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	456 \item {\bf Fonts} TODO!!!
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	457
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	458 \item {\bf Captchas}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	459 The Captcha data source is an adaptation of the \emph{pycaptcha} library (a python based captcha generator library) for
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	460 generating characters of the same format as the NIST dataset. The core of this data source is composed with a random character
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	461 generator and various kinds of tranformations similar to those described in the previous sections.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	462 In order to increase the variability of the data generated, different fonts are used for generating the characters.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	463 Transformations (slant, distorsions, rotation, translation) are applied to each randomly generated character with a complexity
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	464 depending on the value of the complexity parameter provided by the user of the data source. Two levels of complexity are
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	465 allowed and can be controlled via an easy to use facade class.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	466 \item {\bf OCR data}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	467 \end{itemize}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	468
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	469 \subsubsection{Data Sets}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	470 \begin{itemize}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	471 \item {\bf NIST} This is the raw NIST special database 19.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	472 \item {\bf P07}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	473 The dataset P07 is sampled with our transformation pipeline with a complexity parameter of $0.7$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	474 For each new exemple to generate, we choose one source with the following probability: $0.1$ for the fonts,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	475 $0.25$ for the captchas, $0.25$ for OCR data and $0.4$ for NIST. We apply all the transformations in their order
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	476 and for each of them we sample uniformly a complexity in the range $[0,0.7]$.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	477 \item {\bf NISTP} NISTP is equivalent to P07 (complexity parameter of $0.7$ with the same sources proportion)
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	478 except that we only apply
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	479 transformations from slant to pinch. Therefore, the character is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	480 transformed but no additionnal noise is added to the image, giving images
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	481 closer to the NIST dataset.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	482 \end{itemize}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	483
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	484 \subsection{Models and their Hyperparameters}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	485
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	486 \subsubsection{Multi-Layer Perceptrons (MLP)}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	487
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	488 An MLP is a family of functions that are described by stacking layers of of a function similar to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	489 $$g(x) = \tanh(b+Wx)$$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	490 The input, $x$, is a $d$-dimension vector.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	491 The output, $g(x)$, is a $m$-dimension vector.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	492 The parameter $W$ is a $m\times d$ matrix and is called the weight matrix.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	493 The parameter $b$ is a $m$-vector and is called the bias vector.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	494 The non-linearity (here $\tanh$) is applied element-wise to the output vector.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	495 Usually the input is referred to a input layer and similarly for the output.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	496 You can of course chain several such functions to obtain a more complex one.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	497 Here is a common example
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	498 $$f(x) = c + V\tanh(b+Wx)$$
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	499 In this case the intermediate layer corresponding to $\tanh(b+Wx)$ is called a hidden layer.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	500 Here the output layer does not have the same non-linearity as the hidden layer.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	501 This is a common case where some specialized non-linearity is applied to the output layer only depending on the task at hand.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	502
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	503 If you put 3 or more hidden layers in such a network you obtain what is called a deep MLP.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	504 The parameters to adapt are the weight matrix and the bias vector for each layer.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	505
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	506 \subsubsection{Stacked Denoising Auto-Encoders (SDAE)}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	507 \label{SdA}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	508
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	509 Auto-encoders are essentially a way to initialize the weights of the network to enable better generalization.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	510 This is essentially unsupervised training where the layer is made to reconstruct its input through and encoding and decoding phase.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	511 Denoising auto-encoders are a variant where the input is corrupted with random noise but the target is the uncorrupted input.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	512 The principle behind these initialization methods is that the network will learn the inherent relation between portions of the data and be able to represent them thus helping with whatever task we want to perform.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	513
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	514 An auto-encoder unit is formed of two MLP layers with the bottom one called the encoding layer and the top one the decoding layer.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	515 Usually the top and bottom weight matrices are the transpose of each other and are fixed this way.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	516 The network is trained as such and, when sufficiently trained, the MLP layer is initialized with the parameters of the encoding layer.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	517 The other parameters are discarded.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	518
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	519 The stacked version is an adaptation to deep MLPs where you initialize each layer with a denoising auto-encoder starting from the bottom.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	520 During the initialization, which is usually called pre-training, the bottom layer is treated as if it were an isolated auto-encoder.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	521 The second and following layers receive the same treatment except that they take as input the encoded version of the data that has gone through the layers before it.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	522 For additional details see \cite{vincent:icml08}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	523
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	524 \section{Experimental Results}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	525
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	526 \subsection{SDA vs MLP vs Humans}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	527
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	528 We compare here the best MLP (according to validation set error) that we found against
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	529 the best SDA (again according to validation set error), along with a precise estimate
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	530 of human performance obtained via Amazon's Mechanical Turk (AMT)
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	531 service\footnote{http://mturk.com}. AMT users are paid small amounts
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	532 of money to perform tasks for which human intelligence is required.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	533 Mechanical Turk has been used extensively in natural language
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	534 processing \cite{SnowEtAl2008} and vision
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	535 \cite{SorokinAndForsyth2008,whitehill09}. AMT users where presented
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	536 with 10 character images and asked to type 10 corresponding ascii
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	537 characters. Hence they were forced to make a hard choice among the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	538 62 character classes. Three users classified each image, allowing
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	539 to estimate inter-human variability (shown as +/- in parenthesis below).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	540
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	541 \begin{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	542 \caption{Overall comparison of error rates ($\pm$ std.err.) on 62 character classes (10 digits +
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	543 26 lower + 26 upper), except for last columns -- digits only, between deep architecture with pre-training
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	544 (SDA=Stacked Denoising Autoencoder) and ordinary shallow architecture
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	545 (MLP=Multi-Layer Perceptron). The models shown are all trained using perturbed data (NISTP or P07)
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	546 and using a validation set to select hyper-parameters and other training choices.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	547 \{SDA,MLP\}0 are trained on NIST,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	548 \{SDA,MLP\}1 are trained on NISTP, and \{SDA,MLP\}2 are trained on P07.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	549 The human error rate on digits is a lower bound because it does not count digits that were
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	550 recognized as letters. For comparison, the results found in the literature
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	551 on NIST digits classification using the same test set are included.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	552 \label{tab:sda-vs-mlp-vs-humans}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	553 \begin{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	554 \begin{tabular}{\|l\|r\|r\|r\|r\|} \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	555 & NIST test & NISTP test & P07 test & NIST test digits \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	556 Humans& 18.2\% $\pm$.1\% & 39.4\%$\pm$.1\% & 46.9\%$\pm$.1\% & $>1.1\%$ \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	557 SDA0 & 23.7\% $\pm$.14\% & 65.2\%$\pm$.34\% & 97.45\%$\pm$.06\% & 2.7\% $\pm$.14\%\\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	558 SDA1 & 17.1\% $\pm$.13\% & 29.7\%$\pm$.3\% & 29.7\%$\pm$.3\% & 1.4\% $\pm$.1\%\\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	559 SDA2 & 18.7\% $\pm$.13\% & 33.6\%$\pm$.3\% & 39.9\%$\pm$.17\% & 1.7\% $\pm$.1\%\\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	560 MLP0 & 24.2\% $\pm$.15\% & 68.8\%$\pm$.33\% & 78.70\%$\pm$.14\% & 3.45\% $\pm$.15\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	561 MLP1 & 23.0\% $\pm$.15\% & 41.8\%$\pm$.35\% & 90.4\%$\pm$.1\% & 3.85\% $\pm$.16\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	562 MLP2 & 24.3\% $\pm$.15\% & 46.0\%$\pm$.35\% & 54.7\%$\pm$.17\% & 4.85\% $\pm$.18\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	563 [5] & & & & 4.95\% $\pm$.18\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	564 [2] & & & & 3.71\% $\pm$.16\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	565 [3] & & & & 2.4\% $\pm$.13\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	566 [4] & & & & 2.1\% $\pm$.12\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	567 \end{tabular}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	568 \end{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	569 \end{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	570
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	571 \subsection{Perturbed Training Data More Helpful for SDAE}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	572
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	573 \begin{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	574 \caption{Relative change in error rates due to the use of perturbed training data,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	575 either using NISTP, for the MLP1/SDA1 models, or using P07, for the MLP2/SDA2 models.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	576 A positive value indicates that training on the perturbed data helped for the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	577 given test set (the first 3 columns on the 62-class tasks and the last one is
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	578 on the clean 10-class digits). Clearly, the deep learning models did benefit more
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	579 from perturbed training data, even when testing on clean data, whereas the MLP
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	580 trained on perturbed data performed worse on the clean digits and about the same
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	581 on the clean characters. }
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	582 \label{tab:sda-vs-mlp-vs-humans}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	583 \begin{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	584 \begin{tabular}{\|l\|r\|r\|r\|r\|} \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	585 & NIST test & NISTP test & P07 test & NIST test digits \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	586 SDA0/SDA1-1 & 38\% & 84\% & 228\% & 93\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	587 SDA0/SDA2-1 & 27\% & 94\% & 144\% & 59\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	588 MLP0/MLP1-1 & 5.2\% & 65\% & -13\% & -10\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	589 MLP0/MLP2-1 & -0.4\% & 49\% & 44\% & -29\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	590 \end{tabular}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	591 \end{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	592 \end{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	593
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	594
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	595 \subsection{Multi-Task Learning Effects}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	596
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	597 As previously seen, the SDA is better able to benefit from the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	598 transformations applied to the data than the MLP. In this experiment we
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	599 define three tasks: recognizing digits (knowing that the input is a digit),
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	600 recognizing upper case characters (knowing that the input is one), and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	601 recognizing lower case characters (knowing that the input is one). We
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	602 consider the digit classification task as the target task and we want to
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	603 evaluate whether training with the other tasks can help or hurt, and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	604 whether the effect is different for MLPs versus SDAs. The goal is to find
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	605 out if deep learning can benefit more (or less) from multiple related tasks
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	606 (i.e. the multi-task setting) compared to a corresponding purely supervised
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	607 shallow learner.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	608
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	609 We use a single hidden layer MLP with 1000 hidden units, and a SDA
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	610 with 3 hidden layers (1000 hidden units per layer), pre-trained and
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	611 fine-tuned on NIST.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	612
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	613 Our results show that the MLP benefits marginally from the multi-task setting
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	614 in the case of digits (5\% relative improvement) but is actually hurt in the case
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	615 of characters (respectively 3\% and 4\% worse for lower and upper class characters).
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	616 On the other hand the SDA benefitted from the multi-task setting, with relative
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	617 error rate improvements of 27\%, 15\% and 13\% respectively for digits,
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	618 lower and upper case characters, as shown in Table~\ref{tab:multi-task}.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	619
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	620 \begin{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	621 \caption{Test error rates and relative change in error rates due to the use of
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	622 a multi-task setting, i.e., training on each task in isolation vs training
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	623 for all three tasks together, for MLPs vs SDAs. The SDA benefits much
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	624 more from the multi-task setting. All experiments on only on the
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	625 unperturbed NIST data, using validation error for model selection.
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	626 Relative improvement is 1 - single-task error / multi-task error.}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	627 \label{tab:multi-task}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	628 \begin{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	629 \begin{tabular}{\|l\|r\|r\|r\|} \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	630 & single-task & multi-task & relative \\
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	631 & setting & setting & improvement \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	632 MLP-digits & 3.77\% & 3.99\% & 5.6\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	633 MLP-lower & 17.4\% & 16.8\% & -4.1\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	634 MLP-upper & 7.84\% & 7.54\% & -3.6\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	635 SDA-digits & 2.6\% & 3.56\% & 27\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	636 SDA-lower & 12.3\% & 14.4\% & 15\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	637 SDA-upper & 5.93\% & 6.78\% & 13\% \\ \hline
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	638 \end{tabular}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	639 \end{center}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	640 \end{table}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	641
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	642 \section{Conclusions}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	643
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	644 \bibliography{strings,ml,aigaion,specials}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	645 \bibliographystyle{mlapa}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	646 %\bibliographystyle{apalike}
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	647
24f4a8b53fcc nips2010_submission.tex Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	648 \end{document}

Mercurial > ift6266

annotate writeup/nips2010_submission.tex @ 466:6205481bf33f