Mercurial > ift6266

annotate writeup/techreport.tex @ 412:6478eef4f8aa

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added support for calculating the test error over different set of classes (lower,upper,digits,all,36)
author humel
date Thu, 29 Apr 2010 14:28:52 -0400
parents 4f69d915d142
children 1e9788ce1680
rev   line source
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 \documentclass[12pt,letterpaper]{article}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 \usepackage[utf8]{inputenc}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 \usepackage{graphicx}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4 \usepackage{times}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 \usepackage{mlapa}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 \begin{document}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 \title{Generating and Exploiting Perturbed Training Data for Deep Architectures}
381
0a91fc69ff90 authors
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 379
diff changeset
9 \author{The IFT6266 Gang}
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10 \date{April 2010, Technical Report, Dept. IRO, U. Montreal}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12 \maketitle
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14 \begin{abstract}
392
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
15 Recent theoretical and empirical work in statistical machine learning has
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
16 demonstrated the importance of learning algorithms for deep
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
17 architectures, i.e., function classes obtained by composing multiple
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
18 non-linear transformations. In the area of handwriting recognition,
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
19 deep learning algorithms
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
20 had been evaluated on rather small datasets with a few tens of thousands
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
21 of examples. Here we propose a powerful generator of variations
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
22 of examples for character images based on a pipeline of stochastic
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
23 transformations that include not only the usual affine transformations
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
24 but also the addition of slant, local elastic deformations, changes
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
25 in thickness, background images, color, contrast, occlusion, and
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
26 various types of pixel and spatially correlated noise.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
27 We evaluate a deep learning algorithm (Stacked Denoising Autoencoders)
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
28 on the task of learning to classify digits and letters transformed
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
29 with this pipeline, using the hundreds of millions of generated examples
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
30 and testing on the full NIST test set.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
31 We find that the SDA outperforms its
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
32 shallow counterpart, an ordinary Multi-Layer Perceptron,
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
33 and that it is better able to take advantage of the additional
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
34 generated data.
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 \end{abstract}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 \section{Introduction}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38
392
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
39 Deep Learning has emerged as a promising new area of research in
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
40 statistical machine learning (see~\emcite{Bengio-2009} for a review).
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
41 Learning algorithms for deep architectures are centered on the learning
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
42 of useful representations of data, which are better suited to the task at hand.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
43 This is in great part inspired by observations of the mammalian visual cortex,
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
44 which consists of a chain of processing elements, each of which is associated with a
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
45 different representation. In fact,
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
46 it was found recently that the features learnt in deep architectures resemble
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
47 those observed in the first two of these stages (in areas V1 and V2
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
48 of visual cortex)~\cite{HonglakL2008}.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
49 Processing images typically involves transforming the raw pixel data into
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
50 new {\bf representations} that can be used for analysis or classification.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
51 For example, a principal component analysis representation linearly projects
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
52 the input image into a lower-dimensional feature space.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
53 Why learn a representation? Current practice in the computer vision
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
54 literature converts the raw pixels into a hand-crafted representation
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
55 (e.g.\ SIFT features~\cite{Lowe04}), but deep learning algorithms
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
56 tend to discover similar features in their first few
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
57 levels~\cite{HonglakL2008,ranzato-08,Koray-08,VincentPLarochelleH2008-very-small}.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
58 Learning increases the
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
59 ease and practicality of developing representations that are at once
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
60 tailored to specific tasks, yet are able to borrow statistical strength
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
61 from other related tasks (e.g., modeling different kinds of objects). Finally, learning the
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
62 feature representation can lead to higher-level (more abstract, more
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
63 general) features that are more robust to unanticipated sources of
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
64 variance extant in real data.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
65
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
66 Whereas a deep architecture can in principle be more powerful than a shallow
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
67 one in terms of representation, depth appears to render the training problem
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
68 more difficult in terms of optimization and local minima.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
69 It is also only recently that
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
70 successful algorithms were proposed to overcome some of these
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
71 difficulties.
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
72
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73 \section{Perturbation and Transformation of Character Images}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74
407
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
75 \subsection{Adding Slant}
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
76 In order to mimic a slant effect, we simply shift each row of the image proportionnaly to its height.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
77 The coefficient is randomly sampled according to the complexity level and can be negatif or positif with equal probability.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
78
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
79 \subsection{Changing Thickness}
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
80 To change the thickness of the characters we used morpholigical operators: dilation and erosion~\cite{Haralick87,Serra82}.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
81 The basic idea of such transform is, for each pixel, to multiply in the element-wise manner its neighbourhood with a matrix called the structuring element.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
82 Then for dilation we remplace the pixel value by the maximum of the result, or the minimum for erosion.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
83 This will dilate or erode objects in the image, the strength of the transform only depends on the structuring element.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
84 We used ten different structural elements with various shapes (the biggest is $5\times5$).
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
85 for each image, we radomly sample the operator type (dilation or erosion) and one structural element
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
86 from a subset depending of the complexity (the higher the complexity, the biggest the structural element can be).
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
87 Erosion allows only the five smallest structural elements because when the character is too thin it may erase it completly.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
88
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89 \subsection{Affine Transformations}
407
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
90 We generate an affine transform matrix according to the complexity level, then we apply it directly to the image.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
91 This allows to produce scaling, translation, rotation and shearing variances. We took care that the maximum rotation applied
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
92 to the image is low enough not to confuse classes.
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
93
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94 \subsection{Local Elastic Deformations}
407
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
95 \subsection{GIMP transformation}
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 \subsection{Occlusion}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 \subsection{Background Images}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
98 \subsection{Salt and Pepper Noise}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
99 \subsection{Spatially Gaussian Noise}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
100 \subsection{Color and Contrast Changes}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
101
393
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
102 \begin{figure}[h]
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
103 \resizebox{.99\textwidth}{!}{\includegraphics{images/example_t.png}}\\
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
104 \caption{Illustration of the pipeline of stochastic
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
105 transformations applied to the image of a lower-case t
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
106 (the upper left image). Each image in the pipeline (going from
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
107 left to right, first top line, then bottom line) shows the result
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
108 of applying one of the modules in the pipeline. The last image
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
109 (bottom right) is used as training example.}
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
110 \label{fig:pipeline}
4c840798d290 added examples of figure and table of results
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 392
diff changeset
111 \end{figure}
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
112
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 \section{Learning Algorithms for Deep Architectures}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
115 \section{Experimental Setup}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
117 \subsection{Training Datasets}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
119 \subsubsection{Data Sources}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
120
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
121 \begin{itemize}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122 \item {\bf NIST}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
123 \item {\bf Fonts}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124 \item {\bf Captchas}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
125 \item {\bf OCR data}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126 \end{itemize}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
128 \subsubsection{Data Sets}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
129 \begin{itemize}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130 \item {\bf NIST}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
131 \item {\bf P07}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
132 \item {\bf NISTP} {\em ne pas utiliser PNIST mais NISTP, pour rester politically correct...}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133 \end{itemize}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
134
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 \subsection{Models and their Hyperparameters}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 \subsubsection{Multi-Layer Perceptrons (MLP)}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
138
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
139 An MLP is a family of functions that are described by stacking layers of of a function similar to
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
140 $$g(x) = \tanh(b+Wx)$$
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
141 The input, $x$, is a $d$-dimension vector.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
142 The output, $g(x)$, is a $m$-dimension vector.
411
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
143 The parameter $W$ is a $m\times d$ matrix and is called the weight matrix.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
144 The parameter $b$ is a $m$-vector and is called the bias vector.
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
145 The non-linearity (here $\tanh$) is applied element-wise to the output vector.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
146 Usually the input is referred to a input layer and similarly for the output.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
147 You can of course chain several such functions to obtain a more complex one.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
148 Here is a common example
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
149 $$f(x) = c + V\tanh(b+Wx)$$
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
150 In this case the intermediate layer corresponding to $\tanh(b+Wx)$ is called a hidden layer.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
151 Here the output layer does not have the same non-linearity as the hidden layer.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
152 This is a common case where some specialized non-linearity is applied to the output layer only depending on the task at hand.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
153
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
154 If you put 3 or more hidden layers in such a network you obtain what is called a deep MLP.
411
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
155 The parameters to adapt are the weight matrix and the bias vector for each layer.
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
156
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
157 \subsubsection{Stacked Denoising Auto-Encoders (SDAE)}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
158
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
159 Auto-encoders are essentially a way to initialize the weights of the network to enable better generalization.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
160 Denoising auto-encoders are a variant where the input is corrupted with random noise before trying to repair it.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
161 The principle behind these initialization methods is that the network will learn the inherent relation between portions of the data and be able to represent them thus helping with whatever task we want to perform.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
162
411
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
163 An auto-encoder unit is formed of two MLP layers with the bottom one called the encoding layer and the top one the decoding layer.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
164 Usually the top and bottom weight matrices are the transpose of each other and are fixed this way.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
165 The network is trained as such and, when sufficiently trained, the MLP layer is initialized with the parameters of the encoding layer.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
166 The other parameters are discarded.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
167
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
168 The stacked version is an adaptation to deep MLPs where you initialize each layer with a denoising auto-encoder starting from the bottom.
411
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
169 During the initialization, which is usually called pre-training, the bottom layer is treated as if it were an isolated auto-encoder.
4f69d915d142 Better description of the model parameters.
Arnaud Bergeron <abergeron@gmail.com>
parents: 410
diff changeset
170 The second and following layers receive the same treatment except that they take as input the encoded version of the data that has gone through the layers before it.
410
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
171 For additional details see \cite{vincent:icml08}.
6330298791fb Description brève de MLP et SdA
Arnaud Bergeron <abergeron@gmail.com>
parents: 407
diff changeset
172
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 \section{Experimental Results}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174
392
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
175 \subsection{SDA vs MLP}
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
176
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
177 \begin{center}
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
178 \begin{tabular}{lcc}
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
179 & train w/ & train w/ \\
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
180 & NIST & P07 + NIST \\ \hline
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
181 SDA & & \\ \hline
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
182 MLP & & \\ \hline
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
183 \end{tabular}
5f8fffd7347f possible image for illustrating perturbations
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 381
diff changeset
184 \end{center}
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186 \subsection{Perturbed Training Data More Helpful for SDAE}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 \subsection{Training with More Classes than Necessary}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
189
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190 \section{Conclusions}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
191
407
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
192 \bibliography{strings,ml,aigaion,specials}
379
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193 \bibliographystyle{mlapa}
a21a174c1c18 added writeup skeleton
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194
407
fe2e2964e7a3 description des transformations en cours ajout d un fichier special.bib pour des references specifiques
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 393
diff changeset
195 \end{document}