Mercurial > ift6266

annotate baseline/conv_mlp/convolutional_mlp.py @ 196:168aae8a6419

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix import errors (conv and downsample are no longer in sandbox).
author Arnaud Bergeron <abergeron@gmail.com>
date Tue, 02 Mar 2010 18:03:09 -0500
parents d37c944133c3
children a491d3600a77
rev   line source
146
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
1 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
2 This tutorial introduces the LeNet5 neural network architecture using Theano. LeNet5 is a
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
3 convolutional neural network, good for classifying images. This tutorial shows how to build the
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
4 architecture, and comes with all the hyper-parameters you need to reproduce the paper's MNIST
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
5 results.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
6
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
7 The best results are obtained after X iterations of the main program loop, which takes ***
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
8 minutes on my workstation (an Intel Core i7, circa July 2009), and *** minutes on my GPU (an
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
9 NVIDIA GTX 285 graphics processor).
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
10
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
11 This implementation simplifies the model in the following ways:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
12
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
13 - LeNetConvPool doesn't implement location-specific gain and bias parameters
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
14 - LeNetConvPool doesn't implement pooling by average, it implements pooling by max.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
15 - Digit classification is implemented with a logistic regression rather than an RBF network
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
16 - LeNet5 was not fully-connected convolutions at second layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
17
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
18 References:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
19 - Y. LeCun, L. Bottou, Y. Bengio and P. Haffner: Gradient-Based Learning Applied to Document
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
20 Recognition, Proceedings of the IEEE, 86(11):2278-2324, November 1998.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
21 http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
22 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
23
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
24 import numpy, theano, cPickle, gzip, time
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
25 import theano.tensor as T
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
26 import theano.sandbox.softsign
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
27 import pylearn.datasets.MNIST
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
28 from pylearn.io import filetensor as ft
196
168aae8a6419 Fix import errors (conv and downsample are no longer in sandbox).
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
29 from theano.tensor.signal import downsample
168aae8a6419 Fix import errors (conv and downsample are no longer in sandbox).
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
30 from theano.tensor.nnet import conv
146
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
31
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
32 class LeNetConvPoolLayer(object):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
33
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
34 def __init__(self, rng, input, filter_shape, image_shape, poolsize=(2,2)):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
35 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
36 Allocate a LeNetConvPoolLayer with shared variable internal parameters.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
37 :type rng: numpy.random.RandomState
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
38 :param rng: a random number generator used to initialize weights
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
39 :type input: theano.tensor.dtensor4
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
40 :param input: symbolic image tensor, of shape image_shape
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
41 :type filter_shape: tuple or list of length 4
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
42 :param filter_shape: (number of filters, num input feature maps,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
43 filter height,filter width)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
44 :type image_shape: tuple or list of length 4
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
45 :param image_shape: (batch size, num input feature maps,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
46 image height, image width)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
47 :type poolsize: tuple or list of length 2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
48 :param poolsize: the downsampling (pooling) factor (#rows,#cols)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
49 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
50 assert image_shape[1]==filter_shape[1]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
51 self.input = input
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
52
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
53 # initialize weight values: the fan-in of each hidden neuron is
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
54 # restricted by the size of the receptive fields.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
55 fan_in = numpy.prod(filter_shape[1:])
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
56 W_values = numpy.asarray( rng.uniform( \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
57 low = -numpy.sqrt(3./fan_in), \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
58 high = numpy.sqrt(3./fan_in), \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
59 size = filter_shape), dtype = theano.config.floatX)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
60 self.W = theano.shared(value = W_values)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
61
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
62 # the bias is a 1D tensor -- one bias per output feature map
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
63 b_values = numpy.zeros((filter_shape[0],), dtype= theano.config.floatX)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
64 self.b = theano.shared(value= b_values)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
65
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
66 # convolve input feature maps with filters
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
67 conv_out = conv.conv2d(input, self.W,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
68 filter_shape=filter_shape, image_shape=image_shape)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
69
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
70 # downsample each feature map individually, using maxpooling
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
71 pooled_out = downsample.max_pool2D(conv_out, poolsize, ignore_border=True)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
72
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
73 # add the bias term. Since the bias is a vector (1D array), we first
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
74 # reshape it to a tensor of shape (1,n_filters,1,1). Each bias will thus
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
75 # be broadcasted across mini-batches and feature map width & height
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
76 self.output = T.tanh(pooled_out + self.b.dimshuffle('x', 0, 'x', 'x'))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
77
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
78 # store parameters of this layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
79 self.params = [self.W, self.b]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
80
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
81
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
82 class SigmoidalLayer(object):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
83 def __init__(self, rng, input, n_in, n_out):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
84 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
85 Typical hidden layer of a MLP: units are fully-connected and have
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
86 sigmoidal activation function. Weight matrix W is of shape (n_in,n_out)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
87 and the bias vector b is of shape (n_out,).
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
88
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
89 Hidden unit activation is given by: sigmoid(dot(input,W) + b)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
90
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
91 :type rng: numpy.random.RandomState
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
92 :param rng: a random number generator used to initialize weights
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
93 :type input: theano.tensor.dmatrix
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
94 :param input: a symbolic tensor of shape (n_examples, n_in)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
95 :type n_in: int
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
96 :param n_in: dimensionality of input
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
97 :type n_out: int
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
98 :param n_out: number of hidden units
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
99 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
100 self.input = input
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
101
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
102 W_values = numpy.asarray( rng.uniform( \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
103 low = -numpy.sqrt(6./(n_in+n_out)), \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
104 high = numpy.sqrt(6./(n_in+n_out)), \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
105 size = (n_in, n_out)), dtype = theano.config.floatX)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
106 self.W = theano.shared(value = W_values)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
107
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
108 b_values = numpy.zeros((n_out,), dtype= theano.config.floatX)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
109 self.b = theano.shared(value= b_values)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
110
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
111 self.output = T.tanh(T.dot(input, self.W) + self.b)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
112 self.params = [self.W, self.b]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
113
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
114
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
115 class LogisticRegression(object):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
116 """Multi-class Logistic Regression Class
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
117
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
118 The logistic regression is fully described by a weight matrix :math:`W`
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
119 and bias vector :math:`b`. Classification is done by projecting data
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
120 points onto a set of hyperplanes, the distance to which is used to
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
121 determine a class membership probability.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
122 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
123
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
124 def __init__(self, input, n_in, n_out):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
125 """ Initialize the parameters of the logistic regression
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
126 :param input: symbolic variable that describes the input of the
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
127 architecture (one minibatch)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
128 :type n_in: int
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
129 :param n_in: number of input units, the dimension of the space in
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
130 which the datapoints lie
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
131 :type n_out: int
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
132 :param n_out: number of output units, the dimension of the space in
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
133 which the labels lie
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
134 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
135
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
136 # initialize with 0 the weights W as a matrix of shape (n_in, n_out)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
137 self.W = theano.shared( value=numpy.zeros((n_in,n_out),
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
138 dtype = theano.config.floatX) )
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
139 # initialize the baises b as a vector of n_out 0s
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
140 self.b = theano.shared( value=numpy.zeros((n_out,),
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
141 dtype = theano.config.floatX) )
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
142 # compute vector of class-membership probabilities in symbolic form
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
143 self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W)+self.b)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
144
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
145 # compute prediction as class whose probability is maximal in
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
146 # symbolic form
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
147 self.y_pred=T.argmax(self.p_y_given_x, axis=1)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
148
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
149 # list of parameters for this layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
150 self.params = [self.W, self.b]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
151
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
152 def negative_log_likelihood(self, y):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
153 """Return the mean of the negative log-likelihood of the prediction
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
154 of this model under a given target distribution.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
155 :param y: corresponds to a vector that gives for each example the
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
156 correct label
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
157 Note: we use the mean instead of the sum so that
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
158 the learning rate is less dependent on the batch size
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
159 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
160 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
161
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
162 def errors(self, y):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
163 """Return a float representing the number of errors in the minibatch
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
164 over the total number of examples of the minibatch ; zero one
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
165 loss over the size of the minibatch
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
166 """
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
167 # check if y has same dimension of y_pred
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
168 if y.ndim != self.y_pred.ndim:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
169 raise TypeError('y should have the same shape as self.y_pred',
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
170 ('y', target.type, 'y_pred', self.y_pred.type))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
171
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
172 # check if y is of the correct datatype
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
173 if y.dtype.startswith('int'):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
174 # the T.neq operator returns a vector of 0s and 1s, where 1
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
175 # represents a mistake in prediction
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
176 return T.mean(T.neq(self.y_pred, y))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
177 else:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
178 raise NotImplementedError()
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
179
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
180
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
181 def load_dataset(fname,batch=20):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
182
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
183 # repertoire qui contient les donnees NIST
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
184 # le repertoire suivant va fonctionner si vous etes connecte sur un ordinateur
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
185 # du reseau DIRO
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
186 datapath = '/data/lisa/data/nist/by_class/'
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
187 # le fichier .ft contient chiffres NIST dans un format efficace. Les chiffres
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
188 # sont stockes dans une matrice de NxD, ou N est le nombre d'images, est D est
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
189 # le nombre de pixels par image (32x32 = 1024). Chaque pixel de l'image est une
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
190 # valeur entre 0 et 255, correspondant a un niveau de gris. Les valeurs sont
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
191 # stockees comme des uint8, donc des bytes.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
192 f = open(datapath+'digits/digits_train_data.ft')
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
193 # Verifier que vous avez assez de memoire pour loader les donnees au complet
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
194 # dans le memoire. Sinon, utilisez ft.arraylike, une classe construite
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
195 # specialement pour des fichiers qu'on ne souhaite pas loader dans RAM.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
196 d = ft.read(f)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
197
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
198 # NB: N'oubliez pas de diviser les valeurs des pixels par 255. si jamais vous
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
199 # utilisez les donnees commes entrees dans un reseaux de neurones et que vous
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
200 # voulez des entres entre 0 et 1.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
201 # digits_train_data.ft contient les images, digits_train_labels.ft contient les
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
202 # etiquettes
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
203 f = open(datapath+'digits/digits_train_labels.ft')
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
204 labels = ft.read(f)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
205
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
206
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
207 # Load the dataset
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
208 #f = gzip.open(fname,'rb')
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
209 #train_set, valid_set, test_set = cPickle.load(f)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
210 #f.close()
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
211
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
212 # make minibatches of size 20
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
213 batch_size = batch # sized of the minibatch
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
214
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
215 # Dealing with the training set
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
216 # get the list of training images (x) and their labels (y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
217 (train_set_x, train_set_y) = (d[:4000,:],labels[:4000])
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
218 # initialize the list of training minibatches with empty list
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
219 train_batches = []
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
220 for i in xrange(0, len(train_set_x), batch_size):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
221 # add to the list of minibatches the minibatch starting at
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
222 # position i, ending at position i+batch_size
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
223 # a minibatch is a pair ; the first element of the pair is a list
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
224 # of datapoints, the second element is the list of corresponding
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
225 # labels
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
226 train_batches = train_batches + \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
227 [(train_set_x[i:i+batch_size], train_set_y[i:i+batch_size])]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
228
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
229 #print train_batches[500]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
230
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
231 # Dealing with the validation set
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
232 (valid_set_x, valid_set_y) = (d[4000:5000,:],labels[4000:5000])
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
233 # initialize the list of validation minibatches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
234 valid_batches = []
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
235 for i in xrange(0, len(valid_set_x), batch_size):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
236 valid_batches = valid_batches + \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
237 [(valid_set_x[i:i+batch_size], valid_set_y[i:i+batch_size])]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
238
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
239 # Dealing with the testing set
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
240 (test_set_x, test_set_y) = (d[5000:6000,:],labels[5000:6000])
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
241 # initialize the list of testing minibatches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
242 test_batches = []
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
243 for i in xrange(0, len(test_set_x), batch_size):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
244 test_batches = test_batches + \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
245 [(test_set_x[i:i+batch_size], test_set_y[i:i+batch_size])]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
246
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
247 return train_batches, valid_batches, test_batches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
248
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
249
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
250 def evaluate_lenet5(learning_rate=0.1, n_iter=1, batch_size=20, n_kern0=20,n_kern1=50,filter_shape=5,n_layer=3, dataset='mnist.pkl.gz'):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
251 rng = numpy.random.RandomState(23455)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
252
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
253 print 'Before load dataset'
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
254 train_batches, valid_batches, test_batches = load_dataset(dataset,batch_size)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
255 print 'After load dataset'
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
256
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
257 ishape = (32,32) # this is the size of NIST images
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
258 n_kern2=80
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
259
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
260 # allocate symbolic variables for the data
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
261 x = T.matrix('x') # rasterized images
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
262 y = T.lvector() # the labels are presented as 1D vector of [long int] labels
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
263
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
264
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
265 ######################
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
266 # BUILD ACTUAL MODEL #
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
267 ######################
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
268
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
269 # Reshape matrix of rasterized images of shape (batch_size,28*28)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
270 # to a 4D tensor, compatible with our LeNetConvPoolLayer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
271 layer0_input = x.reshape((batch_size,1,32,32))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
272
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
273 # Construct the first convolutional pooling layer:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
274 # filtering reduces the image size to (32-5+1,32-5+1)=(28,28)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
275 # maxpooling reduces this further to (28/2,28/2) = (14,14)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
276 # 4D output tensor is thus of shape (20,20,14,14)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
277 layer0 = LeNetConvPoolLayer(rng, input=layer0_input,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
278 image_shape=(batch_size,1,32,32),
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
279 filter_shape=(n_kern0,1,filter_shape,filter_shape), poolsize=(2,2))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
280
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
281 if(n_layer>2):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
282
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
283 # Construct the second convolutional pooling layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
284 # filtering reduces the image size to (14-5+1,14-5+1)=(10,10)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
285 # maxpooling reduces this further to (10/2,10/2) = (5,5)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
286 # 4D output tensor is thus of shape (20,50,5,5)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
287 fshape=(32-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
288 layer1 = LeNetConvPoolLayer(rng, input=layer0.output,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
289 image_shape=(batch_size,n_kern0,fshape,fshape),
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
290 filter_shape=(n_kern1,n_kern0,filter_shape,filter_shape), poolsize=(2,2))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
291
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
292 else:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
293
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
294 fshape=(32-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
295 layer1_input = layer0.output.flatten(2)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
296 # construct a fully-connected sigmoidal layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
297 layer1 = SigmoidalLayer(rng, input=layer1_input,n_in=n_kern0*fshape*fshape, n_out=500)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
298
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
299 layer2 = LogisticRegression(input=layer1.output, n_in=500, n_out=10)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
300 cost = layer2.negative_log_likelihood(y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
301 test_model = theano.function([x,y], layer2.errors(y))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
302 params = layer2.params+ layer1.params + layer0.params
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
303
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
304
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
305 if(n_layer>3):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
306
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
307 fshape=(32-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
308 fshape2=(fshape-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
309 fshape3=(fshape2-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
310 layer2 = LeNetConvPoolLayer(rng, input=layer1.output,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
311 image_shape=(batch_size,n_kern1,fshape2,fshape2),
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
312 filter_shape=(n_kern2,n_kern1,filter_shape,filter_shape), poolsize=(2,2))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
313
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
314 layer3_input = layer2.output.flatten(2)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
315
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
316 layer3 = SigmoidalLayer(rng, input=layer3_input,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
317 n_in=n_kern2*fshape3*fshape3, n_out=500)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
318
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
319
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
320 layer4 = LogisticRegression(input=layer3.output, n_in=500, n_out=10)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
321
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
322 cost = layer4.negative_log_likelihood(y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
323
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
324 test_model = theano.function([x,y], layer4.errors(y))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
325
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
326 params = layer4.params+ layer3.params+ layer2.params+ layer1.params + layer0.params
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
327
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
328
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
329 elif(n_layer>2):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
330
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
331 fshape=(32-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
332 fshape2=(fshape-filter_shape+1)/2
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
333
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
334 # the SigmoidalLayer being fully-connected, it operates on 2D matrices of
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
335 # shape (batch_size,num_pixels) (i.e matrix of rasterized images).
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
336 # This will generate a matrix of shape (20,32*4*4) = (20,512)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
337 layer2_input = layer1.output.flatten(2)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
338
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
339 # construct a fully-connected sigmoidal layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
340 layer2 = SigmoidalLayer(rng, input=layer2_input,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
341 n_in=n_kern1*fshape2*fshape2, n_out=500)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
342
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
343
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
344 # classify the values of the fully-connected sigmoidal layer
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
345 layer3 = LogisticRegression(input=layer2.output, n_in=500, n_out=10)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
346
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
347 # the cost we minimize during training is the NLL of the model
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
348 cost = layer3.negative_log_likelihood(y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
349
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
350 # create a function to compute the mistakes that are made by the model
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
351 test_model = theano.function([x,y], layer3.errors(y))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
352
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
353 # create a list of all model parameters to be fit by gradient descent
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
354 params = layer3.params+ layer2.params+ layer1.params + layer0.params
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
355
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
356
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
357
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
358
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
359
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
360 # create a list of gradients for all model parameters
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
361 grads = T.grad(cost, params)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
362
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
363 # train_model is a function that updates the model parameters by SGD
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
364 # Since this model has many parameters, it would be tedious to manually
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
365 # create an update rule for each model parameter. We thus create the updates
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
366 # dictionary by automatically looping over all (params[i],grads[i]) pairs.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
367 updates = {}
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
368 for param_i, grad_i in zip(params, grads):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
369 updates[param_i] = param_i - learning_rate * grad_i
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
370 train_model = theano.function([x, y], cost, updates=updates)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
371
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
372
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
373 ###############
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
374 # TRAIN MODEL #
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
375 ###############
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
376
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
377 n_minibatches = len(train_batches)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
378
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
379 # early-stopping parameters
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
380 patience = 10000 # look as this many examples regardless
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
381 patience_increase = 2 # wait this much longer when a new best is
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
382 # found
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
383 improvement_threshold = 0.995 # a relative improvement of this much is
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
384 # considered significant
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
385 validation_frequency = n_minibatches # go through this many
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
386 # minibatche before checking the network
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
387 # on the validation set; in this case we
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
388 # check every epoch
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
389
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
390 best_params = None
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
391 best_validation_loss = float('inf')
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
392 best_iter = 0
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
393 test_score = 0.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
394 start_time = time.clock()
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
395
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
396 # have a maximum of `n_iter` iterations through the entire dataset
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
397 for iter in xrange(n_iter * n_minibatches):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
398
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
399 # get epoch and minibatch index
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
400 epoch = iter / n_minibatches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
401 minibatch_index = iter % n_minibatches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
402
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
403 # get the minibatches corresponding to `iter` modulo
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
404 # `len(train_batches)`
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
405 x,y = train_batches[ minibatch_index ]
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
406
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
407 if iter %100 == 0:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
408 print 'training @ iter = ', iter
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
409 cost_ij = train_model(x,y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
410
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
411 if (iter+1) % validation_frequency == 0:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
412
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
413 # compute zero-one loss on validation set
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
414 this_validation_loss = 0.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
415 for x,y in valid_batches:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
416 # sum up the errors for each minibatch
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
417 this_validation_loss += test_model(x,y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
418
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
419 # get the average by dividing with the number of minibatches
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
420 this_validation_loss /= len(valid_batches)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
421 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
422 (epoch, minibatch_index+1, n_minibatches, \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
423 this_validation_loss*100.))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
424
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
425
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
426 # if we got the best validation score until now
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
427 if this_validation_loss < best_validation_loss:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
428
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
429 #improve patience if loss improvement is good enough
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
430 if this_validation_loss < best_validation_loss * \
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
431 improvement_threshold :
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
432 patience = max(patience, iter * patience_increase)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
433
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
434 # save best validation score and iteration number
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
435 best_validation_loss = this_validation_loss
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
436 best_iter = iter
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
437
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
438 # test it on the test set
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
439 test_score = 0.
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
440 for x,y in test_batches:
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
441 test_score += test_model(x,y)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
442 test_score /= len(test_batches)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
443 print((' epoch %i, minibatch %i/%i, test error of best '
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
444 'model %f %%') %
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
445 (epoch, minibatch_index+1, n_minibatches,
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
446 test_score*100.))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
447
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
448 if patience <= iter :
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
449 break
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
450
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
451 end_time = time.clock()
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
452 print('Optimization complete.')
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
453 print('Best validation score of %f %% obtained at iteration %i,'\
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
454 'with test performance %f %%' %
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
455 (best_validation_loss * 100., best_iter, test_score*100.))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
456 print('The code ran for %f minutes' % ((end_time-start_time)/60.))
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
457
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
458 return (best_validation_loss * 100., test_score*100., (end_time-start_time)/60., best_iter)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
459
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
460 if __name__ == '__main__':
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
461 evaluate_lenet5()
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
462
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
463 def experiment(state, channel):
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
464 print 'start experiment'
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
465 (best_validation_loss, test_score, minutes_trained, iter) = evaluate_lenet5(state.learning_rate, state.n_iter, state.batch_size, state.n_kern0, state.n_kern1, state.filter_shape, state.n_layer)
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
466 print 'end experiment'
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
467
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
468 state.best_validation_loss = best_validation_loss
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
469 state.test_score = test_score
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
470 state.minutes_trained = minutes_trained
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
471 state.iter = iter
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
472
33038ab4e799 Reseau a convolution
Jeremy Eustache <jeremy.eustache@voila.fr>
parents:
diff changeset
473 return channel.COMPLETE