annotate baseline_algorithms/mlp/mlp_nist.py @ 135:36010ff90255

Added script to test truetype font files validity (corruption)
author boulanni <nicolas_boulanger@hotmail.com>
date Sat, 20 Feb 2010 02:10:30 -0500
parents 93b4b84d86cf
children f341a4efb44a
rev   line source
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
1 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
3
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
9 the problem of MNIST digit classification.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
10
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
11 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
12
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
14
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
15 References:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
16
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
18 Christopher M. Bishop, section 5
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
19
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
21 to do lr first, then add regularization)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
22
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
23 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
25
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
26 import pdb
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
27 import numpy
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
28 import pylab
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
29 import theano
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
30 import theano.tensor as T
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
31 import time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
32 import theano.tensor.nnet
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
33 from pylearn.io import filetensor as ft
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
34
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
35 data_path = '/data/lisa/data/nist/by_class/'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
36
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
37 class MLP(object):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
38 """Multi-Layer Perceptron Class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
39
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
40 A multilayer perceptron is a feedforward artificial neural network model
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
41 that has one layer or more of hidden units and nonlinear activations.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
42 Intermidiate layers usually have as activation function thanh or the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
43 sigmoid function while the top layer is a softamx layer.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
44 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
45
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
46
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
47
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
48 def __init__(self, input, n_in, n_hidden, n_out):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
49 """Initialize the parameters for the multilayer perceptron
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
50
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
51 :param input: symbolic variable that describes the input of the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
52 architecture (one minibatch)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
53
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
54 :param n_in: number of input units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
55 which the datapoints lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
56
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
57 :param n_hidden: number of hidden units
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
58
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
59 :param n_out: number of output units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
60 which the labels lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
61
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
62 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
63
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
64 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
65 # example contains only one hidden layer, but one can have as many
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
66 # layers as he/she wishes, making the network deeper. The only
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
67 # problem making the network deep this way is during learning,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
68 # backpropagation being unable to move the network from the starting
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
69 # point towards; this is where pre-training helps, giving a good
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
70 # starting point for backpropagation, but more about this in the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
71 # other tutorials
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
72
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
73 # `W1` is initialized with `W1_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
74 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
75 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
76 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
77 W1_values = numpy.asarray( numpy.random.uniform( \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
78 low = -numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
79 high = numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
80 size = (n_in, n_hidden)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
81 # `W2` is initialized with `W2_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
82 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
83 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
84 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
85 W2_values = numpy.asarray( numpy.random.uniform(
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
86 low = -numpy.sqrt(6./(n_hidden+n_out)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
87 high= numpy.sqrt(6./(n_hidden+n_out)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
88 size= (n_hidden, n_out)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
89
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
90 self.W1 = theano.shared( value = W1_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
91 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
92 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
93 self.W2 = theano.shared( value = W2_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
94 self.b2 = theano.shared( value = numpy.zeros((n_out,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
95 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
96
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
97 # symbolic expression computing the values of the hidden layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
98 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
99
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
100 # symbolic expression computing the values of the top layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
101 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
102
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
103 # compute prediction as class whose probability is maximal in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
104 # symbolic form
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
105 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
106
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
107 # L1 norm ; one regularization option is to enforce L1 norm to
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
108 # be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
109 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
111 # square of L2 norm ; one regularization option is to enforce
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
112 # square of L2 norm to be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
113 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
114
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
115
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
116
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
117 def negative_log_likelihood(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
118 """Return the mean of the negative log-likelihood of the prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
119 of this model under a given target distribution.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
120
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
121 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
122
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
123 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
124 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
125 \ell (\theta=\{W,b\}, \mathcal{D})
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
126
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
127
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
128 :param y: corresponds to a vector that gives for each example the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
129 :correct label
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
130 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
131 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
132
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
133
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
134
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
135
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
136 def errors(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
137 """Return a float representing the number of errors in the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
138 over the total number of examples of the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
139 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
140
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
141 # check if y has same dimension of y_pred
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
142 if y.ndim != self.y_pred.ndim:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
143 raise TypeError('y should have the same shape as self.y_pred',
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
144 ('y', target.type, 'y_pred', self.y_pred.type))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
145 # check if y is of the correct datatype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
146 if y.dtype.startswith('int'):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
147 # the T.neq operator returns a vector of 0s and 1s, where 1
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
148 # represents a mistake in prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
149 return T.mean(T.neq(self.y_pred, y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
150 else:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
151 raise NotImplementedError()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
152
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
153 #def jobman_mlp(state,channel):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
154 # (validation_error,test_error,nb_exemples,time)=mlp_full_nist(state.learning_rate,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
155 # state.n_iter,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
156 # state.batch_size,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
157 # state.nb_hidden_units)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
158 # state.validation_error = validation_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
159 # state.test_error = test_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
160 # state.nb_exemples = nb_exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
161 # state.time=time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
162 # return channel.COMPLETE
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
163
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
164
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
165
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
166
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
167 def mlp_full_nist( verbose = False,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
168 train_data = 'all/all_train_data.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
169 train_labels = 'all/all_train_labels.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
170 test_data = 'all/all_test_data.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
171 test_labels = 'all/all_test_labels.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
172 learning_rate=0.01,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
173 L1_reg = 0.00,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
174 L2_reg = 0.0001,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
175 nb_max_exemples=1000000,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
176 batch_size=20,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
177 nb_hidden = 500,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
178 nb_targets = 62):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
179
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
180
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
181
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
182 f = open(data_path+train_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
183 g= open(data_path+train_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
184 h = open(data_path+test_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
185 i= open(data_path+test_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
186
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
187 raw_train_data = ft.read(f)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
188 raw_train_labels = ft.read(g)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
189 raw_test_data = ft.read(h)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
190 raw_test_labels = ft.read(i)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
191
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
192 f.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
193 g.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
194 i.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
195 h.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
196 #create a validation set the same size as the test size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
197 #use the end of the training array for this purpose
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
198 #discard the last remaining so we get a %batch_size number
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
199 test_size=len(raw_test_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
200 test_size = int(test_size/batch_size)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
201 test_size*=batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
202 train_size = len(raw_train_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
203 train_size = int(train_size/batch_size)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
204 train_size*=batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
205 validation_size =test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
206 offset = train_size-test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
207 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
208 print 'train size = %d' %train_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
209 print 'test size = %d' %test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
210 print 'valid size = %d' %validation_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
211 print 'offset = %d' %offset
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
212
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
213
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
214 train_set = (raw_train_data,raw_train_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
215 train_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
216 for i in xrange(0, train_size-test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
217 train_batches = train_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
218 [(raw_train_data[i:i+batch_size], raw_train_labels[i:i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
219
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
220 test_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
221 for i in xrange(0, test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
222 test_batches = test_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
223 [(raw_test_data[i:i+batch_size], raw_test_labels[i:i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
224
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
225 validation_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
226 for i in xrange(0, test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
227 validation_batches = validation_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
228 [(raw_train_data[offset+i:offset+i+batch_size], raw_train_labels[offset+i:offset+i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
229
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
230
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
231 ishape = (32,32) # this is the size of NIST images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
232
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
233 # allocate symbolic variables for the data
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
234 x = T.fmatrix() # the data is presented as rasterized images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
235 y = T.lvector() # the labels are presented as 1D vector of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
236 # [long int] labels
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
237
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
238 # construct the logistic regression class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
239 classifier = MLP( input=x.reshape((batch_size,32*32)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
240 n_in=32*32,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
241 n_hidden=nb_hidden,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
242 n_out=nb_targets)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
243
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
244 # the cost we minimize during training is the negative log likelihood of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
245 # the model plus the regularization terms (L1 and L2); cost is expressed
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
246 # here symbolically
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
247 cost = classifier.negative_log_likelihood(y) \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
248 + L1_reg * classifier.L1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
249 + L2_reg * classifier.L2_sqr
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
250
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
251 # compiling a theano function that computes the mistakes that are made by
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
252 # the model on a minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
253 test_model = theano.function([x,y], classifier.errors(y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
254
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
255 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
256 g_W1 = T.grad(cost, classifier.W1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
257 g_b1 = T.grad(cost, classifier.b1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
258 g_W2 = T.grad(cost, classifier.W2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
259 g_b2 = T.grad(cost, classifier.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
260
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
261 # specify how to update the parameters of the model as a dictionary
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
262 updates = \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
263 { classifier.W1: classifier.W1 - learning_rate*g_W1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
264 , classifier.b1: classifier.b1 - learning_rate*g_b1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
265 , classifier.W2: classifier.W2 - learning_rate*g_W2 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
266 , classifier.b2: classifier.b2 - learning_rate*g_b2 }
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
267
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
268 # compiling a theano function `train_model` that returns the cost, but in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
269 # the same time updates the parameter of the model based on the rules
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
270 # defined in `updates`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
271 train_model = theano.function([x, y], cost, updates = updates )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
272 n_minibatches = len(train_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
273
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
274
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
275
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
276
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
277 #conditions for stopping the adaptation:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
278 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
279 #2) validation error is going up (probable overfitting)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
280
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
281 # This means we no longer stop on slow convergence as low learning rates stopped
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
282 # too fast.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
283 patience =nb_max_exemples/batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
284 patience_increase = 2 # wait this much longer when a new best is
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
285 # found
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
286 improvement_threshold = 0.995 # a relative improvement of this much is
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
287 # considered significant
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
288 validation_frequency = n_minibatches/4
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
289
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
290
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
291
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
292
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
293 best_params = None
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
294 best_validation_loss = float('inf')
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
295 best_iter = 0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
296 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
297 start_time = time.clock()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
298 n_iter = nb_max_exemples/batch_size # nb of max times we are allowed to run through all exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
299 n_iter = n_iter/n_minibatches + 1
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
300 n_iter=max(1,n_iter) # run at least once on short debug call
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
301 # have a maximum of `n_iter` iterations through the entire dataset
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
302
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
303 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
304 print 'looping at most %d times through the data set' %n_iter
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
305 for iter in xrange(n_iter* n_minibatches):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
306
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
307 # get epoch and minibatch index
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
308 epoch = iter / n_minibatches
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
309 minibatch_index = iter % n_minibatches
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
310
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
311 # get the minibatches corresponding to `iter` modulo
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
312 # `len(train_batches)`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
313 x,y = train_batches[ minibatch_index ]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
314 # convert to float
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
315 x_float = x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
316 cost_ij = train_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
317
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
318 if (iter+1) % validation_frequency == 0:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
319 # compute zero-one loss on validation set
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
320
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
321 this_validation_loss = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
322 for x,y in validation_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
323 # sum up the errors for each minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
324 x_float = x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
325 this_validation_loss += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
326 # get the average by dividing with the number of minibatches
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
327 this_validation_loss /= len(validation_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
328 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
329 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
330 (epoch, minibatch_index+1, n_minibatches, \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
331 this_validation_loss*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
332
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
333
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
334 # if we got the best validation score until now
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
335 if this_validation_loss < best_validation_loss:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
336
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
337 #improve patience if loss improvement is good enough
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
338 if this_validation_loss < best_validation_loss * \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
339 improvement_threshold :
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
340 patience = max(patience, iter * patience_increase)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
341 elif verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
342 print 'slow convergence stop'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
343
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
344 # save best validation score and iteration number
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
345 best_validation_loss = this_validation_loss
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
346 best_iter = iter
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
347
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
348 # test it on the test set
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
349 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
350 for x,y in test_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
351 x_float=x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
352 test_score += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
353 test_score /= len(test_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
354 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
355 print((' epoch %i, minibatch %i/%i, test error of best '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
356 'model %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
357 (epoch, minibatch_index+1, n_minibatches,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
358 test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
359
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
360 #if the validation error is going up, we are overfitting
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
361 #stop converging
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
362 elif this_validation_loss > best_validation_loss:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
363 #calculate the test error at this point and exit
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
364 # test it on the test set
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
365 if verbose==True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
366 print ' We are diverging'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
367 best_iter = iter
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
368 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
369 for x,y in test_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
370 x_float=x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
371 test_score += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
372 test_score /= len(test_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
373 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
374 print ' validation error is going up, stopping now'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
375 print((' epoch %i, minibatch %i/%i, test error of best '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
376 'model %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
377 (epoch, minibatch_index+1, n_minibatches,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
378 test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
379
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
380 break
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
381
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
382
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
383
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
384 if patience <= iter :
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
385 break
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
386
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
387
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
388 end_time = time.clock()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
389 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
390 print(('Optimization complete. Best validation score of %f %% '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
391 'obtained at iteration %i, with test performance %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
392 (best_validation_loss * 100., best_iter, test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
393 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
394 print iter
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
395 return (best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
396
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
397
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
398 if __name__ == '__main__':
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
399 mlp_full_mnist()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
400
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
401 def jobman_mlp_full_nist(state,channel):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
402 (validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
403 nb_max_exemples=state.nb_max_exemples,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
404 nb_hidden=state.nb_hidden)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
405 state.validation_error=validation_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
406 state.test_error=test_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
407 state.nb_exemples=nb_exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
408 state.time=time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
409 return channel.COMPLETE
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
410
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
411