Mercurial > ift6266

annotate baseline/mlp/mlp_nist.py @ 336:a79db7cee035

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Arrange pour avoir un taux d'apprentissage decroissant decent pour NIST
author sylvainpl
date Thu, 15 Apr 2010 14:41:00 -0400
parents 1763c64030d1
children fca22114bb23
rev   line source
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
1 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
3
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
9 the problem of MNIST digit classification.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
10
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
11 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
12
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
14
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
15 References:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
16
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
18 Christopher M. Bishop, section 5
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
19
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
21 to do lr first, then add regularization)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
22
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
23 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
25
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
26 import pdb
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
27 import numpy
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
28 import pylab
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
29 import theano
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
30 import theano.tensor as T
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
31 import time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
32 import theano.tensor.nnet
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
33 import pylearn
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
34 import theano,pylearn.version,ift6266
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
35 from pylearn.io import filetensor as ft
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
36 from ift6266 import datasets
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
37
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
38 data_path = '/data/lisa/data/nist/by_class/'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
39
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
40 class MLP(object):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
41 """Multi-Layer Perceptron Class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
42
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
43 A multilayer perceptron is a feedforward artificial neural network model
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
44 that has one layer or more of hidden units and nonlinear activations.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
45 Intermidiate layers usually have as activation function thanh or the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
46 sigmoid function while the top layer is a softamx layer.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
47 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
48
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
49
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
50
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
51 def __init__(self, input, n_in, n_hidden, n_out,learning_rate):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
52 """Initialize the parameters for the multilayer perceptron
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
53
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
54 :param input: symbolic variable that describes the input of the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
55 architecture (one minibatch)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
56
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
57 :param n_in: number of input units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
58 which the datapoints lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
59
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
60 :param n_hidden: number of hidden units
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
61
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
62 :param n_out: number of output units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
63 which the labels lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
64
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
65 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
66
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
67 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
68 # example contains only one hidden layer, but one can have as many
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
69 # layers as he/she wishes, making the network deeper. The only
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
70 # problem making the network deep this way is during learning,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
71 # backpropagation being unable to move the network from the starting
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
72 # point towards; this is where pre-training helps, giving a good
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
73 # starting point for backpropagation, but more about this in the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
74 # other tutorials
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
75
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
76 # `W1` is initialized with `W1_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
77 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
78 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
79 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
80 W1_values = numpy.asarray( numpy.random.uniform( \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
81 low = -numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
82 high = numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
83 size = (n_in, n_hidden)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
84 # `W2` is initialized with `W2_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
85 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
86 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
87 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
88 W2_values = numpy.asarray( numpy.random.uniform(
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
89 low = -numpy.sqrt(6./(n_hidden+n_out)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
90 high= numpy.sqrt(6./(n_hidden+n_out)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
91 size= (n_hidden, n_out)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
92
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
93 self.W1 = theano.shared( value = W1_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
94 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
95 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
96 self.W2 = theano.shared( value = W2_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
97 self.b2 = theano.shared( value = numpy.zeros((n_out,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
98 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
99
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
100 #include the learning rate in the classifer so
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
101 #we can modify it on the fly when we want
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
102 lr_value=learning_rate
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
103 self.lr=theano.shared(value=lr_value)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
104 # symbolic expression computing the values of the hidden layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
105 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
106
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
107
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
108
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
109 # symbolic expression computing the values of the top layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
110 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
111
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
112 # compute prediction as class whose probability is maximal in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
113 # symbolic form
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
114 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
115 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
116
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
117
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
118
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
119
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
120 # L1 norm ; one regularization option is to enforce L1 norm to
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
121 # be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
122 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
123
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
124 # square of L2 norm ; one regularization option is to enforce
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
125 # square of L2 norm to be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
126 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
127
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
128
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
129
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
130 def negative_log_likelihood(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
131 """Return the mean of the negative log-likelihood of the prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
132 of this model under a given target distribution.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
133
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
134 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
135
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
136 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
137 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
138 \ell (\theta=\{W,b\}, \mathcal{D})
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
139
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
140
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
141 :param y: corresponds to a vector that gives for each example the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
142 :correct label
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
143 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
144 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
145
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
146
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
147
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
148
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
149 def errors(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
150 """Return a float representing the number of errors in the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
151 over the total number of examples of the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
152 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
153
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
154 # check if y has same dimension of y_pred
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
155 if y.ndim != self.y_pred.ndim:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
156 raise TypeError('y should have the same shape as self.y_pred',
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
157 ('y', target.type, 'y_pred', self.y_pred.type))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
158 # check if y is of the correct datatype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
159 if y.dtype.startswith('int'):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
160 # the T.neq operator returns a vector of 0s and 1s, where 1
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
161 # represents a mistake in prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
162 return T.mean(T.neq(self.y_pred, y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
163 else:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
164 raise NotImplementedError()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
165
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
166
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
167 def mlp_full_nist( verbose = 1,\
145
8ceaaf812891 changed adaptive lr flag from bool to int for jobman issues
XavierMuller
parents: 143
diff changeset
168 adaptive_lr = 0,\
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
169 data_set=0,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
170 learning_rate=0.01,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
171 L1_reg = 0.00,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
172 L2_reg = 0.0001,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
173 nb_max_exemples=1000000,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
174 batch_size=20,\
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
175 nb_hidden = 30,\
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
176 nb_targets = 62,
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
177 tau=1e6,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
178 lr_t2_factor=0.5):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
179
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
180
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
181 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
182
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
183 #save initial learning rate if classical adaptive lr is used
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
184 initial_lr=learning_rate
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
185 max_div_count=3
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
186
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
187
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
188 total_validation_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
189 total_train_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
190 learning_rate_list=[]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
191 best_training_error=float('inf');
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
192 divergence_flag_list=[]
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
193
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
194 if data_set==0:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
195 dataset=datasets.nist_all()
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
196 elif data_set==1:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
197 dataset=datasets.nist_P07()
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
198
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
199
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
200
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
201
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
202 ishape = (32,32) # this is the size of NIST images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
203
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
204 # allocate symbolic variables for the data
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
205 x = T.fmatrix() # the data is presented as rasterized images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
206 y = T.lvector() # the labels are presented as 1D vector of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
207 # [long int] labels
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
208
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
209
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
210 # construct the logistic regression class
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
211 classifier = MLP( input=x,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
212 n_in=32*32,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
213 n_hidden=nb_hidden,\
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
214 n_out=nb_targets,
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
215 learning_rate=learning_rate)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
216
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
217
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
218
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
219
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
220 # the cost we minimize during training is the negative log likelihood of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
221 # the model plus the regularization terms (L1 and L2); cost is expressed
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
222 # here symbolically
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
223 cost = classifier.negative_log_likelihood(y) \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
224 + L1_reg * classifier.L1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
225 + L2_reg * classifier.L2_sqr
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
226
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
227 # compiling a theano function that computes the mistakes that are made by
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
228 # the model on a minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
229 test_model = theano.function([x,y], classifier.errors(y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
230
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
231 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
232 g_W1 = T.grad(cost, classifier.W1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
233 g_b1 = T.grad(cost, classifier.b1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
234 g_W2 = T.grad(cost, classifier.W2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
235 g_b2 = T.grad(cost, classifier.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
236
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
237 # specify how to update the parameters of the model as a dictionary
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
238 updates = \
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
239 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
240 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
241 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
242 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
243
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
244 # compiling a theano function `train_model` that returns the cost, but in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
245 # the same time updates the parameter of the model based on the rules
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
246 # defined in `updates`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
247 train_model = theano.function([x, y], cost, updates = updates )
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
248
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
249
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
250
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
251
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
252
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
253
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
254
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
255
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
256
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
257 #conditions for stopping the adaptation:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
258 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size so we always do at least 1 epoch)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
259 #2) validation error is going up twice in a row(probable overfitting)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
260
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
261 # This means we no longer stop on slow convergence as low learning rates stopped
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
262 # too fast but instead we will wait for the valid error going up 3 times in a row
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
263 # We save the curb of the validation error so we can always go back to check on it
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
264 # and we save the absolute best model anyway, so we might as well explore
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
265 # a bit when diverging
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
266
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
267 #approximate number of samples in the nist training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
268 #this is just to have a validation frequency
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
269 #roughly proportionnal to the original nist training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
270 n_minibatches = 650000/batch_size
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
271
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
272
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
273 patience =2*nb_max_exemples/batch_size #in units of minibatch
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
274 validation_frequency = n_minibatches/4
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
275
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
276
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
277
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
278
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
279
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
280 best_validation_loss = float('inf')
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
281 best_iter = 0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
282 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
283 start_time = time.clock()
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
284 time_n=0 #in unit of exemples
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
285 minibatch_index=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
286 epoch=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
287 temp=0
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
288 divergence_flag=0
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
289
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
290
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
291
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
292 if verbose == 1:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
293 print 'starting training'
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
294 while(minibatch_index*batch_size<nb_max_exemples):
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
295
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
296 for x, y in dataset.train(batch_size):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
297
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
298 #if we are using the classic learning rate deacay, adjust it before training of current mini-batch
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
299 if adaptive_lr==2:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
300 classifier.lr.value = tau*initial_lr/(tau+time_n)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
301
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
302
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
303 #train model
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
304 cost_ij = train_model(x,y)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
305
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
306 if (minibatch_index+1) % validation_frequency == 0:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
307 #save the current learning rate
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
308 learning_rate_list.append(classifier.lr.value)
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
309 divergence_flag_list.append(divergence_flag)
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
310
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
311 # compute the validation error
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
312 this_validation_loss = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
313 temp=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
314 for xv,yv in dataset.valid(1):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
315 # sum up the errors for each minibatch
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
316 this_validation_loss += test_model(xv,yv)
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
317 temp=temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
318 # get the average by dividing with the number of minibatches
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
319 this_validation_loss /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
320 #save the validation loss
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
321 total_validation_error_list.append(this_validation_loss)
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
322 if verbose == 1:
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
323 print(('epoch %i, minibatch %i, learning rate %f current validation error %f ') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
324 (epoch, minibatch_index+1,classifier.lr.value,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
325 this_validation_loss*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
326
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
327 # if we got the best validation score until now
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
328 if this_validation_loss < best_validation_loss:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
329 # save best validation score and iteration number
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
330 best_validation_loss = this_validation_loss
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
331 best_iter = minibatch_index
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
332 #reset divergence flag
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
333 divergence_flag=0
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
334
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
335 #save the best model. Overwrite the current saved best model so
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
336 #we only keep the best
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
337 numpy.savez('best_model.npy', config=configuration, W1=classifier.W1.value, W2=classifier.W2.value, b1=classifier.b1.value,\
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
338 b2=classifier.b2.value, minibatch_index=minibatch_index)
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
339
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
340 # test it on the test set
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
341 test_score = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
342 temp =0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
343 for xt,yt in dataset.test(batch_size):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
344 test_score += test_model(xt,yt)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
345 temp = temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
346 test_score /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
347 if verbose == 1:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
348 print(('epoch %i, minibatch %i, test error of best '
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
349 'model %f %%') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
350 (epoch, minibatch_index+1,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
351 test_score*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
352
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
353 # if the validation error is going up, we are overfitting (or oscillating)
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
354 # check if we are allowed to continue and if we will adjust the learning rate
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
355 elif this_validation_loss >= best_validation_loss:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
356
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
357
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
358 # In non-classic learning rate decay, we modify the weight only when
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
359 # validation error is going up
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
360 if adaptive_lr==1:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
361 classifier.lr.value=classifier.lr.value*lr_t2_factor
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
362
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
363
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
364 #cap the patience so we are allowed to diverge max_div_count times
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
365 #if we are going up max_div_count in a row, we will stop immediatelty by modifying the patience
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
366 divergence_flag = divergence_flag +1
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
367
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
368
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
369 #calculate the test error at this point and exit
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
370 # test it on the test set
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
371 test_score = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
372 temp=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
373 for xt,yt in dataset.test(batch_size):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
374 test_score += test_model(xt,yt)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
375 temp=temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
376 test_score /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
377 if verbose == 1:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
378 print ' validation error is going up, possibly stopping soon'
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
379 print((' epoch %i, minibatch %i, test error of best '
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
380 'model %f %%') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
381 (epoch, minibatch_index+1,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
382 test_score*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
383
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
384
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
385
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
386 # check early stop condition
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
387 if divergence_flag==max_div_count:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
388 minibatch_index=nb_max_exemples
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
389 print 'we have diverged, early stopping kicks in'
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
390 break
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
391
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
392 #check if we have seen enough exemples
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
393 #force one epoch at least
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
394 if epoch>0 and minibatch_index*batch_size>nb_max_exemples:
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
395 break
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
396
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
397
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
398 time_n= time_n + batch_size
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
399 minibatch_index = minibatch_index + 1
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
400
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
401 # we have finished looping through the training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
402 epoch = epoch+1
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
403 end_time = time.clock()
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
404 if verbose == 1:
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
405 print(('Optimization complete. Best validation score of %f %% '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
406 'obtained at iteration %i, with test performance %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
407 (best_validation_loss * 100., best_iter, test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
408 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
409 print minibatch_index
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
410
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
411 #save the model and the weights
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
412 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
413 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
414 learning_rate_list=learning_rate_list, divergence_flag_list=divergence_flag_list)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
415
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
416 return (best_training_error*100.0,best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
417
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
418
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
419 if __name__ == '__main__':
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
420 mlp_full_mnist()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
421
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
422 def jobman_mlp_full_nist(state,channel):
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
423 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
424 nb_max_exemples=state.nb_max_exemples,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
425 nb_hidden=state.nb_hidden,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
426 adaptive_lr=state.adaptive_lr,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
427 tau=state.tau,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
428 verbose = state.verbose,\
324
1763c64030d1 fixed bug in jobman interface
xaviermuller
parents: 323
diff changeset
429 lr_t2_factor=state.lr_t2_factor,
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
430 data_set=state.data_set)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
431 state.train_error=train_error
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
432 state.validation_error=validation_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
433 state.test_error=test_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
434 state.nb_exemples=nb_exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
435 state.time=time
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
436 pylearn.version.record_versions(state,[theano,ift6266,pylearn])
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
437 return channel.COMPLETE
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
438
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
439