Mercurial > ift6266

annotate baseline/mlp/mlp_nist.py @ 324:1763c64030d1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixed bug in jobman interface
author xaviermuller
date Thu, 08 Apr 2010 11:22:11 -0400
parents 7a7615f940e8
children fca22114bb23
rev   line source
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
1 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
3
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
9 the problem of MNIST digit classification.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
10
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
11 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
12
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
14
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
15 References:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
16
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
18 Christopher M. Bishop, section 5
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
19
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
21 to do lr first, then add regularization)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
22
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
23 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
25
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
26 import pdb
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
27 import numpy
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
28 import pylab
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
29 import theano
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
30 import theano.tensor as T
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
31 import time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
32 import theano.tensor.nnet
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
33 import pylearn
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
34 import theano,pylearn.version,ift6266
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
35 from pylearn.io import filetensor as ft
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
36 from ift6266 import datasets
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
37
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
38 data_path = '/data/lisa/data/nist/by_class/'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
39
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
40 class MLP(object):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
41 """Multi-Layer Perceptron Class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
42
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
43 A multilayer perceptron is a feedforward artificial neural network model
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
44 that has one layer or more of hidden units and nonlinear activations.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
45 Intermidiate layers usually have as activation function thanh or the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
46 sigmoid function while the top layer is a softamx layer.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
47 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
48
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
49
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
50
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
51 def __init__(self, input, n_in, n_hidden, n_out,learning_rate):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
52 """Initialize the parameters for the multilayer perceptron
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
53
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
54 :param input: symbolic variable that describes the input of the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
55 architecture (one minibatch)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
56
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
57 :param n_in: number of input units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
58 which the datapoints lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
59
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
60 :param n_hidden: number of hidden units
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
61
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
62 :param n_out: number of output units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
63 which the labels lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
64
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
65 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
66
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
67 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
68 # example contains only one hidden layer, but one can have as many
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
69 # layers as he/she wishes, making the network deeper. The only
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
70 # problem making the network deep this way is during learning,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
71 # backpropagation being unable to move the network from the starting
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
72 # point towards; this is where pre-training helps, giving a good
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
73 # starting point for backpropagation, but more about this in the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
74 # other tutorials
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
75
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
76 # `W1` is initialized with `W1_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
77 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
78 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
79 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
80 W1_values = numpy.asarray( numpy.random.uniform( \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
81 low = -numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
82 high = numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
83 size = (n_in, n_hidden)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
84 # `W2` is initialized with `W2_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
85 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
86 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
87 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
88 W2_values = numpy.asarray( numpy.random.uniform(
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
89 low = -numpy.sqrt(6./(n_hidden+n_out)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
90 high= numpy.sqrt(6./(n_hidden+n_out)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
91 size= (n_hidden, n_out)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
92
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
93 self.W1 = theano.shared( value = W1_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
94 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
95 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
96 self.W2 = theano.shared( value = W2_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
97 self.b2 = theano.shared( value = numpy.zeros((n_out,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
98 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
99
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
100 #include the learning rate in the classifer so
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
101 #we can modify it on the fly when we want
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
102 lr_value=learning_rate
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
103 self.lr=theano.shared(value=lr_value)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
104 # symbolic expression computing the values of the hidden layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
105 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
106
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
107
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
108
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
109 # symbolic expression computing the values of the top layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
110 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
111
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
112 # compute prediction as class whose probability is maximal in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
113 # symbolic form
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
114 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
115 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
116
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
117
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
118
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
119
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
120 # L1 norm ; one regularization option is to enforce L1 norm to
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
121 # be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
122 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
123
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
124 # square of L2 norm ; one regularization option is to enforce
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
125 # square of L2 norm to be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
126 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
127
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
128
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
129
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
130 def negative_log_likelihood(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
131 """Return the mean of the negative log-likelihood of the prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
132 of this model under a given target distribution.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
133
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
134 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
135
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
136 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
137 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
138 \ell (\theta=\{W,b\}, \mathcal{D})
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
139
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
140
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
141 :param y: corresponds to a vector that gives for each example the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
142 :correct label
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
143 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
144 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
145
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
146
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
147
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
148
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
149 def errors(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
150 """Return a float representing the number of errors in the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
151 over the total number of examples of the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
152 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
153
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
154 # check if y has same dimension of y_pred
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
155 if y.ndim != self.y_pred.ndim:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
156 raise TypeError('y should have the same shape as self.y_pred',
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
157 ('y', target.type, 'y_pred', self.y_pred.type))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
158 # check if y is of the correct datatype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
159 if y.dtype.startswith('int'):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
160 # the T.neq operator returns a vector of 0s and 1s, where 1
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
161 # represents a mistake in prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
162 return T.mean(T.neq(self.y_pred, y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
163 else:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
164 raise NotImplementedError()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
165
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
166
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
167 def mlp_full_nist( verbose = 1,\
145
8ceaaf812891 changed adaptive lr flag from bool to int for jobman issues
XavierMuller
parents: 143
diff changeset
168 adaptive_lr = 0,\
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
169 data_set=0,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
170 learning_rate=0.01,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
171 L1_reg = 0.00,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
172 L2_reg = 0.0001,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
173 nb_max_exemples=1000000,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
174 batch_size=20,\
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
175 nb_hidden = 30,\
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
176 nb_targets = 62,
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
177 tau=1e6,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
178 lr_t2_factor=0.5):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
179
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
180
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
181 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
182
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
183 #save initial learning rate if classical adaptive lr is used
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
184 initial_lr=learning_rate
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
185 max_div_count=3
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
186
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
187
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
188 total_validation_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
189 total_train_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
190 learning_rate_list=[]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
191 best_training_error=float('inf');
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
192 divergence_flag_list=[]
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
193
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
194 if data_set==0:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
195 dataset=datasets.nist_all()
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
196 elif data_set==1:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
197 dataset=datasets.nist_P07()
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
198
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
199
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
200
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
201
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
202 ishape = (32,32) # this is the size of NIST images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
203
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
204 # allocate symbolic variables for the data
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
205 x = T.fmatrix() # the data is presented as rasterized images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
206 y = T.lvector() # the labels are presented as 1D vector of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
207 # [long int] labels
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
208
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
209
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
210 # construct the logistic regression class
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
211 classifier = MLP( input=x,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
212 n_in=32*32,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
213 n_hidden=nb_hidden,\
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
214 n_out=nb_targets,
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
215 learning_rate=learning_rate)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
216
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
217
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
218
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
219
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
220 # the cost we minimize during training is the negative log likelihood of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
221 # the model plus the regularization terms (L1 and L2); cost is expressed
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
222 # here symbolically
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
223 cost = classifier.negative_log_likelihood(y) \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
224 + L1_reg * classifier.L1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
225 + L2_reg * classifier.L2_sqr
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
226
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
227 # compiling a theano function that computes the mistakes that are made by
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
228 # the model on a minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
229 test_model = theano.function([x,y], classifier.errors(y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
230
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
231 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
232 g_W1 = T.grad(cost, classifier.W1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
233 g_b1 = T.grad(cost, classifier.b1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
234 g_W2 = T.grad(cost, classifier.W2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
235 g_b2 = T.grad(cost, classifier.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
236
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
237 # specify how to update the parameters of the model as a dictionary
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
238 updates = \
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
239 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
240 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
241 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
242 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
243
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
244 # compiling a theano function `train_model` that returns the cost, but in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
245 # the same time updates the parameter of the model based on the rules
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
246 # defined in `updates`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
247 train_model = theano.function([x, y], cost, updates = updates )
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
248
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
249
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
250
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
251
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
252
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
253
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
254
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
255
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
256
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
257 #conditions for stopping the adaptation:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
258 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size so we always do at least 1 epoch)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
259 #2) validation error is going up twice in a row(probable overfitting)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
260
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
261 # This means we no longer stop on slow convergence as low learning rates stopped
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
262 # too fast but instead we will wait for the valid error going up 3 times in a row
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
263 # We save the curb of the validation error so we can always go back to check on it
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
264 # and we save the absolute best model anyway, so we might as well explore
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
265 # a bit when diverging
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
266
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
267 #approximate number of samples in the nist training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
268 #this is just to have a validation frequency
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
269 #roughly proportionnal to the original nist training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
270 n_minibatches = 650000/batch_size
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
271
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
272
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
273 patience =2*nb_max_exemples/batch_size #in units of minibatch
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
274 validation_frequency = n_minibatches/4
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
275
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
276
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
277
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
278
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
279
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
280 best_validation_loss = float('inf')
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
281 best_iter = 0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
282 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
283 start_time = time.clock()
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
284 time_n=0 #in unit of exemples
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
285 minibatch_index=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
286 epoch=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
287 temp=0
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
288 divergence_flag=0
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
289
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
290
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
291
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
292 if verbose == 1:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
293 print 'starting training'
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
294 while(minibatch_index*batch_size<nb_max_exemples):
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
295
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
296 for x, y in dataset.train(batch_size):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
297
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
298 #if we are using the classic learning rate deacay, adjust it before training of current mini-batch
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
299 if adaptive_lr==2:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
300 classifier.lr.value = tau*initial_lr/(tau+time_n)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
301
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
302
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
303 #train model
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
304 cost_ij = train_model(x,y)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
305
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
306 if (minibatch_index+1) % validation_frequency == 0:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
307 #save the current learning rate
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
308 learning_rate_list.append(classifier.lr.value)
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
309 divergence_flag_list.append(divergence_flag)
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
310
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
311 # compute the validation error
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
312 this_validation_loss = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
313 temp=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
314 for xv,yv in dataset.valid(1):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
315 # sum up the errors for each minibatch
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
316 this_validation_loss += test_model(xv,yv)
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
317 temp=temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
318 # get the average by dividing with the number of minibatches
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
319 this_validation_loss /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
320 #save the validation loss
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
321 total_validation_error_list.append(this_validation_loss)
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
322 if verbose == 1:
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
323 print(('epoch %i, minibatch %i, learning rate %f current validation error %f ') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
324 (epoch, minibatch_index+1,classifier.lr.value,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
325 this_validation_loss*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
326
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
327 # if we got the best validation score until now
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
328 if this_validation_loss < best_validation_loss:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
329 # save best validation score and iteration number
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
330 best_validation_loss = this_validation_loss
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
331 best_iter = minibatch_index
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
332 #reset divergence flag
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
333 divergence_flag=0
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
334
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
335 #save the best model. Overwrite the current saved best model so
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
336 #we only keep the best
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
337 numpy.savez('best_model.npy', config=configuration, W1=classifier.W1.value, W2=classifier.W2.value, b1=classifier.b1.value,\
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
338 b2=classifier.b2.value, minibatch_index=minibatch_index)
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
339
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
340 # test it on the test set
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
341 test_score = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
342 temp =0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
343 for xt,yt in dataset.test(batch_size):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
344 test_score += test_model(xt,yt)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
345 temp = temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
346 test_score /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
347 if verbose == 1:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
348 print(('epoch %i, minibatch %i, test error of best '
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
349 'model %f %%') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
350 (epoch, minibatch_index+1,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
351 test_score*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
352
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
353 # if the validation error is going up, we are overfitting (or oscillating)
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
354 # check if we are allowed to continue and if we will adjust the learning rate
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
355 elif this_validation_loss >= best_validation_loss:
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
356
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
357
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
358 # In non-classic learning rate decay, we modify the weight only when
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
359 # validation error is going up
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
360 if adaptive_lr==1:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
361 classifier.lr.value=classifier.lr.value*lr_t2_factor
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
362
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
363
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
364 #cap the patience so we are allowed to diverge max_div_count times
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
365 #if we are going up max_div_count in a row, we will stop immediatelty by modifying the patience
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
366 divergence_flag = divergence_flag +1
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
367
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
368
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
369 #calculate the test error at this point and exit
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
370 # test it on the test set
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
371 test_score = 0.
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
372 temp=0
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
373 for xt,yt in dataset.test(batch_size):
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
374 test_score += test_model(xt,yt)
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
375 temp=temp+1
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
376 test_score /= temp
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
377 if verbose == 1:
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
378 print ' validation error is going up, possibly stopping soon'
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
379 print((' epoch %i, minibatch %i, test error of best '
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
380 'model %f %%') %
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
381 (epoch, minibatch_index+1,
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
382 test_score*100.))
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
383
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
384
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
385
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
386 # check early stop condition
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
387 if divergence_flag==max_div_count:
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
388 minibatch_index=nb_max_exemples
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
389 print 'we have diverged, early stopping kicks in'
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
390 break
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
391
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
392 #check if we have seen enough exemples
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
393 #force one epoch at least
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
394 if epoch>0 and minibatch_index*batch_size>nb_max_exemples:
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
395 break
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
396
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
397
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
398 time_n= time_n + batch_size
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
399 minibatch_index = minibatch_index + 1
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
400
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
401 # we have finished looping through the training set
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
402 epoch = epoch+1
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
403 end_time = time.clock()
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
404 if verbose == 1:
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
405 print(('Optimization complete. Best validation score of %f %% '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
406 'obtained at iteration %i, with test performance %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
407 (best_validation_loss * 100., best_iter, test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
408 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
322
743907366476 code clean up in progress
xaviermuller
parents: 304
diff changeset
409 print minibatch_index
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
410
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
411 #save the model and the weights
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
412 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
413 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
414 learning_rate_list=learning_rate_list, divergence_flag_list=divergence_flag_list)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
415
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
416 return (best_training_error*100.0,best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
417
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
418
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
419 if __name__ == '__main__':
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
420 mlp_full_mnist()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
421
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
422 def jobman_mlp_full_nist(state,channel):
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
423 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
424 nb_max_exemples=state.nb_max_exemples,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
425 nb_hidden=state.nb_hidden,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
426 adaptive_lr=state.adaptive_lr,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
427 tau=state.tau,\
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
428 verbose = state.verbose,\
324
1763c64030d1 fixed bug in jobman interface
xaviermuller
parents: 323
diff changeset
429 lr_t2_factor=state.lr_t2_factor,
323
7a7615f940e8 finished code clean up and testing
xaviermuller
parents: 322
diff changeset
430 data_set=state.data_set)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
431 state.train_error=train_error
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
432 state.validation_error=validation_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
433 state.test_error=test_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
434 state.nb_exemples=nb_exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
435 state.time=time
304
1e4bf5a5b46d added type 2 adaptive learning configurable learning weight + versionning
xaviermuller
parents: 237
diff changeset
436 pylearn.version.record_versions(state,[theano,ift6266,pylearn])
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
437 return channel.COMPLETE
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
438
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
439