Mercurial > ift6266

annotate baseline/mlp/mlp_nist.py @ 212:e390b0454515

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added classic lr time decay and py code to calculate the error based on a saved model
author xaviermuller
date Wed, 10 Mar 2010 16:17:59 -0500
parents d37c944133c3
children 9b6e0af062af
rev   line source
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
1 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
3
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
9 the problem of MNIST digit classification.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
10
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
11 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
12
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
14
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
15 References:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
16
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
18 Christopher M. Bishop, section 5
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
19
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
21 to do lr first, then add regularization)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
22
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
23 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
25
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
26 import pdb
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
27 import numpy
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
28 import pylab
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
29 import theano
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
30 import theano.tensor as T
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
31 import time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
32 import theano.tensor.nnet
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
33 import pylearn
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
34 import theano,pylearn.version
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
35 from pylearn.io import filetensor as ft
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
36
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
37 data_path = '/data/lisa/data/nist/by_class/'
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
38
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
39 class MLP(object):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
40 """Multi-Layer Perceptron Class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
41
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
42 A multilayer perceptron is a feedforward artificial neural network model
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
43 that has one layer or more of hidden units and nonlinear activations.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
44 Intermidiate layers usually have as activation function thanh or the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
45 sigmoid function while the top layer is a softamx layer.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
46 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
47
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
48
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
49
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
50 def __init__(self, input, n_in, n_hidden, n_out,learning_rate):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
51 """Initialize the parameters for the multilayer perceptron
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
52
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
53 :param input: symbolic variable that describes the input of the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
54 architecture (one minibatch)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
55
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
56 :param n_in: number of input units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
57 which the datapoints lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
58
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
59 :param n_hidden: number of hidden units
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
60
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
61 :param n_out: number of output units, the dimension of the space in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
62 which the labels lie
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
63
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
64 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
65
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
66 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
67 # example contains only one hidden layer, but one can have as many
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
68 # layers as he/she wishes, making the network deeper. The only
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
69 # problem making the network deep this way is during learning,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
70 # backpropagation being unable to move the network from the starting
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
71 # point towards; this is where pre-training helps, giving a good
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
72 # starting point for backpropagation, but more about this in the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
73 # other tutorials
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
74
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
75 # `W1` is initialized with `W1_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
76 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
77 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
78 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
79 W1_values = numpy.asarray( numpy.random.uniform( \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
80 low = -numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
81 high = numpy.sqrt(6./(n_in+n_hidden)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
82 size = (n_in, n_hidden)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
83 # `W2` is initialized with `W2_values` which is uniformely sampled
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
84 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
85 # the output of uniform if converted using asarray to dtype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
86 # theano.config.floatX so that the code is runable on GPU
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
87 W2_values = numpy.asarray( numpy.random.uniform(
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
88 low = -numpy.sqrt(6./(n_hidden+n_out)), \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
89 high= numpy.sqrt(6./(n_hidden+n_out)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
90 size= (n_hidden, n_out)), dtype = theano.config.floatX)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
91
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
92 self.W1 = theano.shared( value = W1_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
93 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
94 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
95 self.W2 = theano.shared( value = W2_values )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
96 self.b2 = theano.shared( value = numpy.zeros((n_out,),
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
97 dtype= theano.config.floatX))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
98
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
99 #include the learning rate in the classifer so
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
100 #we can modify it on the fly when we want
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
101 lr_value=learning_rate
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
102 self.lr=theano.shared(value=lr_value)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
103 # symbolic expression computing the values of the hidden layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
104 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
105
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
106
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
107
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
108 # symbolic expression computing the values of the top layer
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
109 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
111 # compute prediction as class whose probability is maximal in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
112 # symbolic form
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
113 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
114 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
115
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
116
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
117
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
118
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
119 # L1 norm ; one regularization option is to enforce L1 norm to
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
120 # be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
121 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
122
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
123 # square of L2 norm ; one regularization option is to enforce
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
124 # square of L2 norm to be small
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
125 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
126
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
127
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
128
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
129 def negative_log_likelihood(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
130 """Return the mean of the negative log-likelihood of the prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
131 of this model under a given target distribution.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
132
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
133 .. math::
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
134
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
135 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
136 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
137 \ell (\theta=\{W,b\}, \mathcal{D})
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
138
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
139
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
140 :param y: corresponds to a vector that gives for each example the
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
141 :correct label
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
142 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
143 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
144
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
145
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
146
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
147
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
148 def errors(self, y):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
149 """Return a float representing the number of errors in the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
150 over the total number of examples of the minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
151 """
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
152
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
153 # check if y has same dimension of y_pred
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
154 if y.ndim != self.y_pred.ndim:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
155 raise TypeError('y should have the same shape as self.y_pred',
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
156 ('y', target.type, 'y_pred', self.y_pred.type))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
157 # check if y is of the correct datatype
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
158 if y.dtype.startswith('int'):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
159 # the T.neq operator returns a vector of 0s and 1s, where 1
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
160 # represents a mistake in prediction
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
161 return T.mean(T.neq(self.y_pred, y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
162 else:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
163 raise NotImplementedError()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
164
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
165
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
166 def mlp_full_nist( verbose = False,\
145
8ceaaf812891 changed adaptive lr flag from bool to int for jobman issues
XavierMuller
parents: 143
diff changeset
167 adaptive_lr = 0,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
168 train_data = 'all/all_train_data.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
169 train_labels = 'all/all_train_labels.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
170 test_data = 'all/all_test_data.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
171 test_labels = 'all/all_test_labels.ft',\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
172 learning_rate=0.01,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
173 L1_reg = 0.00,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
174 L2_reg = 0.0001,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
175 nb_max_exemples=1000000,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
176 batch_size=20,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
177 nb_hidden = 500,\
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
178 nb_targets = 62,
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
179 tau=1e6):
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
180
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
181
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
182 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
183
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
184 #save initial learning rate if classical adaptive lr is used
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
185 initial_lr=learning_rate
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
186
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
187 total_validation_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
188 total_train_error_list = []
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
189 learning_rate_list=[]
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
190 best_training_error=float('inf');
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
191
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
192
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
193
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
194
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
195 f = open(data_path+train_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
196 g= open(data_path+train_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
197 h = open(data_path+test_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
198 i= open(data_path+test_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
199
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
200 raw_train_data = ft.read(f)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
201 raw_train_labels = ft.read(g)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
202 raw_test_data = ft.read(h)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
203 raw_test_labels = ft.read(i)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
204
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
205 f.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
206 g.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
207 i.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
208 h.close()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
209 #create a validation set the same size as the test size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
210 #use the end of the training array for this purpose
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
211 #discard the last remaining so we get a %batch_size number
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
212 test_size=len(raw_test_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
213 test_size = int(test_size/batch_size)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
214 test_size*=batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
215 train_size = len(raw_train_data)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
216 train_size = int(train_size/batch_size)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
217 train_size*=batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
218 validation_size =test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
219 offset = train_size-test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
220 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
221 print 'train size = %d' %train_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
222 print 'test size = %d' %test_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
223 print 'valid size = %d' %validation_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
224 print 'offset = %d' %offset
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
225
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
226
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
227 train_set = (raw_train_data,raw_train_labels)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
228 train_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
229 for i in xrange(0, train_size-test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
230 train_batches = train_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
231 [(raw_train_data[i:i+batch_size], raw_train_labels[i:i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
232
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
233 test_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
234 for i in xrange(0, test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
235 test_batches = test_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
236 [(raw_test_data[i:i+batch_size], raw_test_labels[i:i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
237
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
238 validation_batches = []
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
239 for i in xrange(0, test_size, batch_size):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
240 validation_batches = validation_batches + \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
241 [(raw_train_data[offset+i:offset+i+batch_size], raw_train_labels[offset+i:offset+i+batch_size])]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
242
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
243
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
244 ishape = (32,32) # this is the size of NIST images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
245
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
246 # allocate symbolic variables for the data
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
247 x = T.fmatrix() # the data is presented as rasterized images
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
248 y = T.lvector() # the labels are presented as 1D vector of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
249 # [long int] labels
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
250
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
251 if verbose==True:
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
252 print 'finished parsing the data'
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
253 # construct the logistic regression class
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
254 classifier = MLP( input=x.reshape((batch_size,32*32)),\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
255 n_in=32*32,\
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
256 n_hidden=nb_hidden,\
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
257 n_out=nb_targets,
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
258 learning_rate=learning_rate)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
259
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
260
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
261
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
262
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
263 # the cost we minimize during training is the negative log likelihood of
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
264 # the model plus the regularization terms (L1 and L2); cost is expressed
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
265 # here symbolically
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
266 cost = classifier.negative_log_likelihood(y) \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
267 + L1_reg * classifier.L1 \
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
268 + L2_reg * classifier.L2_sqr
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
269
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
270 # compiling a theano function that computes the mistakes that are made by
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
271 # the model on a minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
272 test_model = theano.function([x,y], classifier.errors(y))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
273
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
274 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
275 g_W1 = T.grad(cost, classifier.W1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
276 g_b1 = T.grad(cost, classifier.b1)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
277 g_W2 = T.grad(cost, classifier.W2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
278 g_b2 = T.grad(cost, classifier.b2)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
279
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
280 # specify how to update the parameters of the model as a dictionary
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
281 updates = \
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
282 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
283 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
284 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
285 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
286
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
287 # compiling a theano function `train_model` that returns the cost, but in
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
288 # the same time updates the parameter of the model based on the rules
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
289 # defined in `updates`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
290 train_model = theano.function([x, y], cost, updates = updates )
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
291 n_minibatches = len(train_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
292
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
293
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
294
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
295
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
296
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
297
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
298 #conditions for stopping the adaptation:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
299 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
300 #2) validation error is going up twice in a row(probable overfitting)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
301
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
302 # This means we no longer stop on slow convergence as low learning rates stopped
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
303 # too fast.
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
304
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
305 # no longer relevant
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
306 patience =nb_max_exemples/batch_size
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
307 patience_increase = 2 # wait this much longer when a new best is
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
308 # found
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
309 improvement_threshold = 0.995 # a relative improvement of this much is
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
310 # considered significant
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
311 validation_frequency = n_minibatches/4
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
312
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
313
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
314
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
315
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
316 best_params = None
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
317 best_validation_loss = float('inf')
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
318 best_iter = 0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
319 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
320 start_time = time.clock()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
321 n_iter = nb_max_exemples/batch_size # nb of max times we are allowed to run through all exemples
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
322 n_iter = n_iter/n_minibatches + 1 #round up
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
323 n_iter=max(1,n_iter) # run at least once on short debug call
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
324 time_n=0 #in unit of exemples
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
325
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
326
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
327
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
328 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
329 print 'looping at most %d times through the data set' %n_iter
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
330 for iter in xrange(n_iter* n_minibatches):
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
331
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
332 # get epoch and minibatch index
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
333 epoch = iter / n_minibatches
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
334 minibatch_index = iter % n_minibatches
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
335
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
336
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
337 if adaptive_lr==2:
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
338 classifier.lr.value = tau*initial_lr/(tau+time_n)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
339
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
340
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
341 # get the minibatches corresponding to `iter` modulo
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
342 # `len(train_batches)`
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
343 x,y = train_batches[ minibatch_index ]
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
344 # convert to float
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
345 x_float = x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
346 cost_ij = train_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
347
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
348 if (iter+1) % validation_frequency == 0:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
349 # compute zero-one loss on validation set
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
350
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
351 this_validation_loss = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
352 for x,y in validation_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
353 # sum up the errors for each minibatch
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
354 x_float = x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
355 this_validation_loss += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
356 # get the average by dividing with the number of minibatches
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
357 this_validation_loss /= len(validation_batches)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
358 #save the validation loss
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
359 total_validation_error_list.append(this_validation_loss)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
360
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
361 #get the training error rate
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
362 this_train_loss=0
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
363 for x,y in train_batches:
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
364 # sum up the errors for each minibatch
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
365 x_float = x/255.0
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
366 this_train_loss += test_model(x_float,y)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
367 # get the average by dividing with the number of minibatches
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
368 this_train_loss /= len(train_batches)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
369 #save the validation loss
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
370 total_train_error_list.append(this_train_loss)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
371 if(this_train_loss<best_training_error):
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
372 best_training_error=this_train_loss
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
373
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
374 if verbose == True:
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
375 print('epoch %i, minibatch %i/%i, validation error %f, training error %f %%' % \
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
376 (epoch, minibatch_index+1, n_minibatches, \
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
377 this_validation_loss*100.,this_train_loss*100))
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
378 print 'learning rate = %f' %classifier.lr.value
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
379 print 'time = %i' %time_n
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
380
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
381
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
382 #save the learning rate
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
383 learning_rate_list.append(classifier.lr.value)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
384
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
385
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
386 # if we got the best validation score until now
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
387 if this_validation_loss < best_validation_loss:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
388 # save best validation score and iteration number
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
389 best_validation_loss = this_validation_loss
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
390 best_iter = iter
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
391 # reset patience if we are going down again
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
392 # so we continue exploring
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
393 patience=nb_max_exemples/batch_size
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
394 # test it on the test set
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
395 test_score = 0.
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
396 for x,y in test_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
397 x_float=x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
398 test_score += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
399 test_score /= len(test_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
400 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
401 print((' epoch %i, minibatch %i/%i, test error of best '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
402 'model %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
403 (epoch, minibatch_index+1, n_minibatches,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
404 test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
405
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
406 # if the validation error is going up, we are overfitting (or oscillating)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
407 # stop converging but run at least to next validation
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
408 # to check overfitting or ocsillation
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
409 # the saved weights of the model will be a bit off in that case
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
410 elif this_validation_loss >= best_validation_loss:
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
411 #calculate the test error at this point and exit
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
412 # test it on the test set
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
413 # however, if adaptive_lr is true, try reducing the lr to
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
414 # get us out of an oscilliation
145
8ceaaf812891 changed adaptive lr flag from bool to int for jobman issues
XavierMuller
parents: 143
diff changeset
415 if adaptive_lr==1:
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
416 classifier.lr.value=classifier.lr.value/2.0
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
417
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
418 test_score = 0.
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
419 #cap the patience so we are allowed one more validation error
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
420 #calculation before aborting
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
421 patience = iter+validation_frequency+1
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
422 for x,y in test_batches:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
423 x_float=x/255.0
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
424 test_score += test_model(x_float,y)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
425 test_score /= len(test_batches)
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
426 if verbose == True:
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
427 print ' validation error is going up, possibly stopping soon'
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
428 print((' epoch %i, minibatch %i/%i, test error of best '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
429 'model %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
430 (epoch, minibatch_index+1, n_minibatches,
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
431 test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
432
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
433
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
434
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
435
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
436 if iter>patience:
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
437 print 'we have diverged'
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
438 break
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
439
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
440
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
441 time_n= time_n + batch_size
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
442 end_time = time.clock()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
443 if verbose == True:
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
444 print(('Optimization complete. Best validation score of %f %% '
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
445 'obtained at iteration %i, with test performance %f %%') %
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
446 (best_validation_loss * 100., best_iter, test_score*100.))
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
447 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
448 print iter
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
449
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
450 #save the model and the weights
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
451 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
452 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
453 learning_rate_list=learning_rate_list)
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
454
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
455 return (best_training_error*100.0,best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
456
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
457
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
458 if __name__ == '__main__':
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
459 mlp_full_mnist()
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
460
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
461 def jobman_mlp_full_nist(state,channel):
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
462 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
463 nb_max_exemples=state.nb_max_exemples,\
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
464 nb_hidden=state.nb_hidden,\
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
465 adaptive_lr=state.adaptive_lr,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents: 169
diff changeset
466 tau=tau)
143
f341a4efb44a added adaptive lr, weight file save, traine error and error curves
XavierMuller
parents: 110
diff changeset
467 state.train_error=train_error
110
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
468 state.validation_error=validation_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
469 state.test_error=test_error
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
470 state.nb_exemples=nb_exemples
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
471 state.time=time
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
472 return channel.COMPLETE
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
473
93b4b84d86cf added simple mlp file
XavierMuller
parents:
diff changeset
474