Mercurial > ift6266

annotate baseline/mlp/v_youssouf/mlp_nist.py @ 413:f2dd75248483

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
initial commit of mlp with options for detection and 36 classes
author youssouf
date Thu, 29 Apr 2010 16:51:03 -0400
parents
children
rev   line source
413
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
1 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
3
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
9 the problem of MNIST digit classification.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
10
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
11 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
12
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
14
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
15 References:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
16
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
18 Christopher M. Bishop, section 5
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
19
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
21 to do lr first, then add regularization)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
22
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
23 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
25
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
26 import pdb
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
27 import numpy
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
28 import pylab
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
29 import theano
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
30 import theano.tensor as T
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
31 import time
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
32 import theano.tensor.nnet
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
33 import pylearn
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
34 import theano,pylearn.version,ift6266
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
35 from pylearn.io import filetensor as ft
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
36 from ift6266 import datasets
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
37
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
38 data_path = '/data/lisa/data/nist/by_class/'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
39
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
40 class MLP(object):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
41 """Multi-Layer Perceptron Class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
42
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
43 A multilayer perceptron is a feedforward artificial neural network model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
44 that has one layer or more of hidden units and nonlinear activations.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
45 Intermidiate layers usually have as activation function thanh or the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
46 sigmoid function while the top layer is a softamx layer.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
47 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
48
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
49
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
50
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
51 def __init__(self, input, n_in, n_hidden, n_out,learning_rate, detection_mode=0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
52 """Initialize the parameters for the multilayer perceptron
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
53
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
54 :param input: symbolic variable that describes the input of the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
55 architecture (one minibatch)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
56
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
57 :param n_in: number of input units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
58 which the datapoints lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
59
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
60 :param n_hidden: number of hidden units
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
61
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
62 :param n_out: number of output units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
63 which the labels lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
64
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
65 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
66
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
67 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
68 # example contains only one hidden layer, but one can have as many
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
69 # layers as he/she wishes, making the network deeper. The only
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
70 # problem making the network deep this way is during learning,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
71 # backpropagation being unable to move the network from the starting
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
72 # point towards; this is where pre-training helps, giving a good
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
73 # starting point for backpropagation, but more about this in the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
74 # other tutorials
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
75
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
76 # `W1` is initialized with `W1_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
77 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
78 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
79 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
80 W1_values = numpy.asarray( numpy.random.uniform( \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
81 low = -numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
82 high = numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
83 size = (n_in, n_hidden)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
84 # `W2` is initialized with `W2_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
85 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
86 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
87 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
88 W2_values = numpy.asarray( numpy.random.uniform(
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
89 low = -numpy.sqrt(6./(n_hidden+n_out)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
90 high= numpy.sqrt(6./(n_hidden+n_out)),\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
91 size= (n_hidden, n_out)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
92
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
93 self.W1 = theano.shared( value = W1_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
94 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
95 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
96 self.W2 = theano.shared( value = W2_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
97 self.b2 = theano.shared( value = numpy.zeros((n_out,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
98 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
99
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
100 #include the learning rate in the classifer so
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
101 #we can modify it on the fly when we want
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
102 lr_value=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
103 self.lr=theano.shared(value=lr_value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
104 # symbolic expression computing the values of the hidden layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
105 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
106
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
107
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
108
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
109 # symbolic expression computing the values of the top layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
110 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
111 self.p_y_given_x= T.nnet.sigmoid(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
112 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
113 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
114
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
115 # compute prediction as class whose probability is maximal in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
116 # symbolic form
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
117 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
118 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
119
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
120
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
121
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
122
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
123 # L1 norm ; one regularization option is to enforce L1 norm to
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
124 # be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
125 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
126
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
127 # square of L2 norm ; one regularization option is to enforce
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
128 # square of L2 norm to be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
129 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
130
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
131
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
132
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
133 def negative_log_likelihood(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
134 """Return the mean of the negative log-likelihood of the prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
135 of this model under a given target distribution.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
136
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
137 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
138
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
139 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
140 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
141 \ell (\theta=\{W,b\}, \mathcal{D})
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
142
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
143
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
144 :param y: corresponds to a vector that gives for each example the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
145 :correct label
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
146 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
147 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
148
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
149
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
150 def cross_entropy(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
151 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y]+T.sum(T.log(1-self.p_y_given_x), axis=1)-T.log(1-self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
152
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
153 def errors(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
154 """Return a float representing the number of errors in the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
155 over the total number of examples of the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
156 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
157
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
158 # check if y has same dimension of y_pred
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
159 if y.ndim != self.y_pred.ndim:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
160 raise TypeError('y should have the same shape as self.y_pred',
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
161 ('y', target.type, 'y_pred', self.y_pred.type))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
162 # check if y is of the correct datatype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
163 if y.dtype.startswith('int'):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
164 # the T.neq operator returns a vector of 0s and 1s, where 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
165 # represents a mistake in prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
166 return T.mean(T.neq(self.y_pred, y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
167 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
168 raise NotImplementedError()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
169
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
170
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
171 def mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
172 adaptive_lr = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
173 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
174 learning_rate=0.01,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
175 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
176 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
177 nb_max_exemples=1000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
178 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
179 nb_hidden = 30,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
180 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
181 tau=1e6,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
182 lr_t2_factor=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
183 detection_mode = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
184 reduce_label = 0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
185
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
186
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
187 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr, detection_mode, reduce_label]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
188
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
189 if(verbose):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
190 print(('verbose: %i') % (verbose))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
191 print(('adaptive_lr: %i') % (adaptive_lr))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
192 print(('data_set: %i') % (data_set))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
193 print(('learning_rate: %f') % (learning_rate))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
194 print(('L1_reg: %f') % (L1_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
195 print(('L2_reg: %f') % (L2_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
196 print(('nb_max_exemples: %i') % (nb_max_exemples))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
197 print(('batch_size: %i') % (batch_size))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
198 print(('nb_hidden: %i') % (nb_hidden))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
199 print(('nb_targets: %f') % (nb_targets))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
200 print(('tau: %f') % (tau))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
201 print(('lr_t2_factor: %f') % (lr_t2_factor))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
202 print(('detection_mode: %i') % (detection_mode))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
203 print(('reduce_label: %i') % (reduce_label))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
204
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
205 # define the number of output - reduce_label : merge the lower and upper case. i.e a and A will both have label 10
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
206 if(reduce_label):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
207 nb_targets = 36
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
208 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
209 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
210
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
211 #save initial learning rate if classical adaptive lr is used
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
212 initial_lr=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
213
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
214 total_validation_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
215 total_train_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
216 learning_rate_list=[]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
217 best_training_error=float('inf');
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
218
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
219 if data_set==0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
220 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
221
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
222
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
223
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
224
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
225 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
226
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
227 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
228 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
229 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
230 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
231
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
232
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
233 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
234 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
235 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
236 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
237 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
238 learning_rate=learning_rate,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
239 detection_mode = detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
240
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
241
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
242
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
243
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
244 # the cost we minimize during training is the negative log likelihood of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
245 # the model plus the regularization terms (L1 and L2); cost is expressed
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
246 # here symbolically
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
247 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
248 cost = classifier.cross_entropy(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
249 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
250 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
251 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
252 cost = classifier.negative_log_likelihood(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
253 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
254 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
255
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
256 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
257 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
258 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
259
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
260 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
261 g_W1 = T.grad(cost, classifier.W1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
262 g_b1 = T.grad(cost, classifier.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
263 g_W2 = T.grad(cost, classifier.W2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
264 g_b2 = T.grad(cost, classifier.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
265
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
266 # specify how to update the parameters of the model as a dictionary
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
267 updates = \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
268 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
269 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
270 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
271 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
272
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
273 # compiling a theano function `train_model` that returns the cost, but in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
274 # the same time updates the parameter of the model based on the rules
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
275 # defined in `updates`
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
276 train_model = theano.function([x, y], cost, updates = updates )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
277
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
278
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
279
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
280
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
281
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
282
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
283
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
284
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
285
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
286 #conditions for stopping the adaptation:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
287 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
288 #2) validation error is going up twice in a row(probable overfitting)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
289
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
290 # This means we no longer stop on slow convergence as low learning rates stopped
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
291 # too fast.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
292
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
293 #approximate number of samples in the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
294 #this is just to have a validation frequency
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
295 #roughly proportionnal to the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
296 n_minibatches = 650000/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
297
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
298
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
299 patience =nb_max_exemples/batch_size #in units of minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
300 patience_increase = 2 # wait this much longer when a new best is
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
301 # found
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
302 improvement_threshold = 0.995 # a relative improvement of this much is
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
303 # considered significant
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
304 validation_frequency = n_minibatches/4
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
305
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
306
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
307
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
308
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
309
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
310 best_validation_loss = float('inf')
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
311 best_iter = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
312 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
313 start_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
314 time_n=0 #in unit of exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
315 minibatch_index=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
316 epoch=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
317 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
318
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
319
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
320
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
321 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
322 print 'looking at most at %i exemples' %nb_max_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
323 while(minibatch_index*batch_size<nb_max_exemples):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
324
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
325 for x, y in dataset.train(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
326
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
327 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
328 y[y > 35] = y[y > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
329 minibatch_index = minibatch_index + 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
330 if adaptive_lr==2:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
331 classifier.lr.value = tau*initial_lr/(tau+time_n)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
332
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
333
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
334 #train model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
335 cost_ij = train_model(x,y)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
336
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
337 if (minibatch_index+1) % validation_frequency == 0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
338
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
339 #save the current learning rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
340 learning_rate_list.append(classifier.lr.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
341
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
342 # compute the validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
343 this_validation_loss = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
344 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
345 for xv,yv in dataset.valid(1):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
346 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
347 yv[yv > 35] = yv[yv > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
348 # sum up the errors for each minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
349 axxa=test_model(xv,yv)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
350 this_validation_loss += axxa
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
351 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
352 # get the average by dividing with the number of minibatches
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
353 this_validation_loss /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
354 #save the validation loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
355 total_validation_error_list.append(this_validation_loss)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
356 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
357 print(('epoch %i, minibatch %i, learning rate %f current validation error %f ') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
358 (epoch, minibatch_index+1,classifier.lr.value,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
359 this_validation_loss*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
360
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
361 # if we got the best validation score until now
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
362 if this_validation_loss < best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
363 # save best validation score and iteration number
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
364 best_validation_loss = this_validation_loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
365 best_iter = minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
366 # reset patience if we are going down again
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
367 # so we continue exploring
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
368 patience=nb_max_exemples/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
369 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
370 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
371 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
372 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
373 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
374 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
375 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
376 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
377 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
378 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
379 print(('epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
380 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
381 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
382 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
383
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
384 # if the validation error is going up, we are overfitting (or oscillating)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
385 # stop converging but run at least to next validation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
386 # to check overfitting or ocsillation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
387 # the saved weights of the model will be a bit off in that case
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
388 elif this_validation_loss >= best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
389 #calculate the test error at this point and exit
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
390 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
391 # however, if adaptive_lr is true, try reducing the lr to
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
392 # get us out of an oscilliation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
393 if adaptive_lr==1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
394 classifier.lr.value=classifier.lr.value*lr_t2_factor
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
395
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
396 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
397 #cap the patience so we are allowed one more validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
398 #calculation before aborting
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
399 patience = minibatch_index+validation_frequency+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
400 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
401 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
402 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
403 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
404
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
405 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
406 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
407 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
408 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
409 print ' validation error is going up, possibly stopping soon'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
410 print((' epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
411 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
412 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
413 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
414
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
415
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
416
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
417
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
418 if minibatch_index>patience:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
419 print 'we have diverged'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
420 break
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
421
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
422
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
423 time_n= time_n + batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
424 epoch = epoch+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
425 end_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
426 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
427 print(('Optimization complete. Best validation score of %f %% '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
428 'obtained at iteration %i, with test performance %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
429 (best_validation_loss * 100., best_iter, test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
430 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
431 print minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
432
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
433 #save the model and the weights
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
434 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
435 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
436 learning_rate_list=learning_rate_list)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
437
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
438 return (best_training_error*100.0,best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
439
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
440 def test_error(model_file):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
441
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
442 print((' test error on all NIST'))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
443 # load the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
444 a=numpy.load(model_file)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
445 W1=a['W1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
446 W2=a['W2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
447 b1=a['b1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
448 b2=a['b2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
449 configuration=a['config']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
450 #configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
451 learning_rate = configuration[0]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
452 nb_max_exemples = configuration[1]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
453 nb_hidden = configuration[2]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
454 adaptive_lr = configuration[3]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
455
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
456 if(len(configuration) == 6):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
457 detection_mode = configuration[4]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
458 reduce_label = configuration[5]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
459 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
460 detection_mode = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
461 reduce_label = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
462
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
463 # define the batch size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
464 batch_size=20
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
465 #define the nb of target
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
466 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
467
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
468 # create the mlp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
469 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
470
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
471 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
472 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
473 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
474 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
475
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
476
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
477 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
478 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
479 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
480 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
481 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
482 learning_rate=learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
483 detection_mode=detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
484
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
485
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
486 # set the weight into the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
487 classifier.W1.value = W1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
488 classifier.b1.value = b1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
489 classifier.W2.value = W2
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
490 classifier.b2.value = b2
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
491
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
492
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
493 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
494 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
495 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
496
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
497 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
498
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
499 # load NIST ALL
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
500 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
501 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
502 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
503 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
504 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
505 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
506 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
507 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
508 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
509
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
510 print(( ' test error NIST ALL : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
511
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
512 # load NIST DIGITS
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
513 dataset=datasets.nist_digits()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
514 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
515 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
516 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
517 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
518 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
519 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
520 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
521 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
522
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
523 print(( ' test error NIST digits : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
524
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
525 # load NIST lower
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
526 dataset=datasets.nist_lower()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
527 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
528 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
529 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
530 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
531 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
532 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
533 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
534 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
535
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
536 print(( ' test error NIST lower : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
537
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
538 # load NIST upper
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
539 dataset=datasets.nist_upper()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
540 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
541 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
542 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
543 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
544 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
545 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
546 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
547 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
548
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
549 print(( ' test error NIST upper : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
550
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
551
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
552 if __name__ == '__main__':
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
553 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
554 mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
555 adaptive_lr = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
556 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
557 learning_rate=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
558 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
559 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
560 nb_max_exemples=10000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
561 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
562 nb_hidden = 500,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
563 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
564 tau=100000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
565 lr_t2_factor=0.5)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
566 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
567
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
568 test_error('model.npy.npz')
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
569
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
570 def jobman_mlp_full_nist(state,channel):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
571 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
572 nb_max_exemples=state.nb_max_exemples,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
573 nb_hidden=state.nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
574 adaptive_lr=state.adaptive_lr,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
575 tau=state.tau,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
576 verbose = state.verbose,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
577 lr_t2_factor=state.lr_t2_factor,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
578 detection_mode = state.detection_mode,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
579 reduce_label = state.reduce_label)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
580 state.train_error=train_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
581 state.validation_error=validation_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
582 state.test_error=test_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
583 state.nb_exemples=nb_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
584 state.time=time
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
585 pylearn.version.record_versions(state,[theano,ift6266,pylearn])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
586 return channel.COMPLETE
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
587
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
588