Mercurial > ift6266

annotate baseline/mlp/v_youssouf/mlp_nist.py @ 538:f0ee2212ea7c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
typos and stuff
author Dumitru Erhan <dumitru.erhan@gmail.com>
date Tue, 01 Jun 2010 19:34:00 -0700
parents f2dd75248483
children
rev   line source
413
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
1 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
2 This tutorial introduces the multilayer perceptron using Theano.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
3
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
4 A multilayer perceptron is a logistic regressor where
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
5 instead of feeding the input to the logistic regression you insert a
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
6 intermidiate layer, called the hidden layer, that has a nonlinear
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
7 activation function (usually tanh or sigmoid) . One can use many such
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
8 hidden layers making the architecture deep. The tutorial will also tackle
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
9 the problem of MNIST digit classification.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
10
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
11 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
12
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
14
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
15 References:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
16
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
17 - textbooks: "Pattern Recognition and Machine Learning" -
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
18 Christopher M. Bishop, section 5
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
19
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
21 to do lr first, then add regularization)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
22
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
23 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
24 __docformat__ = 'restructedtext en'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
25
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
26 import pdb
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
27 import numpy
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
28 import pylab
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
29 import theano
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
30 import theano.tensor as T
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
31 import time
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
32 import theano.tensor.nnet
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
33 import pylearn
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
34 import theano,pylearn.version,ift6266
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
35 from pylearn.io import filetensor as ft
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
36 from ift6266 import datasets
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
37
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
38 data_path = '/data/lisa/data/nist/by_class/'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
39
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
40 class MLP(object):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
41 """Multi-Layer Perceptron Class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
42
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
43 A multilayer perceptron is a feedforward artificial neural network model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
44 that has one layer or more of hidden units and nonlinear activations.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
45 Intermidiate layers usually have as activation function thanh or the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
46 sigmoid function while the top layer is a softamx layer.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
47 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
48
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
49
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
50
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
51 def __init__(self, input, n_in, n_hidden, n_out,learning_rate, detection_mode=0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
52 """Initialize the parameters for the multilayer perceptron
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
53
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
54 :param input: symbolic variable that describes the input of the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
55 architecture (one minibatch)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
56
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
57 :param n_in: number of input units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
58 which the datapoints lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
59
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
60 :param n_hidden: number of hidden units
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
61
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
62 :param n_out: number of output units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
63 which the labels lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
64
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
65 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
66
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
67 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
68 # example contains only one hidden layer, but one can have as many
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
69 # layers as he/she wishes, making the network deeper. The only
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
70 # problem making the network deep this way is during learning,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
71 # backpropagation being unable to move the network from the starting
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
72 # point towards; this is where pre-training helps, giving a good
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
73 # starting point for backpropagation, but more about this in the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
74 # other tutorials
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
75
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
76 # `W1` is initialized with `W1_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
77 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
78 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
79 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
80 W1_values = numpy.asarray( numpy.random.uniform( \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
81 low = -numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
82 high = numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
83 size = (n_in, n_hidden)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
84 # `W2` is initialized with `W2_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
85 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
86 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
87 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
88 W2_values = numpy.asarray( numpy.random.uniform(
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
89 low = -numpy.sqrt(6./(n_hidden+n_out)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
90 high= numpy.sqrt(6./(n_hidden+n_out)),\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
91 size= (n_hidden, n_out)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
92
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
93 self.W1 = theano.shared( value = W1_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
94 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
95 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
96 self.W2 = theano.shared( value = W2_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
97 self.b2 = theano.shared( value = numpy.zeros((n_out,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
98 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
99
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
100 #include the learning rate in the classifer so
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
101 #we can modify it on the fly when we want
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
102 lr_value=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
103 self.lr=theano.shared(value=lr_value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
104 # symbolic expression computing the values of the hidden layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
105 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
106
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
107
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
108
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
109 # symbolic expression computing the values of the top layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
110 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
111 self.p_y_given_x= T.nnet.sigmoid(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
112 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
113 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
114
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
115 # compute prediction as class whose probability is maximal in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
116 # symbolic form
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
117 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
118 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
119
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
120
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
121
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
122
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
123 # L1 norm ; one regularization option is to enforce L1 norm to
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
124 # be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
125 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
126
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
127 # square of L2 norm ; one regularization option is to enforce
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
128 # square of L2 norm to be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
129 self.L2_sqr = (self.W1**2).sum() + (self.W2**2).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
130
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
131
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
132
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
133 def negative_log_likelihood(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
134 """Return the mean of the negative log-likelihood of the prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
135 of this model under a given target distribution.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
136
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
137 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
138
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
139 \frac{1}{|\mathcal{D}|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
140 \frac{1}{|\mathcal{D}|}\sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
141 \ell (\theta=\{W,b\}, \mathcal{D})
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
142
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
143
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
144 :param y: corresponds to a vector that gives for each example the
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
145 :correct label
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
146 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
147 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
148
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
149
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
150 def cross_entropy(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
151 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y]+T.sum(T.log(1-self.p_y_given_x), axis=1)-T.log(1-self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
152
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
153 def errors(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
154 """Return a float representing the number of errors in the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
155 over the total number of examples of the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
156 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
157
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
158 # check if y has same dimension of y_pred
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
159 if y.ndim != self.y_pred.ndim:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
160 raise TypeError('y should have the same shape as self.y_pred',
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
161 ('y', target.type, 'y_pred', self.y_pred.type))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
162 # check if y is of the correct datatype
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
163 if y.dtype.startswith('int'):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
164 # the T.neq operator returns a vector of 0s and 1s, where 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
165 # represents a mistake in prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
166 return T.mean(T.neq(self.y_pred, y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
167 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
168 raise NotImplementedError()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
169
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
170
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
171 def mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
172 adaptive_lr = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
173 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
174 learning_rate=0.01,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
175 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
176 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
177 nb_max_exemples=1000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
178 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
179 nb_hidden = 30,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
180 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
181 tau=1e6,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
182 lr_t2_factor=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
183 detection_mode = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
184 reduce_label = 0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
185
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
186
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
187 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr, detection_mode, reduce_label]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
188
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
189 if(verbose):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
190 print(('verbose: %i') % (verbose))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
191 print(('adaptive_lr: %i') % (adaptive_lr))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
192 print(('data_set: %i') % (data_set))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
193 print(('learning_rate: %f') % (learning_rate))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
194 print(('L1_reg: %f') % (L1_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
195 print(('L2_reg: %f') % (L2_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
196 print(('nb_max_exemples: %i') % (nb_max_exemples))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
197 print(('batch_size: %i') % (batch_size))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
198 print(('nb_hidden: %i') % (nb_hidden))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
199 print(('nb_targets: %f') % (nb_targets))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
200 print(('tau: %f') % (tau))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
201 print(('lr_t2_factor: %f') % (lr_t2_factor))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
202 print(('detection_mode: %i') % (detection_mode))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
203 print(('reduce_label: %i') % (reduce_label))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
204
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
205 # define the number of output - reduce_label : merge the lower and upper case. i.e a and A will both have label 10
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
206 if(reduce_label):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
207 nb_targets = 36
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
208 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
209 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
210
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
211 #save initial learning rate if classical adaptive lr is used
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
212 initial_lr=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
213
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
214 total_validation_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
215 total_train_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
216 learning_rate_list=[]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
217 best_training_error=float('inf');
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
218
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
219 if data_set==0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
220 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
221
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
222
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
223
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
224
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
225 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
226
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
227 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
228 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
229 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
230 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
231
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
232
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
233 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
234 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
235 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
236 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
237 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
238 learning_rate=learning_rate,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
239 detection_mode = detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
240
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
241
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
242
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
243
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
244 # the cost we minimize during training is the negative log likelihood of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
245 # the model plus the regularization terms (L1 and L2); cost is expressed
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
246 # here symbolically
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
247 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
248 cost = classifier.cross_entropy(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
249 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
250 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
251 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
252 cost = classifier.negative_log_likelihood(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
253 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
254 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
255
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
256 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
257 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
258 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
259
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
260 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
261 g_W1 = T.grad(cost, classifier.W1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
262 g_b1 = T.grad(cost, classifier.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
263 g_W2 = T.grad(cost, classifier.W2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
264 g_b2 = T.grad(cost, classifier.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
265
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
266 # specify how to update the parameters of the model as a dictionary
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
267 updates = \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
268 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
269 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
270 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
271 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
272
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
273 # compiling a theano function `train_model` that returns the cost, but in
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
274 # the same time updates the parameter of the model based on the rules
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
275 # defined in `updates`
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
276 train_model = theano.function([x, y], cost, updates = updates )
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
277
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
278
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
279
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
280
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
281
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
282
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
283
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
284
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
285
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
286 #conditions for stopping the adaptation:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
287 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
288 #2) validation error is going up twice in a row(probable overfitting)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
289
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
290 # This means we no longer stop on slow convergence as low learning rates stopped
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
291 # too fast.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
292
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
293 #approximate number of samples in the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
294 #this is just to have a validation frequency
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
295 #roughly proportionnal to the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
296 n_minibatches = 650000/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
297
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
298
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
299 patience =nb_max_exemples/batch_size #in units of minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
300 patience_increase = 2 # wait this much longer when a new best is
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
301 # found
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
302 improvement_threshold = 0.995 # a relative improvement of this much is
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
303 # considered significant
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
304 validation_frequency = n_minibatches/4
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
305
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
306
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
307
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
308
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
309
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
310 best_validation_loss = float('inf')
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
311 best_iter = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
312 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
313 start_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
314 time_n=0 #in unit of exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
315 minibatch_index=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
316 epoch=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
317 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
318
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
319
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
320
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
321 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
322 print 'looking at most at %i exemples' %nb_max_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
323 while(minibatch_index*batch_size<nb_max_exemples):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
324
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
325 for x, y in dataset.train(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
326
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
327 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
328 y[y > 35] = y[y > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
329 minibatch_index = minibatch_index + 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
330 if adaptive_lr==2:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
331 classifier.lr.value = tau*initial_lr/(tau+time_n)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
332
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
333
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
334 #train model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
335 cost_ij = train_model(x,y)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
336
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
337 if (minibatch_index+1) % validation_frequency == 0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
338
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
339 #save the current learning rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
340 learning_rate_list.append(classifier.lr.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
341
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
342 # compute the validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
343 this_validation_loss = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
344 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
345 for xv,yv in dataset.valid(1):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
346 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
347 yv[yv > 35] = yv[yv > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
348 # sum up the errors for each minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
349 axxa=test_model(xv,yv)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
350 this_validation_loss += axxa
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
351 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
352 # get the average by dividing with the number of minibatches
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
353 this_validation_loss /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
354 #save the validation loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
355 total_validation_error_list.append(this_validation_loss)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
356 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
357 print(('epoch %i, minibatch %i, learning rate %f current validation error %f ') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
358 (epoch, minibatch_index+1,classifier.lr.value,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
359 this_validation_loss*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
360
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
361 # if we got the best validation score until now
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
362 if this_validation_loss < best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
363 # save best validation score and iteration number
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
364 best_validation_loss = this_validation_loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
365 best_iter = minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
366 # reset patience if we are going down again
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
367 # so we continue exploring
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
368 patience=nb_max_exemples/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
369 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
370 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
371 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
372 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
373 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
374 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
375 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
376 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
377 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
378 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
379 print(('epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
380 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
381 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
382 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
383
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
384 # if the validation error is going up, we are overfitting (or oscillating)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
385 # stop converging but run at least to next validation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
386 # to check overfitting or ocsillation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
387 # the saved weights of the model will be a bit off in that case
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
388 elif this_validation_loss >= best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
389 #calculate the test error at this point and exit
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
390 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
391 # however, if adaptive_lr is true, try reducing the lr to
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
392 # get us out of an oscilliation
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
393 if adaptive_lr==1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
394 classifier.lr.value=classifier.lr.value*lr_t2_factor
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
395
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
396 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
397 #cap the patience so we are allowed one more validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
398 #calculation before aborting
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
399 patience = minibatch_index+validation_frequency+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
400 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
401 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
402 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
403 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
404
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
405 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
406 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
407 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
408 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
409 print ' validation error is going up, possibly stopping soon'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
410 print((' epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
411 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
412 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
413 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
414
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
415
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
416
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
417
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
418 if minibatch_index>patience:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
419 print 'we have diverged'
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
420 break
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
421
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
422
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
423 time_n= time_n + batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
424 epoch = epoch+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
425 end_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
426 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
427 print(('Optimization complete. Best validation score of %f %% '
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
428 'obtained at iteration %i, with test performance %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
429 (best_validation_loss * 100., best_iter, test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
430 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
431 print minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
432
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
433 #save the model and the weights
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
434 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
435 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
436 learning_rate_list=learning_rate_list)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
437
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
438 return (best_training_error*100.0,best_validation_loss * 100.,test_score*100.,best_iter*batch_size,(end_time-start_time)/60)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
439
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
440 def test_error(model_file):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
441
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
442 print((' test error on all NIST'))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
443 # load the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
444 a=numpy.load(model_file)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
445 W1=a['W1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
446 W2=a['W2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
447 b1=a['b1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
448 b2=a['b2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
449 configuration=a['config']
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
450 #configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
451 learning_rate = configuration[0]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
452 nb_max_exemples = configuration[1]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
453 nb_hidden = configuration[2]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
454 adaptive_lr = configuration[3]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
455
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
456 if(len(configuration) == 6):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
457 detection_mode = configuration[4]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
458 reduce_label = configuration[5]
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
459 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
460 detection_mode = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
461 reduce_label = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
462
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
463 # define the batch size
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
464 batch_size=20
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
465 #define the nb of target
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
466 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
467
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
468 # create the mlp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
469 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
470
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
471 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
472 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
473 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
474 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
475
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
476
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
477 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
478 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
479 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
480 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
481 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
482 learning_rate=learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
483 detection_mode=detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
484
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
485
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
486 # set the weight into the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
487 classifier.W1.value = W1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
488 classifier.b1.value = b1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
489 classifier.W2.value = W2
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
490 classifier.b2.value = b2
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
491
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
492
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
493 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
494 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
495 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
496
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
497 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
498
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
499 # load NIST ALL
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
500 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
501 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
502 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
503 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
504 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
505 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
506 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
507 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
508 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
509
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
510 print(( ' test error NIST ALL : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
511
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
512 # load NIST DIGITS
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
513 dataset=datasets.nist_digits()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
514 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
515 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
516 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
517 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
518 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
519 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
520 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
521 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
522
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
523 print(( ' test error NIST digits : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
524
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
525 # load NIST lower
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
526 dataset=datasets.nist_lower()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
527 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
528 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
529 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
530 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
531 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
532 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
533 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
534 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
535
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
536 print(( ' test error NIST lower : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
537
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
538 # load NIST upper
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
539 dataset=datasets.nist_upper()
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
540 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
541 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
542 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
543 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
544 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
545 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
546 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
547 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
548
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
549 print(( ' test error NIST upper : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
550
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
551
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
552 if __name__ == '__main__':
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
553 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
554 mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
555 adaptive_lr = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
556 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
557 learning_rate=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
558 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
559 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
560 nb_max_exemples=10000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
561 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
562 nb_hidden = 500,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
563 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
564 tau=100000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
565 lr_t2_factor=0.5)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
566 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
567
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
568 test_error('model.npy.npz')
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
569
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
570 def jobman_mlp_full_nist(state,channel):
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
571 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
572 nb_max_exemples=state.nb_max_exemples,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
573 nb_hidden=state.nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
574 adaptive_lr=state.adaptive_lr,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
575 tau=state.tau,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
576 verbose = state.verbose,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
577 lr_t2_factor=state.lr_t2_factor,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
578 detection_mode = state.detection_mode,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
579 reduce_label = state.reduce_label)
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
580 state.train_error=train_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
581 state.validation_error=validation_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
582 state.test_error=test_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
583 state.nb_exemples=nb_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
584 state.time=time
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
585 pylearn.version.record_versions(state,[theano,ift6266,pylearn])
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
586 return channel.COMPLETE
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
587
f2dd75248483 initial commit of mlp with options for detection and 36 classes
youssouf
parents:
diff changeset
588