ift6266: baseline/mlp/v_youssouf/mlp

annotate baseline/mlp/v_youssouf/mlp_nist.py @ 644:e63d23c7c9fb

reviews aistats finales

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Thu, 24 Mar 2011 17:05:05 -0400
parents	f2dd75248483
children

rev	line source
413 f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	1 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	2 This tutorial introduces the multilayer perceptron using Theano.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	3
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	4 A multilayer perceptron is a logistic regressor where
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	5 instead of feeding the input to the logistic regression you insert a
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	6 intermidiate layer, called the hidden layer, that has a nonlinear
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	7 activation function (usually tanh or sigmoid) . One can use many such
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	8 hidden layers making the architecture deep. The tutorial will also tackle
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	9 the problem of MNIST digit classification.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	10
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	11 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	12
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	13 f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))),
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	14
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	15 References:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	16
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	17 - textbooks: "Pattern Recognition and Machine Learning" -
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	18 Christopher M. Bishop, section 5
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	19
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	20 TODO: recommended preprocessing, lr ranges, regularization ranges (explain
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	21 to do lr first, then add regularization)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	22
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	23 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	24 __docformat__ = 'restructedtext en'
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	25
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	26 import pdb
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	27 import numpy
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	28 import pylab
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	29 import theano
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	30 import theano.tensor as T
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	31 import time
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	32 import theano.tensor.nnet
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	33 import pylearn
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	34 import theano,pylearn.version,ift6266
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	35 from pylearn.io import filetensor as ft
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	36 from ift6266 import datasets
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	37
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	38 data_path = '/data/lisa/data/nist/by_class/'
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	39
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	40 class MLP(object):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	41 """Multi-Layer Perceptron Class
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	42
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	43 A multilayer perceptron is a feedforward artificial neural network model
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	44 that has one layer or more of hidden units and nonlinear activations.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	45 Intermidiate layers usually have as activation function thanh or the
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	46 sigmoid function while the top layer is a softamx layer.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	47 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	48
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	49
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	50
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	51 def __init__(self, input, n_in, n_hidden, n_out,learning_rate, detection_mode=0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	52 """Initialize the parameters for the multilayer perceptron
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	53
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	54 :param input: symbolic variable that describes the input of the
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	55 architecture (one minibatch)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	56
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	57 :param n_in: number of input units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	58 which the datapoints lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	59
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	60 :param n_hidden: number of hidden units
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	61
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	62 :param n_out: number of output units, the dimension of the space in
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	63 which the labels lie
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	64
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	65 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	66
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	67 # initialize the parameters theta = (W1,b1,W2,b2) ; note that this
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	68 # example contains only one hidden layer, but one can have as many
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	69 # layers as he/she wishes, making the network deeper. The only
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	70 # problem making the network deep this way is during learning,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	71 # backpropagation being unable to move the network from the starting
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	72 # point towards; this is where pre-training helps, giving a good
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	73 # starting point for backpropagation, but more about this in the
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	74 # other tutorials
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	75
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	76 # `W1` is initialized with `W1_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	77 # from -6./sqrt(n_in+n_hidden) and 6./sqrt(n_in+n_hidden)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	78 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	79 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	80 W1_values = numpy.asarray( numpy.random.uniform( \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	81 low = -numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	82 high = numpy.sqrt(6./(n_in+n_hidden)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	83 size = (n_in, n_hidden)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	84 # `W2` is initialized with `W2_values` which is uniformely sampled
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	85 # from -6./sqrt(n_hidden+n_out) and 6./sqrt(n_hidden+n_out)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	86 # the output of uniform if converted using asarray to dtype
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	87 # theano.config.floatX so that the code is runable on GPU
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	88 W2_values = numpy.asarray( numpy.random.uniform(
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	89 low = -numpy.sqrt(6./(n_hidden+n_out)), \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	90 high= numpy.sqrt(6./(n_hidden+n_out)),\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	91 size= (n_hidden, n_out)), dtype = theano.config.floatX)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	92
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	93 self.W1 = theano.shared( value = W1_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	94 self.b1 = theano.shared( value = numpy.zeros((n_hidden,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	95 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	96 self.W2 = theano.shared( value = W2_values )
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	97 self.b2 = theano.shared( value = numpy.zeros((n_out,),
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	98 dtype= theano.config.floatX))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	99
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	100 #include the learning rate in the classifer so
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	101 #we can modify it on the fly when we want
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	102 lr_value=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	103 self.lr=theano.shared(value=lr_value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	104 # symbolic expression computing the values of the hidden layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	105 self.hidden = T.tanh(T.dot(input, self.W1)+ self.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	106
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	107
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	108
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	109 # symbolic expression computing the values of the top layer
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	110 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	111 self.p_y_given_x= T.nnet.sigmoid(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	112 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	113 self.p_y_given_x= T.nnet.softmax(T.dot(self.hidden, self.W2)+self.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	114
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	115 # compute prediction as class whose probability is maximal in
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	116 # symbolic form
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	117 self.y_pred = T.argmax( self.p_y_given_x, axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	118 self.y_pred_num = T.argmax( self.p_y_given_x[0:9], axis =1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	119
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	120
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	121
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	122
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	123 # L1 norm ; one regularization option is to enforce L1 norm to
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	124 # be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	125 self.L1 = abs(self.W1).sum() + abs(self.W2).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	126
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	127 # square of L2 norm ; one regularization option is to enforce
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	128 # square of L2 norm to be small
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	129 self.L2_sqr = (self.W12).sum() + (self.W22).sum()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	130
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	131
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	132
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	133 def negative_log_likelihood(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	134 """Return the mean of the negative log-likelihood of the prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	135 of this model under a given target distribution.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	136
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	137 .. math::
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	138
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	139 \frac{1}{\|\mathcal{D}\|}\mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	140 \frac{1}{\|\mathcal{D}\|}\sum_{i=0}^{\|\mathcal{D}\|} \log(P(Y=y^{(i)}\|x^{(i)}, W,b)) \\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	141 \ell (\theta=\{W,b\}, \mathcal{D})
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	142
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	143
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	144 :param y: corresponds to a vector that gives for each example the
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	145 :correct label
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	146 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	147 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	148
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	149
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	150 def cross_entropy(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	151 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y]+T.sum(T.log(1-self.p_y_given_x), axis=1)-T.log(1-self.p_y_given_x)[T.arange(y.shape[0]),y])
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	152
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	153 def errors(self, y):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	154 """Return a float representing the number of errors in the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	155 over the total number of examples of the minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	156 """
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	157
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	158 # check if y has same dimension of y_pred
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	159 if y.ndim != self.y_pred.ndim:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	160 raise TypeError('y should have the same shape as self.y_pred',
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	161 ('y', target.type, 'y_pred', self.y_pred.type))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	162 # check if y is of the correct datatype
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	163 if y.dtype.startswith('int'):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	164 # the T.neq operator returns a vector of 0s and 1s, where 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	165 # represents a mistake in prediction
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	166 return T.mean(T.neq(self.y_pred, y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	167 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	168 raise NotImplementedError()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	169
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	170
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	171 def mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	172 adaptive_lr = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	173 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	174 learning_rate=0.01,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	175 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	176 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	177 nb_max_exemples=1000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	178 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	179 nb_hidden = 30,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	180 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	181 tau=1e6,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	182 lr_t2_factor=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	183 detection_mode = 0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	184 reduce_label = 0):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	185
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	186
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	187 configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr, detection_mode, reduce_label]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	188
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	189 if(verbose):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	190 print(('verbose: %i') % (verbose))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	191 print(('adaptive_lr: %i') % (adaptive_lr))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	192 print(('data_set: %i') % (data_set))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	193 print(('learning_rate: %f') % (learning_rate))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	194 print(('L1_reg: %f') % (L1_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	195 print(('L2_reg: %f') % (L2_reg))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	196 print(('nb_max_exemples: %i') % (nb_max_exemples))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	197 print(('batch_size: %i') % (batch_size))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	198 print(('nb_hidden: %i') % (nb_hidden))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	199 print(('nb_targets: %f') % (nb_targets))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	200 print(('tau: %f') % (tau))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	201 print(('lr_t2_factor: %f') % (lr_t2_factor))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	202 print(('detection_mode: %i') % (detection_mode))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	203 print(('reduce_label: %i') % (reduce_label))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	204
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	205 # define the number of output - reduce_label : merge the lower and upper case. i.e a and A will both have label 10
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	206 if(reduce_label):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	207 nb_targets = 36
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	208 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	209 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	210
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	211 #save initial learning rate if classical adaptive lr is used
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	212 initial_lr=learning_rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	213
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	214 total_validation_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	215 total_train_error_list = []
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	216 learning_rate_list=[]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	217 best_training_error=float('inf');
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	218
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	219 if data_set==0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	220 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	221
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	222
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	223
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	224
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	225 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	226
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	227 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	228 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	229 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	230 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	231
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	232
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	233 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	234 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	235 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	236 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	237 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	238 learning_rate=learning_rate,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	239 detection_mode = detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	240
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	241
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	242
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	243
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	244 # the cost we minimize during training is the negative log likelihood of
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	245 # the model plus the regularization terms (L1 and L2); cost is expressed
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	246 # here symbolically
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	247 if(detection_mode):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	248 cost = classifier.cross_entropy(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	249 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	250 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	251 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	252 cost = classifier.negative_log_likelihood(y) \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	253 + L1_reg * classifier.L1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	254 + L2_reg * classifier.L2_sqr
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	255
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	256 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	257 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	258 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	259
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	260 # compute the gradient of cost with respect to theta = (W1, b1, W2, b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	261 g_W1 = T.grad(cost, classifier.W1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	262 g_b1 = T.grad(cost, classifier.b1)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	263 g_W2 = T.grad(cost, classifier.W2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	264 g_b2 = T.grad(cost, classifier.b2)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	265
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	266 # specify how to update the parameters of the model as a dictionary
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	267 updates = \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	268 { classifier.W1: classifier.W1 - classifier.lr*g_W1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	269 , classifier.b1: classifier.b1 - classifier.lr*g_b1 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	270 , classifier.W2: classifier.W2 - classifier.lr*g_W2 \
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	271 , classifier.b2: classifier.b2 - classifier.lr*g_b2 }
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	272
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	273 # compiling a theano function `train_model` that returns the cost, but in
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	274 # the same time updates the parameter of the model based on the rules
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	275 # defined in `updates`
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	276 train_model = theano.function([x, y], cost, updates = updates )
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	277
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	278
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	279
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	280
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	281
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	282
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	283
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	284
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	285
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	286 #conditions for stopping the adaptation:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	287 #1) we have reached nb_max_exemples (this is rounded up to be a multiple of the train size)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	288 #2) validation error is going up twice in a row(probable overfitting)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	289
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	290 # This means we no longer stop on slow convergence as low learning rates stopped
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	291 # too fast.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	292
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	293 #approximate number of samples in the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	294 #this is just to have a validation frequency
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	295 #roughly proportionnal to the training set
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	296 n_minibatches = 650000/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	297
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	298
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	299 patience =nb_max_exemples/batch_size #in units of minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	300 patience_increase = 2 # wait this much longer when a new best is
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	301 # found
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	302 improvement_threshold = 0.995 # a relative improvement of this much is
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	303 # considered significant
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	304 validation_frequency = n_minibatches/4
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	305
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	306
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	307
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	308
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	309
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	310 best_validation_loss = float('inf')
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	311 best_iter = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	312 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	313 start_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	314 time_n=0 #in unit of exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	315 minibatch_index=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	316 epoch=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	317 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	318
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	319
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	320
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	321 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	322 print 'looking at most at %i exemples' %nb_max_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	323 while(minibatch_index*batch_size<nb_max_exemples):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	324
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	325 for x, y in dataset.train(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	326
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	327 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	328 y[y > 35] = y[y > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	329 minibatch_index = minibatch_index + 1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	330 if adaptive_lr==2:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	331 classifier.lr.value = tau*initial_lr/(tau+time_n)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	332
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	333
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	334 #train model
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	335 cost_ij = train_model(x,y)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	336
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	337 if (minibatch_index+1) % validation_frequency == 0:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	338
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	339 #save the current learning rate
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	340 learning_rate_list.append(classifier.lr.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	341
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	342 # compute the validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	343 this_validation_loss = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	344 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	345 for xv,yv in dataset.valid(1):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	346 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	347 yv[yv > 35] = yv[yv > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	348 # sum up the errors for each minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	349 axxa=test_model(xv,yv)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	350 this_validation_loss += axxa
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	351 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	352 # get the average by dividing with the number of minibatches
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	353 this_validation_loss /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	354 #save the validation loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	355 total_validation_error_list.append(this_validation_loss)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	356 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	357 print(('epoch %i, minibatch %i, learning rate %f current validation error %f ') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	358 (epoch, minibatch_index+1,classifier.lr.value,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	359 this_validation_loss*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	360
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	361 # if we got the best validation score until now
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	362 if this_validation_loss < best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	363 # save best validation score and iteration number
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	364 best_validation_loss = this_validation_loss
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	365 best_iter = minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	366 # reset patience if we are going down again
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	367 # so we continue exploring
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	368 patience=nb_max_exemples/batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	369 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	370 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	371 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	372 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	373 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	374 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	375 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	376 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	377 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	378 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	379 print(('epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	380 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	381 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	382 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	383
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	384 # if the validation error is going up, we are overfitting (or oscillating)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	385 # stop converging but run at least to next validation
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	386 # to check overfitting or ocsillation
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	387 # the saved weights of the model will be a bit off in that case
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	388 elif this_validation_loss >= best_validation_loss:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	389 #calculate the test error at this point and exit
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	390 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	391 # however, if adaptive_lr is true, try reducing the lr to
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	392 # get us out of an oscilliation
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	393 if adaptive_lr==1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	394 classifier.lr.value=classifier.lr.value*lr_t2_factor
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	395
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	396 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	397 #cap the patience so we are allowed one more validation error
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	398 #calculation before aborting
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	399 patience = minibatch_index+validation_frequency+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	400 temp=0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	401 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	402 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	403 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	404
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	405 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	406 temp=temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	407 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	408 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	409 print ' validation error is going up, possibly stopping soon'
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	410 print((' epoch %i, minibatch %i, test error of best '
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	411 'model %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	412 (epoch, minibatch_index+1,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	413 test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	414
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	415
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	416
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	417
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	418 if minibatch_index>patience:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	419 print 'we have diverged'
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	420 break
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	421
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	422
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	423 time_n= time_n + batch_size
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	424 epoch = epoch+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	425 end_time = time.clock()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	426 if verbose == 1:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	427 print(('Optimization complete. Best validation score of %f %% '
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	428 'obtained at iteration %i, with test performance %f %%') %
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	429 (best_validation_loss * 100., best_iter, test_score*100.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	430 print ('The code ran for %f minutes' % ((end_time-start_time)/60.))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	431 print minibatch_index
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	432
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	433 #save the model and the weights
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	434 numpy.savez('model.npy', config=configuration, W1=classifier.W1.value,W2=classifier.W2.value, b1=classifier.b1.value,b2=classifier.b2.value)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	435 numpy.savez('results.npy',config=configuration,total_train_error_list=total_train_error_list,total_validation_error_list=total_validation_error_list,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	436 learning_rate_list=learning_rate_list)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	437
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	438 return (best_training_error100.0,best_validation_loss 100.,test_score100.,best_iterbatch_size,(end_time-start_time)/60)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	439
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	440 def test_error(model_file):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	441
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	442 print((' test error on all NIST'))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	443 # load the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	444 a=numpy.load(model_file)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	445 W1=a['W1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	446 W2=a['W2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	447 b1=a['b1']
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	448 b2=a['b2']
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	449 configuration=a['config']
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	450 #configuration = [learning_rate,nb_max_exemples,nb_hidden,adaptive_lr]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	451 learning_rate = configuration[0]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	452 nb_max_exemples = configuration[1]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	453 nb_hidden = configuration[2]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	454 adaptive_lr = configuration[3]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	455
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	456 if(len(configuration) == 6):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	457 detection_mode = configuration[4]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	458 reduce_label = configuration[5]
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	459 else:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	460 detection_mode = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	461 reduce_label = 0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	462
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	463 # define the batch size
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	464 batch_size=20
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	465 #define the nb of target
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	466 nb_targets = 62
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	467
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	468 # create the mlp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	469 ishape = (32,32) # this is the size of NIST images
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	470
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	471 # allocate symbolic variables for the data
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	472 x = T.fmatrix() # the data is presented as rasterized images
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	473 y = T.lvector() # the labels are presented as 1D vector of
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	474 # [long int] labels
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	475
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	476
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	477 # construct the logistic regression class
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	478 classifier = MLP( input=x,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	479 n_in=32*32,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	480 n_hidden=nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	481 n_out=nb_targets,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	482 learning_rate=learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	483 detection_mode=detection_mode)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	484
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	485
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	486 # set the weight into the model
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	487 classifier.W1.value = W1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	488 classifier.b1.value = b1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	489 classifier.W2.value = W2
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	490 classifier.b2.value = b2
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	491
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	492
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	493 # compiling a theano function that computes the mistakes that are made by
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	494 # the model on a minibatch
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	495 test_model = theano.function([x,y], classifier.errors(y))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	496
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	497 # test it on the test set
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	498
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	499 # load NIST ALL
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	500 dataset=datasets.nist_all()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	501 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	502 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	503 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	504 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	505 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	506 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	507 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	508 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	509
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	510 print(( ' test error NIST ALL : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	511
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	512 # load NIST DIGITS
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	513 dataset=datasets.nist_digits()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	514 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	515 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	516 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	517 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	518 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	519 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	520 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	521 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	522
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	523 print(( ' test error NIST digits : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	524
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	525 # load NIST lower
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	526 dataset=datasets.nist_lower()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	527 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	528 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	529 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	530 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	531 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	532 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	533 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	534 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	535
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	536 print(( ' test error NIST lower : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	537
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	538 # load NIST upper
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	539 dataset=datasets.nist_upper()
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	540 test_score = 0.
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	541 temp =0
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	542 for xt,yt in dataset.test(batch_size):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	543 if reduce_label:
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	544 yt[yt > 35] = yt[yt > 35]-26
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	545 test_score += test_model(xt,yt)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	546 temp = temp+1
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	547 test_score /= temp
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	548
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	549 print(( ' test error NIST upper : %f %%') %(test_score*100.0))
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	550
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	551
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	552 if __name__ == '__main__':
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	553 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	554 mlp_full_nist( verbose = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	555 adaptive_lr = 1,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	556 data_set=0,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	557 learning_rate=0.5,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	558 L1_reg = 0.00,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	559 L2_reg = 0.0001,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	560 nb_max_exemples=10000000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	561 batch_size=20,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	562 nb_hidden = 500,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	563 nb_targets = 62,
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	564 tau=100000,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	565 lr_t2_factor=0.5)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	566 '''
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	567
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	568 test_error('model.npy.npz')
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	569
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	570 def jobman_mlp_full_nist(state,channel):
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	571 (train_error,validation_error,test_error,nb_exemples,time)=mlp_full_nist(learning_rate=state.learning_rate,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	572 nb_max_exemples=state.nb_max_exemples,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	573 nb_hidden=state.nb_hidden,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	574 adaptive_lr=state.adaptive_lr,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	575 tau=state.tau,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	576 verbose = state.verbose,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	577 lr_t2_factor=state.lr_t2_factor,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	578 detection_mode = state.detection_mode,\
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	579 reduce_label = state.reduce_label)
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	580 state.train_error=train_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	581 state.validation_error=validation_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	582 state.test_error=test_error
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	583 state.nb_exemples=nb_exemples
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	584 state.time=time
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	585 pylearn.version.record_versions(state,[theano,ift6266,pylearn])
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	586 return channel.COMPLETE
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	587
f2dd75248483 initial commit of mlp with options for detection and 36 classes youssouf parents: diff changeset	588

Mercurial > ift6266

annotate baseline/mlp/v_youssouf/mlp_nist.py @ 644:e63d23c7c9fb