Mercurial > ift6266

annotate deep/deep_mlp/logistic_sgd.py @ 638:677d1b1d8158

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fits
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Sat, 19 Mar 2011 23:11:17 -0400
parents 75dbbe409578
children
rev   line source
626
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
1 import numpy, time, cPickle, gzip, sys, os
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
2
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
3 import theano
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
4 import theano.tensor as T
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
5
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
6 class LogisticRegression(object):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
7 def __init__(self, input, n_in, n_out):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
8 self.W = theano.shared(value=numpy.zeros((n_in,n_out),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
9 dtype = theano.config.floatX),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
10 name='W')
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
11 self.b = theano.shared(value=numpy.zeros((n_out,),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
12 dtype = theano.config.floatX),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
13 name='b')
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
14
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
15 self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W)+self.b)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
16
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
17 self.y_pred=T.argmax(self.p_y_given_x, axis=1)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
18
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
19 self.params = [self.W, self.b]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
20
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
21 def negative_log_likelihood(self, y):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
22 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
23
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
24
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
25 def errors(self, y):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
26 if y.ndim != self.y_pred.ndim:
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
27 raise TypeError('y should have the same shape as self.y_pred',
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
28 ('y', target.type, 'y_pred', self.y_pred.type))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
29
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
30 if y.dtype.startswith('int'):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
31 return T.mean(T.neq(self.y_pred, y))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
32 else:
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
33 raise NotImplementedError()
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
34
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
35
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
36 def load_data(dataset):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
37 ''' Loads the dataset
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
38
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
39 :type dataset: string
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
40 :param dataset: the path to the dataset (here MNIST)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
41 '''
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
42
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
43 #############
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
44 # LOAD DATA #
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
45 #############
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
46 print '... loading data'
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
47
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
48 # Load the dataset
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
49 f = gzip.open(dataset,'rb')
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
50 train_set, valid_set, test_set = cPickle.load(f)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
51 f.close()
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
52
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
53
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
54 def shared_dataset(data_xy):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
55 """ Function that loads the dataset into shared variables
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
56
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
57 The reason we store our dataset in shared variables is to allow
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
58 Theano to copy it into the GPU memory (when code is run on GPU).
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
59 Since copying data into the GPU is slow, copying a minibatch everytime
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
60 is needed (the default behaviour if the data is not in a shared
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
61 variable) would lead to a large decrease in performance.
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
62 """
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
63 data_x, data_y = data_xy
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
64 shared_x = theano.shared(numpy.asarray(data_x, dtype=theano.config.floatX))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
65 shared_y = theano.shared(numpy.asarray(data_y, dtype=theano.config.floatX))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
66 # When storing data on the GPU it has to be stored as floats
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
67 # therefore we will store the labels as ``floatX`` as well
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
68 # (``shared_y`` does exactly that). But during our computations
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
69 # we need them as ints (we use labels as index, and if they are
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
70 # floats it doesn't make sense) therefore instead of returning
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
71 # ``shared_y`` we will have to cast it to int. This little hack
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
72 # lets ous get around this issue
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
73 return shared_x, T.cast(shared_y, 'int32')
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
74
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
75 test_set_x, test_set_y = shared_dataset(test_set)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
76 valid_set_x, valid_set_y = shared_dataset(valid_set)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
77 train_set_x, train_set_y = shared_dataset(train_set)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
78
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
79 rval = [(train_set_x, train_set_y), (valid_set_x,valid_set_y), (test_set_x, test_set_y)]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
80 return rval
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
81
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
82
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
83
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
84
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
85 def sgd_optimization_mnist(learning_rate=0.13, n_epochs=1000, dataset='../data/mnist.pkl.gz',
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
86 batch_size = 600):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
87 datasets = load_data(dataset)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
88
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
89 train_set_x, train_set_y = datasets[0]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
90 valid_set_x, valid_set_y = datasets[1]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
91 test_set_x , test_set_y = datasets[2]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
92
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
93 # compute number of minibatches for training, validation and testing
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
94 n_train_batches = train_set_x.value.shape[0] / batch_size
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
95 n_valid_batches = valid_set_x.value.shape[0] / batch_size
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
96 n_test_batches = test_set_x.value.shape[0] / batch_size
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
97
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
98
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
99 ######################
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
100 # BUILD ACTUAL MODEL #
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
101 ######################
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
102 print '... building the model'
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
103
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
104
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
105 # allocate symbolic variables for the data
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
106 index = T.lscalar() # index to a [mini]batch
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
107 x = T.matrix('x') # the data is presented as rasterized images
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
108 y = T.ivector('y') # the labels are presented as 1D vector of
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
109 # [int] labels
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
110
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
111 # construct the logistic regression class
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
112 # Each MNIST image has size 28*28
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
113 classifier = LogisticRegression( input=x, n_in=28*28, n_out=10)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
114
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
115 # the cost we minimize during training is the negative log likelihood of
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
116 # the model in symbolic format
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
117 cost = classifier.negative_log_likelihood(y)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
118
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
119 # compiling a Theano function that computes the mistakes that are made by
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
120 # the model on a minibatch
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
121 test_model = theano.function(inputs = [index],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
122 outputs = classifier.errors(y),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
123 givens={
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
124 x:test_set_x[index*batch_size:(index+1)*batch_size],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
125 y:test_set_y[index*batch_size:(index+1)*batch_size]})
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
126
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
127 validate_model = theano.function( inputs = [index],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
128 outputs = classifier.errors(y),
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
129 givens={
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
130 x:valid_set_x[index*batch_size:(index+1)*batch_size],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
131 y:valid_set_y[index*batch_size:(index+1)*batch_size]})
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
132
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
133 # compute the gradient of cost with respect to theta = (W,b)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
134 g_W = T.grad(cost = cost, wrt = classifier.W)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
135 g_b = T.grad(cost = cost, wrt = classifier.b)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
136
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
137 # specify how to update the parameters of the model as a dictionary
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
138 updates ={classifier.W: classifier.W - learning_rate*g_W,\
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
139 classifier.b: classifier.b - learning_rate*g_b}
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
140
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
141 # compiling a Theano function `train_model` that returns the cost, but in
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
142 # the same time updates the parameter of the model based on the rules
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
143 # defined in `updates`
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
144 train_model = theano.function(inputs = [index],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
145 outputs = cost,
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
146 updates = updates,
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
147 givens={
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
148 x:train_set_x[index*batch_size:(index+1)*batch_size],
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
149 y:train_set_y[index*batch_size:(index+1)*batch_size]})
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
150
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
151 ###############
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
152 # TRAIN MODEL #
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
153 ###############
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
154 print '... training the model'
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
155 # early-stopping parameters
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
156 patience = 5000 # look as this many examples regardless
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
157 patience_increase = 2 # wait this much longer when a new best is
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
158 # found
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
159 improvement_threshold = 0.995 # a relative improvement of this much is
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
160 # considered significant
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
161 validation_frequency = min(n_train_batches, patience/2)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
162 # go through this many
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
163 # minibatche before checking the network
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
164 # on the validation set; in this case we
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
165 # check every epoch
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
166
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
167 best_params = None
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
168 best_validation_loss = float('inf')
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
169 test_score = 0.
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
170 start_time = time.clock()
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
171
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
172 done_looping = False
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
173 epoch = 0
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
174 while (epoch < n_epochs) and (not done_looping):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
175 epoch = epoch + 1
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
176 for minibatch_index in xrange(n_train_batches):
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
177
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
178 minibatch_avg_cost = train_model(minibatch_index)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
179 # iteration number
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
180 iter = epoch * n_train_batches + minibatch_index
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
181
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
182 if (iter+1) % validation_frequency == 0:
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
183 # compute zero-one loss on validation set
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
184 validation_losses = [validate_model(i) for i in xrange(n_valid_batches)]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
185 this_validation_loss = numpy.mean(validation_losses)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
186
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
187 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
188 (epoch, minibatch_index+1,n_train_batches, \
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
189 this_validation_loss*100.))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
190
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
191
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
192 # if we got the best validation score until now
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
193 if this_validation_loss < best_validation_loss:
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
194 #improve patience if loss improvement is good enough
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
195 if this_validation_loss < best_validation_loss * \
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
196 improvement_threshold :
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
197 patience = max(patience, iter * patience_increase)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
198
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
199 best_validation_loss = this_validation_loss
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
200 # test it on the test set
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
201
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
202 test_losses = [test_model(i) for i in xrange(n_test_batches)]
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
203 test_score = numpy.mean(test_losses)
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
204
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
205 print((' epoch %i, minibatch %i/%i, test error of best '
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
206 'model %f %%') % \
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
207 (epoch, minibatch_index+1, n_train_batches,test_score*100.))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
208
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
209 if patience <= iter :
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
210 done_looping = True
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
211 break
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
212
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
213 end_time = time.clock()
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
214 print(('Optimization complete with best validation score of %f %%,'
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
215 'with test performance %f %%') %
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
216 (best_validation_loss * 100., test_score*100.))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
217 print 'The code run for %d epochs, with %f epochs/sec'%(epoch,1.*epoch/(end_time-start_time))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
218 print >> sys.stderr, ('The code for file '+os.path.split(__file__)[1]+' ran for %.1fs' % ((end_time-start_time)))
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
219
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
220 if __name__ == '__main__':
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
221 sgd_optimization_mnist()
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
222
75dbbe409578 Added code for deep mlp, experiment code to go along with it. Also added code I used to filter the P07 / PNIST07 datasets to keep only digits.
fsavard
parents:
diff changeset
223