Mercurial > ift6266

annotate scripts/stacked_dae.py @ 114:0b4080394f2c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
author fsavard
date Wed, 17 Feb 2010 09:29:19 -0500
parents
children 4f37755d301b
rev   line source
114
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
1 #!/usr/bin/python
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
2 # coding: utf-8
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
3
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
4 # Code for stacked denoising autoencoder
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
5 # Tests with MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
6 # TODO: adapt for NIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
7 # Based almost entirely on deeplearning.net tutorial, modifications by
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
8 # François Savard
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
9
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
10 # Base LogisticRegression, SigmoidalLayer, dA, SdA code taken
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
11 # from the deeplearning.net tutorial. Refactored a bit.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
12 # Changes (mainly):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
13 # - splitted initialization in smaller methods
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
14 # - removed the "givens" thing involving an index in the whole dataset
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
15 # (to allow flexibility in how data is inputted... not necessarily one big tensor)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
16 # - changed the "driver" a lot, altough for the moment the same logic is used
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
17
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
18 import time
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
19 import theano
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
20 import theano.tensor as T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
21 import theano.tensor.nnet
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
22 from theano.tensor.shared_randomstreams import RandomStreams
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
23 import numpy, numpy.random
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
24
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
25 from pylearn.datasets import MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
26
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
27
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
28 # from pylearn codebase
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
29 def update_locals(obj, dct):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
30 if 'self' in dct:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
31 del dct['self']
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
32 obj.__dict__.update(dct)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
33
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
34
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
35 class LogisticRegression(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
36 def __init__(self, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
37 # initialize with 0 the weights W as a matrix of shape (n_in, n_out)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
38 self.W = theano.shared(value=numpy.zeros((n_in,n_out), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
39 name='W')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
40 # initialize the baises b as a vector of n_out 0s
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
41 self.b = theano.shared(value=numpy.zeros((n_out,), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
42 name='b')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
43
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
44 # compute vector of class-membership probabilities in symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
45 self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W)+self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
46
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
47 # compute prediction as class whose probability is maximal in
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
48 # symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
49 self.y_pred=T.argmax(self.p_y_given_x, axis=1)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
50
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
51 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
52
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
53 def negative_log_likelihood(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
54 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
55
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
56 def errors(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
57 # check if y has same dimension of y_pred
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
58 if y.ndim != self.y_pred.ndim:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
59 raise TypeError('y should have the same shape as self.y_pred',
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
60 ('y', target.type, 'y_pred', self.y_pred.type))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
61 # check if y is of the correct datatype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
62 if y.dtype.startswith('int'):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
63 # the T.neq operator returns a vector of 0s and 1s, where 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
64 # represents a mistake in prediction
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
65 return T.mean(T.neq(self.y_pred, y))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
66 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
67 raise NotImplementedError()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
68
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
69
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
70 class SigmoidalLayer(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
71 def __init__(self, rng, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
72 self.input = input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
73
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
74 W_values = numpy.asarray( rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
75 low = -numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
76 high = numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
77 size = (n_in, n_out)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
78 self.W = theano.shared(value = W_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
79
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
80 b_values = numpy.zeros((n_out,), dtype= theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
81 self.b = theano.shared(value= b_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
82
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
83 self.output = T.nnet.sigmoid(T.dot(input, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
84 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
85
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
86
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
87 class dA(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
88 def __init__(self, n_visible= 784, n_hidden= 500, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
89 corruption_level = 0.1, input = None, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
90 shared_W = None, shared_b = None):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
91 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
92
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
93 self.init_randomizer()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
94 self.init_params()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
95 self.init_functions()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
96
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
97 def init_randomizer(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
98 # create a Theano random generator that gives symbolic random values
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
99 self.theano_rng = RandomStreams()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
100 # create a numpy random generator
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
101 self.numpy_rng = numpy.random.RandomState()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
102
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
103 def init_params(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
104 if self.shared_W != None and self.shared_b != None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
105 self.W = self.shared_W
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
106 self.b = self.shared_b
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
107 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
108 # initial values for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
109 # note : W' was written as `W_prime` and b' as `b_prime`
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
110
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
111 # W is initialized with `initial_W` which is uniformely sampled
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
112 # from -6./sqrt(n_visible+n_hidden) and 6./sqrt(n_hidden+n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
113 # the output of uniform if converted using asarray to dtype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
114 # theano.config.floatX so that the code is runable on GPU
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
115 initial_W = numpy.asarray( self.numpy_rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
116 low = -numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
117 high = numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
118 size = (n_visible, n_hidden)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
119 initial_b = numpy.zeros(n_hidden)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
120
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
121 # theano shared variables for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
122 self.W = theano.shared(value = initial_W, name = "W")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
123 self.b = theano.shared(value = initial_b, name = "b")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
124
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
125 initial_b_prime= numpy.zeros(self.n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
126 # tied weights, therefore W_prime is W transpose
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
127 self.W_prime = self.W.T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
128 self.b_prime = theano.shared(value = initial_b_prime, name = "b'")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
129
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
130 def init_functions(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
131 # if no input is given, generate a variable representing the input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
132 if self.input == None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
133 # we use a matrix because we expect a minibatch of several examples,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
134 # each example being a row
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
135 self.x = T.dmatrix(name = 'input')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
136 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
137 self.x = self.input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
138
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
139 # keep 90% of the inputs the same and zero-out randomly selected subset of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
140 # 10% of the inputs
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
141 # note : first argument of theano.rng.binomial is the shape(size) of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
142 # random numbers that it should produce
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
143 # second argument is the number of trials
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
144 # third argument is the probability of success of any trial
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
145 #
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
146 # this will produce an array of 0s and 1s where 1 has a
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
147 # probability of 1 - ``corruption_level`` and 0 with
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
148 # ``corruption_level``
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
149 self.tilde_x = self.theano_rng.binomial(self.x.shape, 1, 1-self.corruption_level) * self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
150 # using tied weights
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
151 self.y = T.nnet.sigmoid(T.dot(self.tilde_x, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
152 self.z = T.nnet.sigmoid(T.dot(self.y, self.W_prime) + self.b_prime)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
153 self.L = - T.sum( self.x*T.log(self.z) + (1-self.x)*T.log(1-self.z), axis=1 )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
154 # note : L is now a vector, where each element is the cross-entropy cost
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
155 # of the reconstruction of the corresponding example of the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
156 # minibatch. We need to compute the average of all these to get
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
157 # the cost of the minibatch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
158 self.cost = T.mean(self.L)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
159
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
160 self.params = [ self.W, self.b, self.b_prime ]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
161
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
162 class SdA():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
163 def __init__(self, batch_size, n_ins,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
164 hidden_layers_sizes, n_outs,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
165 corruption_levels, rng, pretrain_lr, finetune_lr):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
166 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
167
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
168 self.layers = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
169 self.pretrain_functions = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
170 self.params = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
171 self.n_layers = len(hidden_layers_sizes)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
172
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
173 if len(hidden_layers_sizes) < 1 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
174 raiseException (' You must have at least one hidden layer ')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
175
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
176 # allocate symbolic variables for the data
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
177 self.x = T.matrix('x') # the data is presented as rasterized images
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
178 self.y = T.ivector('y') # the labels are presented as 1D vector of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
179 # [int] labels
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
180
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
181 self.create_layers()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
182 self.init_finetuning()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
183
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
184 def create_layers(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
185 for i in xrange( self.n_layers ):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
186 # construct the sigmoidal layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
187
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
188 # the size of the input is either the number of hidden units of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
189 # the layer below or the input size if we are on the first layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
190 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
191 input_size = self.n_ins
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
192 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
193 input_size = self.hidden_layers_sizes[i-1]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
194
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
195 # the input to this layer is either the activation of the hidden
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
196 # layer below or the input of the SdA if you are on the first
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
197 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
198 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
199 layer_input = self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
200 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
201 layer_input = self.layers[-1].output
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
202
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
203 layer = SigmoidalLayer(self.rng, layer_input, input_size,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
204 self.hidden_layers_sizes[i] )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
205 # add the layer to the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
206 self.layers += [layer]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
207 self.params += layer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
208
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
209 # Construct a denoising autoencoder that shared weights with this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
210 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
211 dA_layer = dA(input_size, self.hidden_layers_sizes[i], \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
212 corruption_level = self.corruption_levels[0],\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
213 input = layer_input, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
214 shared_W = layer.W, shared_b = layer.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
215
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
216 self.init_updates_for_layer(dA_layer)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
217
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
218 def init_updates_for_layer(self, dA_layer):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
219 # Construct a function that trains this dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
220 # compute gradients of layer parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
221 gparams = T.grad(dA_layer.cost, dA_layer.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
222 # compute the list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
223 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
224 for param, gparam in zip(dA_layer.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
225 updates[param] = param - gparam * self.pretrain_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
226
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
227 # create a function that trains the dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
228 update_fn = theano.function([self.x], dA_layer.cost, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
229 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
230
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
231 # collect this function into a list
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
232 self.pretrain_functions += [update_fn]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
233
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
234 def init_finetuning(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
235 # We now need to add a logistic layer on top of the MLP
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
236 self.logLayer = LogisticRegression(\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
237 input = self.layers[-1].output,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
238 n_in = self.hidden_layers_sizes[-1], n_out = self.n_outs)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
239
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
240 self.params += self.logLayer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
241 # construct a function that implements one step of finetunining
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
242
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
243 # compute the cost, defined as the negative log likelihood
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
244 cost = self.logLayer.negative_log_likelihood(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
245 # compute the gradients with respect to the model parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
246 gparams = T.grad(cost, self.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
247 # compute list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
248 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
249 for param,gparam in zip(self.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
250 updates[param] = param - gparam*self.finetune_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
251
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
252 self.finetune = theano.function([self.x, self.y], cost,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
253 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
254
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
255 # symbolic variable that points to the number of errors made on the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
256 # minibatch given by self.x and self.y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
257
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
258 self.errors = self.logLayer.errors(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
259
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
260 class MnistIterators:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
261 def __init__(self, minibatch_size):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
262 self.minibatch_size = minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
263
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
264 self.mnist = MNIST.first_1k()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
265
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
266 self.len_train = len(self.mnist.train.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
267 self.len_valid = len(self.mnist.valid.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
268 self.len_test = len(self.mnist.test.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
269
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
270 def train_x_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
271 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
272 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
273 yield self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
274 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
275
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
276 def train_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
277 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
278 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
279 mb_x = self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
280 mb_y = self.mnist.train.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
281 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
282 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
283
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
284 def valid_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
285 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
286 while idx < len(self.mnist.valid.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
287 mb_x = self.mnist.valid.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
288 mb_y = self.mnist.valid.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
289 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
290 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
291
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
292
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
293 class MnistTrainingDriver:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
294 def __init__(self, rng=numpy.random):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
295 self.rng = rng
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
296
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
297 self.init_SdA()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
298
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
299 def init_SdA(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
300 # Hyperparam
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
301 hidden_layers_sizes = [1000, 1000, 1000]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
302 n_outs = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
303 corruption_levels = [0.2, 0.2, 0.2]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
304 minibatch_size = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
305 pretrain_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
306 finetune_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
307
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
308 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
309
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
310 self.mnist = MnistIterators(minibatch_size)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
311
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
312 # construct the stacked denoising autoencoder class
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
313 self.classifier = SdA( batch_size = minibatch_size, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
314 n_ins=28*28, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
315 hidden_layers_sizes = hidden_layers_sizes, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
316 n_outs=n_outs, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
317 corruption_levels = corruption_levels,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
318 rng = self.rng,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
319 pretrain_lr = pretrain_lr, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
320 finetune_lr = finetune_lr)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
321
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
322 def compute_validation_error(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
323 validation_error = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
324
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
325 count = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
326 for mb_x, mb_y in self.mnist.valid_xy_batches():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
327 validation_error += self.classifier.errors(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
328 count += 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
329
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
330 return float(validation_error) / count
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
331
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
332 def pretrain(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
333 pretraining_epochs = 20
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
334
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
335 for layer_idx, update_fn in enumerate(self.classifier.pretrain_functions):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
336 for epoch in xrange(pretraining_epochs):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
337 # go through the training set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
338 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
339 for i, mb_x in enumerate(self.mnist.train_x_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
340 cost_acc += update_fn(mb_x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
341
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
342 if i % 100 == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
343 print i, "avg err = ", cost_acc / 100.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
344 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
345 print 'Pre-training layer %d, epoch %d' % (layer_idx, epoch)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
346
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
347 def finetune(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
348 max_training_epochs = 1000
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
349
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
350 n_train_batches = self.mnist.len_train / self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
351
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
352 # early-stopping parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
353 patience = 10000 # look as this many examples regardless
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
354 patience_increase = 2. # wait this much longer when a new best is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
355 # found
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
356 improvement_threshold = 0.995 # a relative improvement of this much is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
357 # considered significant
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
358 validation_frequency = min(n_train_batches, patience/2)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
359 # go through this many
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
360 # minibatche before checking the network
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
361 # on the validation set; in this case we
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
362 # check every epoch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
363
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
364
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
365 # TODO: use this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
366 best_params = None
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
367 best_validation_loss = float('inf')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
368 test_score = 0.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
369 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
370
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
371 done_looping = False
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
372 epoch = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
373
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
374 while (epoch < max_training_epochs) and (not done_looping):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
375 epoch = epoch + 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
376 for minibatch_index, (mb_x, mb_y) in enumerate(self.mnist.train_xy_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
377 cost_ij = classifier.finetune(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
378 iter = epoch * n_train_batches + minibatch_index
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
379
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
380 if (iter+1) % validation_frequency == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
381 this_validation_loss = self.compute_validation_error()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
382 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
383 (epoch, minibatch_index+1, n_train_batches, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
384 this_validation_loss*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
385
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
386 # if we got the best validation score until now
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
387 if this_validation_loss < best_validation_loss:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
388
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
389 #improve patience if loss improvement is good enough
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
390 if this_validation_loss < best_validation_loss * \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
391 improvement_threshold :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
392 patience = max(patience, iter * patience_increase)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
393 print "Improving patience"
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
394
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
395 # save best validation score and iteration number
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
396 best_validation_loss = this_validation_loss
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
397 best_iter = iter
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
398
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
399 # test it on the test set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
400 #test_losses = [test_model(i) for i in xrange(n_test_batches)]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
401 #test_score = numpy.mean(test_losses)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
402 #print((' epoch %i, minibatch %i/%i, test error of best '
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
403 # 'model %f %%') %
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
404 # (epoch, minibatch_index+1, n_train_batches,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
405 # test_score*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
406
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
407
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
408 if patience <= iter :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
409 done_looping = True
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
410 break
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
411
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
412 def train():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
413 driver = MnistTrainingDriver()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
414 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
415 driver.pretrain()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
416 print "PRETRAINING DONE. STARTING FINETUNING."
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
417 driver.finetune()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
418 end_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
419
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
420 if __name__ == '__main__':
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
421 train()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
422