annotate scripts/stacked_dae.py @ 115:b84a0d009af8

changes on pipeline mecanism: we now sample a different complexity for each transformations, this because when we use the same sampled complexity for all the modules 1/8 of the time we are close to 0 and we obtain an image very close to the source, we now save a complexity for each module in the parameters array
author Xavier Glorot <glorotxa@iro.umontreal.ca>
date Wed, 17 Feb 2010 16:20:15 -0500
parents 0b4080394f2c
children 4f37755d301b
rev   line source
114
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
1 #!/usr/bin/python
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
2 # coding: utf-8
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
3
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
4 # Code for stacked denoising autoencoder
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
5 # Tests with MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
6 # TODO: adapt for NIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
7 # Based almost entirely on deeplearning.net tutorial, modifications by
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
8 # François Savard
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
9
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
10 # Base LogisticRegression, SigmoidalLayer, dA, SdA code taken
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
11 # from the deeplearning.net tutorial. Refactored a bit.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
12 # Changes (mainly):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
13 # - splitted initialization in smaller methods
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
14 # - removed the "givens" thing involving an index in the whole dataset
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
15 # (to allow flexibility in how data is inputted... not necessarily one big tensor)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
16 # - changed the "driver" a lot, altough for the moment the same logic is used
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
17
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
18 import time
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
19 import theano
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
20 import theano.tensor as T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
21 import theano.tensor.nnet
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
22 from theano.tensor.shared_randomstreams import RandomStreams
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
23 import numpy, numpy.random
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
24
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
25 from pylearn.datasets import MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
26
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
27
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
28 # from pylearn codebase
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
29 def update_locals(obj, dct):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
30 if 'self' in dct:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
31 del dct['self']
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
32 obj.__dict__.update(dct)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
33
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
34
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
35 class LogisticRegression(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
36 def __init__(self, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
37 # initialize with 0 the weights W as a matrix of shape (n_in, n_out)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
38 self.W = theano.shared(value=numpy.zeros((n_in,n_out), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
39 name='W')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
40 # initialize the baises b as a vector of n_out 0s
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
41 self.b = theano.shared(value=numpy.zeros((n_out,), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
42 name='b')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
43
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
44 # compute vector of class-membership probabilities in symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
45 self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W)+self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
46
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
47 # compute prediction as class whose probability is maximal in
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
48 # symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
49 self.y_pred=T.argmax(self.p_y_given_x, axis=1)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
50
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
51 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
52
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
53 def negative_log_likelihood(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
54 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
55
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
56 def errors(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
57 # check if y has same dimension of y_pred
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
58 if y.ndim != self.y_pred.ndim:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
59 raise TypeError('y should have the same shape as self.y_pred',
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
60 ('y', target.type, 'y_pred', self.y_pred.type))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
61 # check if y is of the correct datatype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
62 if y.dtype.startswith('int'):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
63 # the T.neq operator returns a vector of 0s and 1s, where 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
64 # represents a mistake in prediction
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
65 return T.mean(T.neq(self.y_pred, y))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
66 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
67 raise NotImplementedError()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
68
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
69
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
70 class SigmoidalLayer(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
71 def __init__(self, rng, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
72 self.input = input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
73
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
74 W_values = numpy.asarray( rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
75 low = -numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
76 high = numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
77 size = (n_in, n_out)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
78 self.W = theano.shared(value = W_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
79
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
80 b_values = numpy.zeros((n_out,), dtype= theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
81 self.b = theano.shared(value= b_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
82
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
83 self.output = T.nnet.sigmoid(T.dot(input, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
84 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
85
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
86
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
87 class dA(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
88 def __init__(self, n_visible= 784, n_hidden= 500, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
89 corruption_level = 0.1, input = None, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
90 shared_W = None, shared_b = None):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
91 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
92
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
93 self.init_randomizer()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
94 self.init_params()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
95 self.init_functions()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
96
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
97 def init_randomizer(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
98 # create a Theano random generator that gives symbolic random values
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
99 self.theano_rng = RandomStreams()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
100 # create a numpy random generator
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
101 self.numpy_rng = numpy.random.RandomState()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
102
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
103 def init_params(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
104 if self.shared_W != None and self.shared_b != None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
105 self.W = self.shared_W
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
106 self.b = self.shared_b
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
107 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
108 # initial values for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
109 # note : W' was written as `W_prime` and b' as `b_prime`
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
110
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
111 # W is initialized with `initial_W` which is uniformely sampled
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
112 # from -6./sqrt(n_visible+n_hidden) and 6./sqrt(n_hidden+n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
113 # the output of uniform if converted using asarray to dtype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
114 # theano.config.floatX so that the code is runable on GPU
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
115 initial_W = numpy.asarray( self.numpy_rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
116 low = -numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
117 high = numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
118 size = (n_visible, n_hidden)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
119 initial_b = numpy.zeros(n_hidden)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
120
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
121 # theano shared variables for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
122 self.W = theano.shared(value = initial_W, name = "W")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
123 self.b = theano.shared(value = initial_b, name = "b")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
124
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
125 initial_b_prime= numpy.zeros(self.n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
126 # tied weights, therefore W_prime is W transpose
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
127 self.W_prime = self.W.T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
128 self.b_prime = theano.shared(value = initial_b_prime, name = "b'")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
129
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
130 def init_functions(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
131 # if no input is given, generate a variable representing the input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
132 if self.input == None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
133 # we use a matrix because we expect a minibatch of several examples,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
134 # each example being a row
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
135 self.x = T.dmatrix(name = 'input')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
136 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
137 self.x = self.input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
138
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
139 # keep 90% of the inputs the same and zero-out randomly selected subset of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
140 # 10% of the inputs
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
141 # note : first argument of theano.rng.binomial is the shape(size) of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
142 # random numbers that it should produce
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
143 # second argument is the number of trials
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
144 # third argument is the probability of success of any trial
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
145 #
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
146 # this will produce an array of 0s and 1s where 1 has a
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
147 # probability of 1 - ``corruption_level`` and 0 with
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
148 # ``corruption_level``
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
149 self.tilde_x = self.theano_rng.binomial(self.x.shape, 1, 1-self.corruption_level) * self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
150 # using tied weights
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
151 self.y = T.nnet.sigmoid(T.dot(self.tilde_x, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
152 self.z = T.nnet.sigmoid(T.dot(self.y, self.W_prime) + self.b_prime)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
153 self.L = - T.sum( self.x*T.log(self.z) + (1-self.x)*T.log(1-self.z), axis=1 )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
154 # note : L is now a vector, where each element is the cross-entropy cost
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
155 # of the reconstruction of the corresponding example of the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
156 # minibatch. We need to compute the average of all these to get
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
157 # the cost of the minibatch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
158 self.cost = T.mean(self.L)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
159
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
160 self.params = [ self.W, self.b, self.b_prime ]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
161
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
162 class SdA():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
163 def __init__(self, batch_size, n_ins,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
164 hidden_layers_sizes, n_outs,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
165 corruption_levels, rng, pretrain_lr, finetune_lr):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
166 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
167
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
168 self.layers = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
169 self.pretrain_functions = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
170 self.params = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
171 self.n_layers = len(hidden_layers_sizes)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
172
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
173 if len(hidden_layers_sizes) < 1 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
174 raiseException (' You must have at least one hidden layer ')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
175
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
176 # allocate symbolic variables for the data
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
177 self.x = T.matrix('x') # the data is presented as rasterized images
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
178 self.y = T.ivector('y') # the labels are presented as 1D vector of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
179 # [int] labels
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
180
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
181 self.create_layers()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
182 self.init_finetuning()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
183
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
184 def create_layers(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
185 for i in xrange( self.n_layers ):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
186 # construct the sigmoidal layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
187
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
188 # the size of the input is either the number of hidden units of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
189 # the layer below or the input size if we are on the first layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
190 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
191 input_size = self.n_ins
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
192 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
193 input_size = self.hidden_layers_sizes[i-1]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
194
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
195 # the input to this layer is either the activation of the hidden
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
196 # layer below or the input of the SdA if you are on the first
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
197 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
198 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
199 layer_input = self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
200 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
201 layer_input = self.layers[-1].output
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
202
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
203 layer = SigmoidalLayer(self.rng, layer_input, input_size,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
204 self.hidden_layers_sizes[i] )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
205 # add the layer to the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
206 self.layers += [layer]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
207 self.params += layer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
208
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
209 # Construct a denoising autoencoder that shared weights with this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
210 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
211 dA_layer = dA(input_size, self.hidden_layers_sizes[i], \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
212 corruption_level = self.corruption_levels[0],\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
213 input = layer_input, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
214 shared_W = layer.W, shared_b = layer.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
215
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
216 self.init_updates_for_layer(dA_layer)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
217
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
218 def init_updates_for_layer(self, dA_layer):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
219 # Construct a function that trains this dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
220 # compute gradients of layer parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
221 gparams = T.grad(dA_layer.cost, dA_layer.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
222 # compute the list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
223 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
224 for param, gparam in zip(dA_layer.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
225 updates[param] = param - gparam * self.pretrain_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
226
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
227 # create a function that trains the dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
228 update_fn = theano.function([self.x], dA_layer.cost, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
229 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
230
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
231 # collect this function into a list
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
232 self.pretrain_functions += [update_fn]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
233
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
234 def init_finetuning(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
235 # We now need to add a logistic layer on top of the MLP
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
236 self.logLayer = LogisticRegression(\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
237 input = self.layers[-1].output,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
238 n_in = self.hidden_layers_sizes[-1], n_out = self.n_outs)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
239
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
240 self.params += self.logLayer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
241 # construct a function that implements one step of finetunining
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
242
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
243 # compute the cost, defined as the negative log likelihood
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
244 cost = self.logLayer.negative_log_likelihood(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
245 # compute the gradients with respect to the model parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
246 gparams = T.grad(cost, self.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
247 # compute list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
248 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
249 for param,gparam in zip(self.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
250 updates[param] = param - gparam*self.finetune_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
251
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
252 self.finetune = theano.function([self.x, self.y], cost,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
253 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
254
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
255 # symbolic variable that points to the number of errors made on the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
256 # minibatch given by self.x and self.y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
257
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
258 self.errors = self.logLayer.errors(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
259
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
260 class MnistIterators:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
261 def __init__(self, minibatch_size):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
262 self.minibatch_size = minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
263
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
264 self.mnist = MNIST.first_1k()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
265
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
266 self.len_train = len(self.mnist.train.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
267 self.len_valid = len(self.mnist.valid.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
268 self.len_test = len(self.mnist.test.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
269
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
270 def train_x_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
271 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
272 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
273 yield self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
274 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
275
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
276 def train_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
277 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
278 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
279 mb_x = self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
280 mb_y = self.mnist.train.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
281 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
282 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
283
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
284 def valid_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
285 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
286 while idx < len(self.mnist.valid.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
287 mb_x = self.mnist.valid.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
288 mb_y = self.mnist.valid.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
289 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
290 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
291
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
292
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
293 class MnistTrainingDriver:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
294 def __init__(self, rng=numpy.random):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
295 self.rng = rng
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
296
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
297 self.init_SdA()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
298
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
299 def init_SdA(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
300 # Hyperparam
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
301 hidden_layers_sizes = [1000, 1000, 1000]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
302 n_outs = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
303 corruption_levels = [0.2, 0.2, 0.2]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
304 minibatch_size = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
305 pretrain_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
306 finetune_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
307
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
308 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
309
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
310 self.mnist = MnistIterators(minibatch_size)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
311
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
312 # construct the stacked denoising autoencoder class
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
313 self.classifier = SdA( batch_size = minibatch_size, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
314 n_ins=28*28, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
315 hidden_layers_sizes = hidden_layers_sizes, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
316 n_outs=n_outs, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
317 corruption_levels = corruption_levels,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
318 rng = self.rng,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
319 pretrain_lr = pretrain_lr, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
320 finetune_lr = finetune_lr)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
321
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
322 def compute_validation_error(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
323 validation_error = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
324
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
325 count = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
326 for mb_x, mb_y in self.mnist.valid_xy_batches():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
327 validation_error += self.classifier.errors(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
328 count += 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
329
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
330 return float(validation_error) / count
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
331
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
332 def pretrain(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
333 pretraining_epochs = 20
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
334
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
335 for layer_idx, update_fn in enumerate(self.classifier.pretrain_functions):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
336 for epoch in xrange(pretraining_epochs):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
337 # go through the training set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
338 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
339 for i, mb_x in enumerate(self.mnist.train_x_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
340 cost_acc += update_fn(mb_x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
341
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
342 if i % 100 == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
343 print i, "avg err = ", cost_acc / 100.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
344 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
345 print 'Pre-training layer %d, epoch %d' % (layer_idx, epoch)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
346
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
347 def finetune(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
348 max_training_epochs = 1000
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
349
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
350 n_train_batches = self.mnist.len_train / self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
351
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
352 # early-stopping parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
353 patience = 10000 # look as this many examples regardless
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
354 patience_increase = 2. # wait this much longer when a new best is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
355 # found
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
356 improvement_threshold = 0.995 # a relative improvement of this much is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
357 # considered significant
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
358 validation_frequency = min(n_train_batches, patience/2)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
359 # go through this many
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
360 # minibatche before checking the network
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
361 # on the validation set; in this case we
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
362 # check every epoch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
363
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
364
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
365 # TODO: use this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
366 best_params = None
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
367 best_validation_loss = float('inf')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
368 test_score = 0.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
369 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
370
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
371 done_looping = False
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
372 epoch = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
373
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
374 while (epoch < max_training_epochs) and (not done_looping):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
375 epoch = epoch + 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
376 for minibatch_index, (mb_x, mb_y) in enumerate(self.mnist.train_xy_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
377 cost_ij = classifier.finetune(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
378 iter = epoch * n_train_batches + minibatch_index
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
379
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
380 if (iter+1) % validation_frequency == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
381 this_validation_loss = self.compute_validation_error()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
382 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
383 (epoch, minibatch_index+1, n_train_batches, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
384 this_validation_loss*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
385
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
386 # if we got the best validation score until now
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
387 if this_validation_loss < best_validation_loss:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
388
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
389 #improve patience if loss improvement is good enough
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
390 if this_validation_loss < best_validation_loss * \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
391 improvement_threshold :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
392 patience = max(patience, iter * patience_increase)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
393 print "Improving patience"
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
394
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
395 # save best validation score and iteration number
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
396 best_validation_loss = this_validation_loss
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
397 best_iter = iter
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
398
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
399 # test it on the test set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
400 #test_losses = [test_model(i) for i in xrange(n_test_batches)]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
401 #test_score = numpy.mean(test_losses)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
402 #print((' epoch %i, minibatch %i/%i, test error of best '
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
403 # 'model %f %%') %
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
404 # (epoch, minibatch_index+1, n_train_batches,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
405 # test_score*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
406
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
407
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
408 if patience <= iter :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
409 done_looping = True
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
410 break
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
411
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
412 def train():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
413 driver = MnistTrainingDriver()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
414 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
415 driver.pretrain()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
416 print "PRETRAINING DONE. STARTING FINETUNING."
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
417 driver.finetune()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
418 end_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
419
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
420 if __name__ == '__main__':
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
421 train()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished.
fsavard
parents:
diff changeset
422