ift6266: scripts/stacked

annotate scripts/stacked_dae.py @ 115:b84a0d009af8

changes on pipeline mecanism: we now sample a different complexity for each transformations, this because when we use the same sampled complexity for all the modules 1/8 of the time we are close to 0 and we obtain an image very close to the source, we now save a complexity for each module in the parameters array

author	Xavier Glorot <glorotxa@iro.umontreal.ca>
date	Wed, 17 Feb 2010 16:20:15 -0500
parents	0b4080394f2c
children	4f37755d301b

rev	line source
114 0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	1 #!/usr/bin/python
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	2 # coding: utf-8
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	3
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	4 # Code for stacked denoising autoencoder
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	5 # Tests with MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	6 # TODO: adapt for NIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	7 # Based almost entirely on deeplearning.net tutorial, modifications by
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	8 # François Savard
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	9
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	10 # Base LogisticRegression, SigmoidalLayer, dA, SdA code taken
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	11 # from the deeplearning.net tutorial. Refactored a bit.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	12 # Changes (mainly):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	13 # - splitted initialization in smaller methods
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	14 # - removed the "givens" thing involving an index in the whole dataset
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	15 # (to allow flexibility in how data is inputted... not necessarily one big tensor)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	16 # - changed the "driver" a lot, altough for the moment the same logic is used
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	17
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	18 import time
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	19 import theano
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	20 import theano.tensor as T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	21 import theano.tensor.nnet
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	22 from theano.tensor.shared_randomstreams import RandomStreams
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	23 import numpy, numpy.random
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	24
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	25 from pylearn.datasets import MNIST
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	26
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	27
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	28 # from pylearn codebase
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	29 def update_locals(obj, dct):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	30 if 'self' in dct:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	31 del dct['self']
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	32 obj.__dict__.update(dct)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	33
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	34
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	35 class LogisticRegression(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	36 def __init__(self, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	37 # initialize with 0 the weights W as a matrix of shape (n_in, n_out)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	38 self.W = theano.shared(value=numpy.zeros((n_in,n_out), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	39 name='W')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	40 # initialize the baises b as a vector of n_out 0s
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	41 self.b = theano.shared(value=numpy.zeros((n_out,), dtype = theano.config.floatX),
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	42 name='b')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	43
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	44 # compute vector of class-membership probabilities in symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	45 self.p_y_given_x = T.nnet.softmax(T.dot(input, self.W)+self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	46
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	47 # compute prediction as class whose probability is maximal in
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	48 # symbolic form
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	49 self.y_pred=T.argmax(self.p_y_given_x, axis=1)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	50
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	51 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	52
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	53 def negative_log_likelihood(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	54 return -T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]),y])
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	55
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	56 def errors(self, y):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	57 # check if y has same dimension of y_pred
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	58 if y.ndim != self.y_pred.ndim:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	59 raise TypeError('y should have the same shape as self.y_pred',
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	60 ('y', target.type, 'y_pred', self.y_pred.type))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	61 # check if y is of the correct datatype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	62 if y.dtype.startswith('int'):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	63 # the T.neq operator returns a vector of 0s and 1s, where 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	64 # represents a mistake in prediction
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	65 return T.mean(T.neq(self.y_pred, y))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	66 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	67 raise NotImplementedError()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	68
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	69
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	70 class SigmoidalLayer(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	71 def __init__(self, rng, input, n_in, n_out):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	72 self.input = input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	73
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	74 W_values = numpy.asarray( rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	75 low = -numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	76 high = numpy.sqrt(6./(n_in+n_out)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	77 size = (n_in, n_out)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	78 self.W = theano.shared(value = W_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	79
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	80 b_values = numpy.zeros((n_out,), dtype= theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	81 self.b = theano.shared(value= b_values)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	82
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	83 self.output = T.nnet.sigmoid(T.dot(input, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	84 self.params = [self.W, self.b]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	85
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	86
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	87 class dA(object):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	88 def __init__(self, n_visible= 784, n_hidden= 500, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	89 corruption_level = 0.1, input = None, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	90 shared_W = None, shared_b = None):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	91 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	92
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	93 self.init_randomizer()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	94 self.init_params()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	95 self.init_functions()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	96
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	97 def init_randomizer(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	98 # create a Theano random generator that gives symbolic random values
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	99 self.theano_rng = RandomStreams()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	100 # create a numpy random generator
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	101 self.numpy_rng = numpy.random.RandomState()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	102
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	103 def init_params(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	104 if self.shared_W != None and self.shared_b != None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	105 self.W = self.shared_W
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	106 self.b = self.shared_b
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	107 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	108 # initial values for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	109 # note : W' was written as `W_prime` and b' as `b_prime`
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	110
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	111 # W is initialized with `initial_W` which is uniformely sampled
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	112 # from -6./sqrt(n_visible+n_hidden) and 6./sqrt(n_hidden+n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	113 # the output of uniform if converted using asarray to dtype
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	114 # theano.config.floatX so that the code is runable on GPU
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	115 initial_W = numpy.asarray( self.numpy_rng.uniform( \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	116 low = -numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	117 high = numpy.sqrt(6./(n_hidden+n_visible)), \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	118 size = (n_visible, n_hidden)), dtype = theano.config.floatX)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	119 initial_b = numpy.zeros(n_hidden)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	120
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	121 # theano shared variables for weights and biases
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	122 self.W = theano.shared(value = initial_W, name = "W")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	123 self.b = theano.shared(value = initial_b, name = "b")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	124
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	125 initial_b_prime= numpy.zeros(self.n_visible)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	126 # tied weights, therefore W_prime is W transpose
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	127 self.W_prime = self.W.T
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	128 self.b_prime = theano.shared(value = initial_b_prime, name = "b'")
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	129
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	130 def init_functions(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	131 # if no input is given, generate a variable representing the input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	132 if self.input == None :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	133 # we use a matrix because we expect a minibatch of several examples,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	134 # each example being a row
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	135 self.x = T.dmatrix(name = 'input')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	136 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	137 self.x = self.input
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	138
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	139 # keep 90% of the inputs the same and zero-out randomly selected subset of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	140 # 10% of the inputs
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	141 # note : first argument of theano.rng.binomial is the shape(size) of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	142 # random numbers that it should produce
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	143 # second argument is the number of trials
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	144 # third argument is the probability of success of any trial
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	145 #
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	146 # this will produce an array of 0s and 1s where 1 has a
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	147 # probability of 1 - ``corruption_level`` and 0 with
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	148 # ``corruption_level``
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	149 self.tilde_x = self.theano_rng.binomial(self.x.shape, 1, 1-self.corruption_level) * self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	150 # using tied weights
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	151 self.y = T.nnet.sigmoid(T.dot(self.tilde_x, self.W) + self.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	152 self.z = T.nnet.sigmoid(T.dot(self.y, self.W_prime) + self.b_prime)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	153 self.L = - T.sum( self.xT.log(self.z) + (1-self.x)T.log(1-self.z), axis=1 )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	154 # note : L is now a vector, where each element is the cross-entropy cost
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	155 # of the reconstruction of the corresponding example of the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	156 # minibatch. We need to compute the average of all these to get
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	157 # the cost of the minibatch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	158 self.cost = T.mean(self.L)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	159
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	160 self.params = [ self.W, self.b, self.b_prime ]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	161
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	162 class SdA():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	163 def __init__(self, batch_size, n_ins,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	164 hidden_layers_sizes, n_outs,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	165 corruption_levels, rng, pretrain_lr, finetune_lr):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	166 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	167
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	168 self.layers = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	169 self.pretrain_functions = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	170 self.params = []
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	171 self.n_layers = len(hidden_layers_sizes)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	172
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	173 if len(hidden_layers_sizes) < 1 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	174 raiseException (' You must have at least one hidden layer ')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	175
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	176 # allocate symbolic variables for the data
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	177 self.x = T.matrix('x') # the data is presented as rasterized images
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	178 self.y = T.ivector('y') # the labels are presented as 1D vector of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	179 # [int] labels
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	180
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	181 self.create_layers()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	182 self.init_finetuning()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	183
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	184 def create_layers(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	185 for i in xrange( self.n_layers ):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	186 # construct the sigmoidal layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	187
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	188 # the size of the input is either the number of hidden units of
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	189 # the layer below or the input size if we are on the first layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	190 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	191 input_size = self.n_ins
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	192 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	193 input_size = self.hidden_layers_sizes[i-1]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	194
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	195 # the input to this layer is either the activation of the hidden
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	196 # layer below or the input of the SdA if you are on the first
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	197 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	198 if i == 0 :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	199 layer_input = self.x
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	200 else:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	201 layer_input = self.layers[-1].output
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	202
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	203 layer = SigmoidalLayer(self.rng, layer_input, input_size,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	204 self.hidden_layers_sizes[i] )
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	205 # add the layer to the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	206 self.layers += [layer]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	207 self.params += layer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	208
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	209 # Construct a denoising autoencoder that shared weights with this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	210 # layer
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	211 dA_layer = dA(input_size, self.hidden_layers_sizes[i], \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	212 corruption_level = self.corruption_levels[0],\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	213 input = layer_input, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	214 shared_W = layer.W, shared_b = layer.b)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	215
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	216 self.init_updates_for_layer(dA_layer)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	217
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	218 def init_updates_for_layer(self, dA_layer):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	219 # Construct a function that trains this dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	220 # compute gradients of layer parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	221 gparams = T.grad(dA_layer.cost, dA_layer.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	222 # compute the list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	223 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	224 for param, gparam in zip(dA_layer.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	225 updates[param] = param - gparam * self.pretrain_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	226
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	227 # create a function that trains the dA
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	228 update_fn = theano.function([self.x], dA_layer.cost, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	229 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	230
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	231 # collect this function into a list
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	232 self.pretrain_functions += [update_fn]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	233
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	234 def init_finetuning(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	235 # We now need to add a logistic layer on top of the MLP
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	236 self.logLayer = LogisticRegression(\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	237 input = self.layers[-1].output,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	238 n_in = self.hidden_layers_sizes[-1], n_out = self.n_outs)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	239
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	240 self.params += self.logLayer.params
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	241 # construct a function that implements one step of finetunining
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	242
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	243 # compute the cost, defined as the negative log likelihood
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	244 cost = self.logLayer.negative_log_likelihood(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	245 # compute the gradients with respect to the model parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	246 gparams = T.grad(cost, self.params)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	247 # compute list of updates
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	248 updates = {}
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	249 for param,gparam in zip(self.params, gparams):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	250 updates[param] = param - gparam*self.finetune_lr
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	251
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	252 self.finetune = theano.function([self.x, self.y], cost,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	253 updates = updates)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	254
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	255 # symbolic variable that points to the number of errors made on the
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	256 # minibatch given by self.x and self.y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	257
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	258 self.errors = self.logLayer.errors(self.y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	259
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	260 class MnistIterators:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	261 def __init__(self, minibatch_size):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	262 self.minibatch_size = minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	263
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	264 self.mnist = MNIST.first_1k()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	265
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	266 self.len_train = len(self.mnist.train.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	267 self.len_valid = len(self.mnist.valid.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	268 self.len_test = len(self.mnist.test.x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	269
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	270 def train_x_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	271 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	272 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	273 yield self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	274 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	275
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	276 def train_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	277 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	278 while idx < len(self.mnist.train.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	279 mb_x = self.mnist.train.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	280 mb_y = self.mnist.train.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	281 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	282 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	283
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	284 def valid_xy_batches(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	285 idx = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	286 while idx < len(self.mnist.valid.x):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	287 mb_x = self.mnist.valid.x[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	288 mb_y = self.mnist.valid.y[idx:idx+self.minibatch_size]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	289 yield mb_x, mb_y
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	290 idx += self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	291
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	292
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	293 class MnistTrainingDriver:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	294 def __init__(self, rng=numpy.random):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	295 self.rng = rng
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	296
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	297 self.init_SdA()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	298
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	299 def init_SdA(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	300 # Hyperparam
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	301 hidden_layers_sizes = [1000, 1000, 1000]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	302 n_outs = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	303 corruption_levels = [0.2, 0.2, 0.2]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	304 minibatch_size = 10
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	305 pretrain_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	306 finetune_lr = 0.001
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	307
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	308 update_locals(self, locals())
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	309
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	310 self.mnist = MnistIterators(minibatch_size)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	311
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	312 # construct the stacked denoising autoencoder class
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	313 self.classifier = SdA( batch_size = minibatch_size, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	314 n_ins=28*28, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	315 hidden_layers_sizes = hidden_layers_sizes, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	316 n_outs=n_outs, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	317 corruption_levels = corruption_levels,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	318 rng = self.rng,\
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	319 pretrain_lr = pretrain_lr, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	320 finetune_lr = finetune_lr)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	321
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	322 def compute_validation_error(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	323 validation_error = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	324
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	325 count = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	326 for mb_x, mb_y in self.mnist.valid_xy_batches():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	327 validation_error += self.classifier.errors(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	328 count += 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	329
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	330 return float(validation_error) / count
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	331
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	332 def pretrain(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	333 pretraining_epochs = 20
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	334
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	335 for layer_idx, update_fn in enumerate(self.classifier.pretrain_functions):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	336 for epoch in xrange(pretraining_epochs):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	337 # go through the training set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	338 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	339 for i, mb_x in enumerate(self.mnist.train_x_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	340 cost_acc += update_fn(mb_x)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	341
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	342 if i % 100 == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	343 print i, "avg err = ", cost_acc / 100.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	344 cost_acc = 0.0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	345 print 'Pre-training layer %d, epoch %d' % (layer_idx, epoch)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	346
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	347 def finetune(self):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	348 max_training_epochs = 1000
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	349
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	350 n_train_batches = self.mnist.len_train / self.minibatch_size
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	351
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	352 # early-stopping parameters
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	353 patience = 10000 # look as this many examples regardless
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	354 patience_increase = 2. # wait this much longer when a new best is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	355 # found
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	356 improvement_threshold = 0.995 # a relative improvement of this much is
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	357 # considered significant
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	358 validation_frequency = min(n_train_batches, patience/2)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	359 # go through this many
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	360 # minibatche before checking the network
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	361 # on the validation set; in this case we
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	362 # check every epoch
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	363
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	364
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	365 # TODO: use this
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	366 best_params = None
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	367 best_validation_loss = float('inf')
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	368 test_score = 0.
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	369 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	370
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	371 done_looping = False
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	372 epoch = 0
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	373
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	374 while (epoch < max_training_epochs) and (not done_looping):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	375 epoch = epoch + 1
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	376 for minibatch_index, (mb_x, mb_y) in enumerate(self.mnist.train_xy_batches()):
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	377 cost_ij = classifier.finetune(mb_x, mb_y)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	378 iter = epoch * n_train_batches + minibatch_index
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	379
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	380 if (iter+1) % validation_frequency == 0:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	381 this_validation_loss = self.compute_validation_error()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	382 print('epoch %i, minibatch %i/%i, validation error %f %%' % \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	383 (epoch, minibatch_index+1, n_train_batches, \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	384 this_validation_loss*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	385
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	386 # if we got the best validation score until now
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	387 if this_validation_loss < best_validation_loss:
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	388
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	389 #improve patience if loss improvement is good enough
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	390 if this_validation_loss < best_validation_loss * \
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	391 improvement_threshold :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	392 patience = max(patience, iter * patience_increase)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	393 print "Improving patience"
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	394
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	395 # save best validation score and iteration number
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	396 best_validation_loss = this_validation_loss
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	397 best_iter = iter
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	398
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	399 # test it on the test set
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	400 #test_losses = [test_model(i) for i in xrange(n_test_batches)]
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	401 #test_score = numpy.mean(test_losses)
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	402 #print((' epoch %i, minibatch %i/%i, test error of best '
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	403 # 'model %f %%') %
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	404 # (epoch, minibatch_index+1, n_train_batches,
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	405 # test_score*100.))
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	406
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	407
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	408 if patience <= iter :
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	409 done_looping = True
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	410 break
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	411
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	412 def train():
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	413 driver = MnistTrainingDriver()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	414 start_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	415 driver.pretrain()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	416 print "PRETRAINING DONE. STARTING FINETUNING."
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	417 driver.finetune()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	418 end_time = time.clock()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	419
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	420 if __name__ == '__main__':
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	421 train()
0b4080394f2c Added stacked DAE code for my experiments, based on tutorial code. Quite unfinished. fsavard parents: diff changeset	422

Mercurial > ift6266

annotate scripts/stacked_dae.py @ 115:b84a0d009af8