Mercurial > pylearn

annotate denoising_aa.py @ 227:17c5d080964b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
reinstating changeset 216 clobbered accidentally by 218
author James Bergstra <bergstrj@iro.umontreal.ca>
date Tue, 27 May 2008 13:23:05 -0400
parents df3fae88ab46
children 9e96fe8b955c
rev   line source
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 """
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 A denoising auto-encoder
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 """
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 import theano
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 from theano.formula import *
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 from learner import *
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 from theano import tensor as t
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9 from nnet_ops import *
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10 import math
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 from misc import *
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12 from theano.tensor_random import binomial
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14 def hiding_corruption_formula(seed,average_fraction_hidden):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15 """
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16 Return a formula for the corruption process, in which a random
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17 subset of the input numbers are hidden (mapped to 0).
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 @param seed: seed of the random generator
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 @type seed: anything that numpy.random.RandomState accepts
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 @param average_fraction_hidden: the probability with which each
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 input number is hidden (set to 0).
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 @type average_fraction_hidden: 0 <= real number <= 1
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 """
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 class HidingCorruptionFormula(Formulas):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 x = t.matrix()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 corrupted_x = x * binomial(seed,x,1,fraction_sampled)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 return HidingCorruptionFormula()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 def squash_affine_formula(squash_function=sigmoid):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 """
218
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
34 Simply does: squash_function(b + xW)
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 By convention prefix the parameters by _
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 """
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 class SquashAffineFormula(Formulas):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 x = t.matrix() # of dimensions minibatch_size x n_inputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 _b = t.row() # of dimensions 1 x n_outputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 _W = t.matrix() # of dimensions n_inputs x n_outputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 a = _b + t.dot(x,_W) # of dimensions minibatch_size x n_outputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 y = squash_function(a)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 return SquashAffineFormula()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 def gradient_descent_update_formula():
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46 class GradientDescentUpdateFormula(Formula):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 param = t.matrix()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 learning_rate = t.scalar()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 cost = t.column() # cost of each example in a minibatch
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 param_update = t.add_inplace(param, -learning_rate*t.sgrad(cost))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 return gradient_descent_update_formula()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 def probabilistic_classifier_loss_formula():
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 class ProbabilisticClassifierLossFormula(Formulas):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 a = t.matrix() # of dimensions minibatch_size x n_classes, pre-softmax output
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 target_class = t.ivector() # dimension (minibatch_size)
218
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
57 nll, probability_predictions = crossentropy_softmax_1hot(a, target_class) # defined in nnet_ops.py
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 return ProbabilisticClassifierLossFormula()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 def binomial_cross_entropy_formula():
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 class BinomialCrossEntropyFormula(Formulas):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 a = t.matrix() # pre-sigmoid activations, minibatch_size x dim
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 p = sigmoid(a) # model prediction
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64 q = t.matrix() # target binomial probabilities, minibatch_size x dim
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 # using the identity softplus(a) - softplus(-a) = a,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 # we obtain that q log(p) + (1-q) log(1-p) = q a - softplus(a)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 nll = -t.sum(q*a - softplus(-a))
218
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
68 # next line was missing... hope it's all correct above
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
69 return BinomialCrossEntropyFormula()
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 def squash_affine_autoencoder_formula(hidden_squash=t.tanh,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 reconstruction_squash=sigmoid,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73 share_weights=True,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 reconstruction_nll_formula=binomial_cross_entropy_formula(),
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 update_formula=gradient_descent_update_formula):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76 if share_weights:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 autoencoder = squash_affine_formula(hidden_squash).rename(a='code_a') + \
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 squash_affine_formula(reconstruction_squash).rename(x='hidden',y='reconstruction',_b='_c') + \
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 reconstruction_nll_formula
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 else:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 autoencoder = squash_affine_formula(hidden_squash).rename(a='code_a',_W='_W1') + \
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 squash_affine_formula(reconstruction_squash).rename(x='hidden',y='reconstruction',_b='_c',_W='_W2') + \
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 reconstruction_nll_formula
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 autoencoder = autoencoder + [update_formula().rename(cost = 'nll',
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85 param = p)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86 for p in autoencoder.get_all('_.*')]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 return autoencoder
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 # @todo: try other corruption formulae. The above is the default one.
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91 # not quite used in the ICML paper... (had a fixed number of 0s).
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
92
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
93 class DenoisingAutoEncoder(LearningAlgorithm):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
95 def __init__(self,n_inputs,n_hidden_per_layer,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 learning_rate=0.1,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 max_n_epochs=100,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
98 L1_regularizer=0,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
99 init_range=1.,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
100 corruption_formula = hiding_corruption_formula(),
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
101 autoencoder = squash_affine_autoencoder_formula(),
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
102 minibatch_size=None,linker = "c|py"):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 for name,val in locals().items():
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 if val is not self: self.__setattribute__(name,val)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 self.denoising_autoencoder_formula = corruption_formula + autoencoder.rename(x='corrupted_x')
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 def __call__(self, training_set=None):
218
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
108 """ Allocate and optionnaly train a model"""
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
109 model = DenoisingAutoEncoderModel(self)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110 if training_set:
218
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
111 print 'DenoisingAutoEncoder(): what do I do if training_set????'
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
112 # copied from mlp_factory_approach:
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
113 if len(trainset) == sys.maxint:
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
114 raise NotImplementedError('Learning from infinite streams is not supported')
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
115 nval = int(self.validation_portion * len(trainset))
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
116 nmin = len(trainset) - nval
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
117 assert nmin >= 0
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
118 minset = trainset[:nmin] #real training set for minimizing loss
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
119 valset = trainset[nmin:] #validation set for early stopping
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
120 best = model
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
121 for stp in self.early_stopper():
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
122 model.update(
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
123 minset.minibatches([input, target], minibatch_size=min(32,
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
124 len(trainset))))
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
125 #print 'mlp.__call__(), we did an update'
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
126 if stp.set_score:
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
127 stp.score = model(valset, ['loss_01'])
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
128 if (stp.score < stp.best_score):
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
129 best = copy.copy(model)
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
130 model = best
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
131 # end of the copy from mlp_factory_approach
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
132
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
133 return model
df3fae88ab46 small debugging
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 211
diff changeset
134
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 def compile(self, inputs, outputs):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 return theano.function(inputs,outputs,unpack_single=False,linker=self.linker)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
138
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
139 class DenoisingAutoEncoderModel(LearnerModel):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 def __init__(self,learning_algorithm,params):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 self.learning_algorithm=learning_algorithm
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
142 self.params=params
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143 v = learning_algorithm.v
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 self.update_fn = learning_algorithm.compile(learning_algorithm.denoising_autoencoder_formula.inputs,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
145 learning_algorithm.denoising_autoencoder_formula.outputs)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
146
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
147 def update(self, training_set, train_stats_collector=None):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
148
211
bd728c83faff in __get__, problem if the i.stop was None, i being the slice, added one line replacing None by the len(self)
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 210
diff changeset
149 print 'dont update you crazy frog!'
210
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
150
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
151 # old stuff
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
152
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 # self._learning_rate = t.scalar('learning_rate') # this is the symbol
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
154 # self.L1_regularizer = L1_regularizer
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
155 # self._L1_regularizer = t.scalar('L1_regularizer')
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
156 # self._input = t.matrix('input') # n_examples x n_inputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
157 # self._W = t.matrix('W')
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
158 # self._b = t.row('b')
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
159 # self._c = t.row('b')
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
160 # self._regularization_term = self._L1_regularizer * t.sum(t.abs(self._W))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
161 # self._corrupted_input = corruption_process(self._input)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162 # self._hidden = t.tanh(self._b + t.dot(self._input, self._W.T))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163 # self._reconstruction_activations =self._c+t.dot(self._hidden,self._W)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 # self._nll,self._output = crossentropy_softmax_1hot(Print("output_activations")(self._output_activations),self._target_vector)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 # self._output_class = t.argmax(self._output,1)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
166 # self._class_error = t.neq(self._output_class,self._target_vector)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 # self._minibatch_criterion = self._nll + self._regularization_term / t.shape(self._input)[0]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
168 # OnlineGradientTLearner.__init__(self)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
170 # def attributeNames(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 # return ["parameters","b1","W2","b2","W2", "L2_regularizer","regularization_term"]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
172
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 # def parameterAttributes(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174 # return ["b1","W1", "b2", "W2"]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 # def updateMinibatchInputFields(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177 # return ["input","target"]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 # def updateEndOutputAttributes(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 # return ["regularization_term"]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 # def lossAttribute(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 # return "minibatch_criterion"
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
184
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185 # def defaultOutputFields(self, input_fields):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186 # output_fields = ["output", "output_class",]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187 # if "target" in input_fields:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 # output_fields += ["class_error", "nll"]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
189 # return output_fields
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
191 # def allocate(self,minibatch):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
192 # minibatch_n_inputs = minibatch["input"].shape[1]
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193 # if not self._n_inputs:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 # self._n_inputs = minibatch_n_inputs
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
195 # self.b1 = numpy.zeros((1,self._n_hidden))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
196 # self.b2 = numpy.zeros((1,self._n_outputs))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 # self.forget()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 # elif self._n_inputs!=minibatch_n_inputs:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199 # # if the input changes dimension on the fly, we resize and forget everything
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
200 # self.forget()
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
201
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
202 # def forget(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
203 # if self._n_inputs:
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204 # r = self._init_range/math.sqrt(self._n_inputs)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 # self.W1 = numpy.random.uniform(low=-r,high=r,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 # size=(self._n_hidden,self._n_inputs))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 # r = self._init_range/math.sqrt(self._n_hidden)
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 # self.W2 = numpy.random.uniform(low=-r,high=r,
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
209 # size=(self._n_outputs,self._n_hidden))
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
210 # self.b1[:]=0
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 # self.b2[:]=0
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212 # self._n_epochs=0
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
213
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
214 # def isLastEpoch(self):
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 # self._n_epochs +=1
ffd50efefb70 work in progress denoising auto-encoder
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 # return self._n_epochs>=self._max_n_epochs