Mercurial > pylearn

annotate pylearn/algorithms/exponential_mean.py @ 1499:f82b80c841b2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Remove deprecation warning.
author Frederic Bastien <nouiz@nouiz.org>
date Fri, 09 Sep 2011 10:49:54 -0400
parents b4aa46f856c1
children
rev   line source
677
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """Modules for maintaining statistics based on exponential decay"""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 __docformat__ = "restructuredtext en"
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 import copy
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 import numpy
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 import theano
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 import theano.tensor
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 class ExponentialMean(theano.Module):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 """Maintain an exponentially-decaying estimate of the mean
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 This module computes the exact mean of the first `max_denom` values of `x`.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13 After the first `max_denom` values, it tracks the mean using the formula:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 :math:`self.curval = (1.0 - (1.0/max_denom)) * self.old_curval + (1.0/max_denom) * x`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
750
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
17
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
18 The symbolic buffer containing the running mean is called `old_curval`. (This has a value
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
19 in the ModuleInstance).
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
20
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
21 The symbolic variable for the updated running mean is called `curval`.
b4aa46f856c1 added doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 691
diff changeset
22
677
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 max_denom = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 """The average will be updated as if the current estimated average was estimated from at
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 most `max_denom-1` values."""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 ival = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 """The initial value for the estimated average."""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 def __init__(self, x, max_denom, ival):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 :param x: track the mean of this Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 :param max_denom: see `self.max_denom`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 :param ival: This should be a tensor of zeros with a shape that matches `x`'s runtime
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 value.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 super(ExponentialMean, self).__init__()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 self.max_denom = max_denom
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 self.ival = ival
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 if len(ival.shape) == 0:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 x_type = theano.tensor.dscalar
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 elif len(ival.shape) == 1:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 x_type = theano.tensor.dvector
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 elif len(ival.shape) == 2:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 x_type = theano.tensor.dmatrix
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 else:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 #TODO: x_type = theano.tensor.TensorType(...)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 raise NotImplementedError()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 self.old_curval = (x_type())
678
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
58 # TODO: making this an lscalar caused different optimizations, followed by integer
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
59 # division somewhere were i wanted float division.... and the wrong answer.
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
60 self.denom = (theano.tensor.dscalar())
677
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 alpha = 1.0 / self.denom
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 self.curval = (1.0 - alpha) * self.old_curval + alpha * x
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 def updates(self):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67 :returns: the symbolic updates necessary to refresh the mean estimate
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68 :rtype: dict Variable -> Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70 return {
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 self.old_curval: self.curval,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72 self.denom: theano.tensor.smallest(self.denom + 1, self.max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 }
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75 def _instance_initialize(self, obj):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
76 obj.denom = 1
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
77 obj.old_curval = copy.copy(self.ival)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79 def exp_mean(x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80 """Return an `ExponentialMean` to track a Variable `x` with given shape
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
82 :type x: Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
83 :type x_shape: tuple
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
84 :type max_denom: int
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 :rtype: ExponentialMean instance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88 return ExponentialMean(x,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 max_denom=max_denom,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90 ival=numpy.zeros(x_shape))
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92 def exp_mean_sqr(x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 """Return an `ExponentialMean` to track a Variable `x`'s square with given shape
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 :type x: Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 :type x_shape: tuple
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97 :type max_denom: int
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 :rtype: ExponentialMean instance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101 return ExponentialMean(x**2,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 max_denom=max_denom,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103 ival=numpy.zeros(x_shape))
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 class exp_var(theano.Module):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 """Module with interface similar to `ExponentialMean` for tracking elementwise variance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 mean = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 """`ExponentialMean`: current estimate of mean of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 mean_sqr = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 """`ExponentialMean`: current estimate of mean sqr of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116 curval = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117 """Variable: Current estimate of the variance of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
119 def __init__(self, x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121 :param x: track the variance of this Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
123 :param max_denom: see `self.max_denom`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 :param ival: This should be a tensor of zeros with a shape that matches `x`'s runtime
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126 value.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
128 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
129 super(exp_var,self).__init__()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
130
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
131 self.mean = exp_mean(x, x_shape, max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
132 self.mean_sqr = exp_mean_sqr(x, x_shape, max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
133
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
134 self.curval = self.mean_sqr.curval - self.mean.curval**2
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 def updates(self):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 :returns: the symbolic updates necessary to refresh the variance estimate
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 :rtype: dict Variable -> Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 rval = {}
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 rval.update(self.mean.updates())
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 rval.update(self.mean_sqr.updates())
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 return rval
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 def _instance_initialize(self, obj):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 obj.mean.initialize()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148 obj.mean_sqr.initialize()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149
691
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
150 class DynamicNormalizer(theano.Module):
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
151 """
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
152 Normalizes `input` using geometric-decaying estimates of the mean and variance. The
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
153 `output` should mean near zero, and variance near 1.
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
154 """
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
155 def __init__(self, input, input_shape, max_denom=100, eps=1.0e-8):
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
156 super(DynamicNormalizer, self).__init__()
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
157 self.input = input
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
158 self.d_mean = exp_mean(input, input_shape, max_denom=max_denom)
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
159 self.d_var = exp_var(input, input_shape, max_denom=max_denom)
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
160 self.output = (input - self.d_mean.curval) / theano.tensor.sqrt(self.d_var.curval+eps)
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
161
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
162 def updates(self):
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
163 rval = {}
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
164 rval.update(self.d_mean.updates())
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
165 rval.update(self.d_var.updates())
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
166 return rval
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
167
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
168 def _instance_initialize(self, obj):
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
169 obj.d_mean.initialize()
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
170 obj.d_var.initialize()
e69249897f89 added DynamicNormalizer to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 678
diff changeset
171