Mercurial > pylearn

annotate pylearn/algorithms/exponential_mean.py @ 678:790c5e44906c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
small correction to exponential_mean
author James Bergstra <bergstrj@iro.umontreal.ca>
date Thu, 02 Apr 2009 17:57:51 -0400
parents 273d782b5a20
children e69249897f89
rev   line source
677
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """Modules for maintaining statistics based on exponential decay"""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 __docformat__ = "restructuredtext en"
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 import copy
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 import numpy
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 import theano
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 import theano.tensor
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 class ExponentialMean(theano.Module):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 """Maintain an exponentially-decaying estimate of the mean
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 This module computes the exact mean of the first `max_denom` values of `x`.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13 After the first `max_denom` values, it tracks the mean using the formula:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 :math:`self.curval = (1.0 - (1.0/max_denom)) * self.old_curval + (1.0/max_denom) * x`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 max_denom = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 """The average will be updated as if the current estimated average was estimated from at
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 most `max_denom-1` values."""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 ival = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 """The initial value for the estimated average."""
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 def __init__(self, x, max_denom, ival):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 :param x: track the mean of this Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 :param max_denom: see `self.max_denom`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 :param ival: This should be a tensor of zeros with a shape that matches `x`'s runtime
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 value.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 super(ExponentialMean, self).__init__()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 self.max_denom = max_denom
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 self.ival = ival
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 if len(ival.shape) == 0:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 x_type = theano.tensor.dscalar
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43 elif len(ival.shape) == 1:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 x_type = theano.tensor.dvector
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 elif len(ival.shape) == 2:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 x_type = theano.tensor.dmatrix
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 else:
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 #TODO: x_type = theano.tensor.TensorType(...)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 raise NotImplementedError()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 self.old_curval = (x_type())
678
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
52 # TODO: making this an lscalar caused different optimizations, followed by integer
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
53 # division somewhere were i wanted float division.... and the wrong answer.
790c5e44906c small correction to exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 677
diff changeset
54 self.denom = (theano.tensor.dscalar())
677
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 alpha = 1.0 / self.denom
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 self.curval = (1.0 - alpha) * self.old_curval + alpha * x
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 def updates(self):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 :returns: the symbolic updates necessary to refresh the mean estimate
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 :rtype: dict Variable -> Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64 return {
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 self.old_curval: self.curval,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66 self.denom: theano.tensor.smallest(self.denom + 1, self.max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67 }
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69 def _instance_initialize(self, obj):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70 obj.denom = 1
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 obj.old_curval = copy.copy(self.ival)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 def exp_mean(x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 """Return an `ExponentialMean` to track a Variable `x` with given shape
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
76 :type x: Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
77 :type x_shape: tuple
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78 :type max_denom: int
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80 :rtype: ExponentialMean instance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
82 return ExponentialMean(x,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
83 max_denom=max_denom,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
84 ival=numpy.zeros(x_shape))
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 def exp_mean_sqr(x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 """Return an `ExponentialMean` to track a Variable `x`'s square with given shape
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 :type x: Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90 :type x_shape: tuple
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91 :type max_denom: int
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 :rtype: ExponentialMean instance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 return ExponentialMean(x**2,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 max_denom=max_denom,
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97 ival=numpy.zeros(x_shape))
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 class exp_var(theano.Module):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 """Module with interface similar to `ExponentialMean` for tracking elementwise variance
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 mean = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 """`ExponentialMean`: current estimate of mean of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 mean_sqr = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 """`ExponentialMean`: current estimate of mean sqr of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 curval = None
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 """Variable: Current estimate of the variance of `x` """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 def __init__(self, x, x_shape, max_denom=100):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115 :param x: track the variance of this Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117 :param max_denom: see `self.max_denom`
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
119 :param ival: This should be a tensor of zeros with a shape that matches `x`'s runtime
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120 value.
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
123 super(exp_var,self).__init__()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 self.mean = exp_mean(x, x_shape, max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126 self.mean_sqr = exp_mean_sqr(x, x_shape, max_denom)
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
128 self.curval = self.mean_sqr.curval - self.mean.curval**2
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
129
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
130 def updates(self):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
131 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
132 :returns: the symbolic updates necessary to refresh the variance estimate
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
133 :rtype: dict Variable -> Variable
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
134 """
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135 rval = {}
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 rval.update(self.mean.updates())
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 rval.update(self.mean_sqr.updates())
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 return rval
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 def _instance_initialize(self, obj):
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 obj.mean.initialize()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 obj.mean_sqr.initialize()
273d782b5a20 added exponential_mean
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143