Mercurial > pylearn

annotate pylearn/gd/dbd.py @ 1422:8c209c847087

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
adding delta-bar-delta optimization updates to gd module
author James Bergstra <bergstrj@iro.umontreal.ca>
date Fri, 04 Feb 2011 16:07:27 -0500
parents
children
rev   line source
1422
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 Delta-Bar-Delta gradient descent algorithm.
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 Reference: TODO
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 Math: TODO
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 """
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8 import sys
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 import numpy
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 from theano import shared, tensor
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 def dbd_updates(params, grads, stepsizes, cost,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13 global_step_inc=1.1,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 global_step_dec=0.5,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 multiplier_min=.1,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16 multiplier_max=10,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 multiplier_inc=.05, #additive
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 multiplier_dec=.95, #multiplicative
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 ):
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 """
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 Parameter description TODO.
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 Return value TODO.
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 """
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 dtype = cost.dtype
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 if grads is None:
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 grads = tensor.grad(cost, params)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 paramvals = [p.get_value(borrow=False) for p in params]
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 last_params = [shared(numpy.asarray(pv)) for pv in paramvals]
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 last_grads = [shared(numpy.zeros_like(pv)) for pv in paramvals]
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 multipliers = [shared(numpy.ones_like(pv)) for pv in paramvals]
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 global_stepsize = shared(numpy.asarray(1.0, dtype=dtype))
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 #DebugMode complains by default about inf
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 last_cost = shared(numpy.asarray(sys.maxint, dtype=dtype))
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 ups = dict()
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40 cost_improvement = (cost < last_cost)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 ups[last_cost] = tensor.switch(cost_improvement, cost, last_cost)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 ups[global_stepsize] = new_gs = tensor.switch(cost_improvement,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43 global_step_inc*global_stepsize,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 global_step_dec*global_stepsize,)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 for lp, p in zip(last_params, params):
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 ups[lp] = tensor.switch(cost_improvement, p, lp)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 for lg, g in zip(last_grads, grads):
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 ups[lg] = tensor.switch(cost_improvement, g, lg)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 for m, lg, g in zip(multipliers, last_grads, grads):
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 ups[m] = tensor.switch(cost_improvement,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 tensor.clip(
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 tensor.switch(g*lg >= 0,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 m+multiplier_inc,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 m*multiplier_dec),
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 multiplier_min,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 multiplier_max),
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 m)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 for lp, p, lg, g, stepsize, m in zip(
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 last_params, params,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 last_grads, grads,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 stepsizes, multipliers):
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 ups[p] = tensor.switch(cost_improvement,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 p - new_gs*ups[m]*stepsize*g,
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64 lp - new_gs*ups[m]*stepsize*lg)
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 return ups
8c209c847087 adding delta-bar-delta optimization updates to gd module
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66