Mercurial > pylearn

annotate sandbox/weights.py @ 477:8ff412852d66

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added sgd
author James Bergstra <bergstrj@iro.umontreal.ca>
date Fri, 24 Oct 2008 13:28:00 -0400
parents 23221eefb70e
children 3daabc7f94ff
rev   line source
466
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
1 """
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
2 Routine to initialize weights.
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
3
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
4 @note: We assume that numpy.random.seed() has already been performed.
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
5 """
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
6
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
7 from math import sqrt
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
8 import numpy.random
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
9 def random_weights(nin, nout, scale_by=sqrt(3)):
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
10 """
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
11 Generate an initial weight matrix with nin inputs (rows) and nout
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
12 outputs (cols).
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
13 Each weight is chosen uniformly at random to be in range:
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
14 [-scale_by/sqrt(nin), +scale_by/sqrt(nin)]
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
15 @note: Play with scale_by!
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
16 Ronan derives scale_by=sqrt(3) because that gives variance of
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
17 1 to something (I forget, ask Yoshua for the derivation). However,
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
18 Ronan got better results by accidentally using scale_by=1. Yoshua
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
19 hypothesizes this is because the variance will get telescopically
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
20 smaller as we go up the layers [need more explanation of this
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
21 argument].
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
22 @note: Things may get even trickier if the same weights are being
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
23 shared in multiple places.
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
24 """
23221eefb70e Added pylearn.sandbox.weights.random_weights
Joseph Turian <turian@iro.umontreal.ca>
parents:
diff changeset
25 return (numpy.random.rand(nin, nout) * 2.0 - 1) * scale_by / sqrt(nin)