Mercurial > pylearn

annotate sandbox/rbm/model.py @ 396:e0c9357456e0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Bug fixed in RBM
author Joseph Turian <turian@gmail.com>
date Tue, 08 Jul 2008 20:37:36 -0400
parents 70019965f888
children 6e55ccb7e2bf
rev   line source
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
1 """
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
2 The model for an autoassociator for sparse inputs, using Ronan Collobert + Jason
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
3 Weston's sampling trick (2008).
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
4 """
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
5
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
6 import parameters
372
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
7
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
8 import globals
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
9 from globals import LR
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
10
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
11 import numpy
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
12 from numpy import dot
372
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
13 import random
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
14 random.seed(globals.SEED)
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
15
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
16 import pylearn.nnet_ops
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
17
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
18 def sigmoid(v):
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
19 # if x < -30.0: return 0.0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
20 # if x > 30.0: return 1.0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
21 return 1.0 / (1.0 + numpy.exp(-v))
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
22
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
23 def sample(v):
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
24 assert len(v.shape) == 2
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
25 x = numpy.zeros(v.shape)
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
26 for j in range(v.shape[0]):
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
27 for i in range(v.shape[1]):
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
28 assert v[j][i] >= 0 and v[j][i] <= 1
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
29 if random.random() < v[j][i]: x[j][i] = 1
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
30 else: x[j][i] = 0
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
31 return x
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
32
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
33 class Model:
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
34 def __init__(self):
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
35 self.parameters = parameters.Parameters(randomly_initialize=True)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
36
372
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
37 def update(self, instance):
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
38 """
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
39 Update the L{Model} using one training instance.
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
40 @param instance: A dict from feature index to (non-zero) value.
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
41 @todo: Should assert that nonzero_indices and zero_indices
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
42 are correct (i.e. are truly nonzero/zero).
75bab24bb2d8 Moved more logic into model.py
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
43 """
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
44 v0 = numpy.zeros((1, globals.INPUT_DIMENSION))
386
a474341861fa Added a simple AA
Joseph Turian <turian@gmail.com>
parents: 373
diff changeset
45 for idx in instance.keys():
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
46 v0[0][idx] = instance[idx]
386
a474341861fa Added a simple AA
Joseph Turian <turian@gmail.com>
parents: 373
diff changeset
47
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
48 q0 = sigmoid(self.parameters.b + dot(v0, self.parameters.w))
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
49 h0 = sample(q0)
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
50 p0 = sigmoid(self.parameters.c + dot(h0, self.parameters.w.T))
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
51 v1 = sample(p0)
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
52 q1 = sigmoid(self.parameters.b + dot(v1, self.parameters.w))
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
53 print
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
54 print "v[0]:", v0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
55 print "Q(h[0][i] = 1 | v[0]):", q0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
56 print "h[0]:", h0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
57 print "P(v[1][j] = 1 | h[0]):", p0
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
58 print "v[1]:", v1
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
59 print "Q(h[1][i] = 1 | v[1]):", q1
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
60
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
61 print self.parameters.w.shape
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
62 self.parameters.w += LR * (dot(v0.T, h0) - dot(v1.T, q1))
395
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
63 self.parameters.b += LR * (h0 - q1)
70019965f888 Basic, broken RBM implementation
Joseph Turian <turian@gmail.com>
parents: 393
diff changeset
64 self.parameters.c += LR * (v0 - v1)
396
e0c9357456e0 Bug fixed in RBM
Joseph Turian <turian@gmail.com>
parents: 395
diff changeset
65 print self.parameters