Mercurial > pylearn

annotate pylearn/algorithms/tests/test_mcRBM.py @ 1267:075c193afd1b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
refactoring mcRBM
author James Bergstra <bergstrj@iro.umontreal.ca>
date Fri, 03 Sep 2010 12:35:10 -0400
parents d4a14c6c36e0
children ba25c6e4f55d
rev   line source
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 from pylearn.algorithms.mcRBM import *
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 def test_reproduce_ranzato_hinton_2010(dataset='MAR', as_unittest=True):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 dataset='MAR'
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 n_vis=105
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 n_patches=10240
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 R,C= 16,16 # the size of image patches
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 n_vis=R*C
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11 n_patches=100000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13 n_train_iters=5000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 n_burnin_steps=10000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
17
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 l1_penalty=1e-3
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 no_l1_epochs = 10
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 effective_l1_penalty=0.0
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22 epoch_size=n_patches
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 batchsize = 128
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 lr = 0.075 / batchsize
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 s_lr = TT.scalar()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 n_K=256
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 n_J=100
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 def l2(X):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 return numpy.sqrt((X**2).sum())
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 tile = pylearn.dataset_ops.image_patches.save_filters_of_ranzato_hinton_2010
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 def tile(X, fname):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 _img = image_tiling.tile_raster_images(X,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 img_shape=(R,C),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 min_dynamic_range=1e-2)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 image_tiling.save_tiled_raster_images(_img, fname)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40 batch_idx = TT.iscalar()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43 train_batch = pylearn.dataset_ops.image_patches.ranzato_hinton_2010_op(batch_idx * batchsize + np.arange(batchsize))
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 train_batch = pylearn.dataset_ops.image_patches.image_patches(
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 dims = (n_patches,R,C),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 center=True,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 unitvar=True,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 dtype=floatX,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 rasterized=True)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 if not as_unittest:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 imgs_fn = function([batch_idx], outputs=train_batch)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
56 trainer = mcRBMTrainer.alloc(
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
57 mcRBM.alloc(n_I=n_vis, n_K=n_K, n_J=n_J),
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
58 train_batch,
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
59 batchsize, l1_penalty=TT.scalar())
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
60 rbm=trainer.rbm
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
61 smplr = trainer.sampler
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
62
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
63 grads = trainer.contrastive_grads(train_batch)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
64 learn_fn = function([batch_idx, trainer.l1_penalty],
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
65 outputs=[grads[0].norm(2), grads[0].norm(2), grads[1].norm(2)],
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
66 updates=trainer.cd_updates(train_batch))
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68 print "Learning..."
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69 last_epoch = -1
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70 for jj in xrange(n_train_iters):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 epoch = jj*batchsize / epoch_size
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 print_jj = epoch != last_epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 last_epoch = epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
76 if as_unittest and epoch == 5:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
77 U = rbm.U.value
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78 W = rbm.W.value
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79 def allclose(a,b):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80 return numpy.allclose(a,b,rtol=1.01,atol=1e-3)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81 print ""
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
82 print "--------------"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
83 print "assert allclose(l2(U), %f)"%l2(U)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
84 print "assert allclose(l2(W), %f)"%l2(W)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85 print "assert allclose(U.min(), %f)"%U.min()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 print "assert allclose(U.max(), %f)"%U.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 print "assert allclose(W.min(),%f)"%W.min()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88 print "assert allclose(W.max(), %f)"%W.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 print "--------------"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91 assert allclose(l2(U), 21.351664)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92 assert allclose(l2(W), 6.275828)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 assert allclose(U.min(), -1.176703)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94 assert allclose(U.max(), 0.859802)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 assert allclose(W.min(),-0.223128)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 assert allclose(W.max(), 0.227558 )
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98 break
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 if print_jj:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101 if not as_unittest:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 tile(imgs_fn(jj), "imgs_%06i.png"%jj)
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
103 tile(smplr.positions.value, "sample_%06i.png"%jj)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 tile(rbm.U.value.T, "U_%06i.png"%jj)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 tile(rbm.W.value.T, "W_%06i.png"%jj)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109 print 'l2(U)', l2(rbm.U.value),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 print 'l2(W)', l2(rbm.W.value)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 print 'U min max', rbm.U.value.min(), rbm.U.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 print 'W min max', rbm.W.value.min(), rbm.W.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 print 'a min max', rbm.a.value.min(), rbm.a.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115 print 'b min max', rbm.b.value.min(), rbm.b.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116 print 'c min max', rbm.c.value.min(), rbm.c.value.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
118 print 'parts min', smplr.positions.value.min(),
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
119 print 'max',smplr.positions.value.max(),
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120 print 'HMC step', smplr.stepsize,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121 print 'arate', smplr.avg_acceptance_rate
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
123
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
124 if 0:
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
125 # Continue HMC chain
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
126 smplr.simulate()
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
128 # Do CD update
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
129 l2_of_Ugrad = learn_fn(jj,
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
130 lr/max(1, jj/(20*epoch_size/batchsize)),
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
131 effective_l1_penalty)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
132
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
133 learn_fn(jj, effective_l1_penalty)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
134
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135 if print_jj:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 #print 'FE+', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 #print 'FE+[0]', float(l2_of_Ugrad[3]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 #print 'FE+[1]', float(l2_of_Ugrad[4]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 #print 'FE+[2]', float(l2_of_Ugrad[5]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 #print 'FE+[3]', float(l2_of_Ugrad[6])
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 if jj == no_l1_epochs * epoch_size/batchsize:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 print "Activating L1 weight decay"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 effective_l1_penalty = 1e-3
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149 # weird normalization technique...
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
150 # It constrains all the columns of the matrix to have the same length
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
151 if 0:
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
152 U = rbm.U.value
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
153 U_norms = np.sqrt((U*U).sum(axis=0))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
154 assert len(U_norms) == n_K
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
155 normVF = .95 * normVF + .05 * np.mean(U_norms)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
156 rbm.U.value = rbm.U.value * normVF/U_norms
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
157