Mercurial > pylearn

annotate pylearn/algorithms/tests/test_mcRBM.py @ 1525:9c24a2bdbe90

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
a test is too slow when run in debugmode, so we force fast_run in that case.
author Frederic Bastien <nouiz@nouiz.org>
date Fri, 09 Nov 2012 14:48:26 -0500
parents 9d21919e2332
children
rev   line source
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
1 import sys
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
2
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
3 import numpy
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
4 import theano
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
5 from theano import tensor
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
6
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
7 from pylearn.algorithms.mcRBM import mcRBM, mcRBMTrainer, mcRBM_withP, l2
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
8 #import pylearn.datasets.cifar10
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
9 import pylearn.dataset_ops.cifar10
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
10 from pylearn.shared.layers.logreg import LogisticRegression
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
11 from pylearn.io import image_tiling
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
12 import pylearn.dataset_ops.image_patches
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
13
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
14
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
15 def _default_rbm_alloc(n_I, n_K=256, n_J=100):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
16 return mcRBM.alloc(n_I, n_K, n_J)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
17
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
18
1508
b28e8730c948 fix test.
Frederic Bastien <nouiz@nouiz.org>
parents: 1507
diff changeset
19 def _default_trainer_alloc(rbm, train_batch, batchsize, initial_lr_per_example,
b28e8730c948 fix test.
Frederic Bastien <nouiz@nouiz.org>
parents: 1507
diff changeset
20 l1_penalty, l1_penalty_start, persistent_chains):
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
21 return mcRBMTrainer.alloc(rbm, train_batch, batchsize,
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
22 l1_penalty=l1_penalty,
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
23 l1_penalty_start=l1_penalty_start,
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
24 persistent_chains=persistent_chains)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
25
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
26
1525
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
27 def test_reproduce_ranzato_hinton_2010():
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
28 # We need to modify the global mode as otherwise there is a
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
29 # function being compiler in the sample that won't use the mode we
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
30 # want.
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
31 try:
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
32 orig_mode = theano.config.mode
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
33 if theano.config.mode in ["DebugMode", "DEBUG_MODE"]:
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
34 theano.config.mode = "FAST_RUN"
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
35 reproduce_ranzato_hinton_2010()
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
36 finally:
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
37 theano.config.mode = orig_mode
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
38
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
39
9c24a2bdbe90 a test is too slow when run in debugmode, so we force fast_run in that case.
Frederic Bastien <nouiz@nouiz.org>
parents: 1524
diff changeset
40 def reproduce_ranzato_hinton_2010(
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
41 dataset='MAR', as_unittest=True, n_train_iters=5000,
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
42 rbm_alloc=_default_rbm_alloc, trainer_alloc=_default_trainer_alloc,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
43 lr_per_example=.075,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
44 l1_penalty=1e-3,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
45 l1_penalty_start=1000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
46 persistent_chains=True,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
47 ):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
48
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
49 batchsize = 128
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
50
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 if dataset == 'MAR':
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
52 n_vis = 105
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
53 n_patches = 10240
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
54 epoch_size = n_patches
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
55 elif dataset == 'cifar10patches8x8':
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
56 R, C = 8, 8 # the size of image patches
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
57 n_vis = 96 # pca components
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
58 epoch_size = batchsize * 500
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
59 n_patches = epoch_size * 20
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
60 elif dataset == 'tinyimages_patches':
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
61 R, C = 8, 8
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
62 n_vis = 81
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
63 epoch_size = batchsize * 500
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
64 n_patches = epoch_size * 20
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 else:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
66 R, C = 16, 16 # the size of image patches
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
67 n_vis = R * C
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
68 n_patches = 100000
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
69 epoch_size = n_patches
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 def l2(X):
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
72 return numpy.sqrt((X ** 2).sum())
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
73
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75 tile = pylearn.dataset_ops.image_patches.save_filters_of_ranzato_hinton_2010
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
76 elif dataset == 'cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
77 def tile(X, fname):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
78 _img = pylearn.datasets.cifar10.tile_rasterized_examples(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
79 pylearn.preprocessing.pca.pca_whiten_inverse(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
80 pylearn.dataset_ops.cifar10.random_cifar_patches_pca(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
81 n_vis, None, 'float32', n_patches, R, C,),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
82 X),
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
83 img_shape=(R, C))
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
84 image_tiling.save_tiled_raster_images(_img, fname)
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
85 elif dataset == 'tinyimages_patches':
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
86 tile = pylearn.dataset_ops.tinyimages.save_filters
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88 def tile(X, fname):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 _img = image_tiling.tile_raster_images(X,
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
90 img_shape=(R, C),
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91 min_dynamic_range=1e-2)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92 image_tiling.save_tiled_raster_images(_img, fname)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
94 batch_idx = tensor.iscalar()
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
95 batch_range = batch_idx * batchsize + numpy.arange(batchsize)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97 if dataset == 'MAR':
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
98 train_batch = pylearn.dataset_ops.image_patches.ranzato_hinton_2010_op(batch_range)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
99 elif dataset == 'cifar10patches8x8':
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
100 train_batch = pylearn.dataset_ops.cifar10.cifar10_patches(
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
101 batch_range, 'train', n_patches=n_patches, patch_size=(R, C),
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
102 pca_components=n_vis)
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
103 elif dataset == 'tinyimages_patches':
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
104 train_batch = pylearn.dataset_ops.tinyimages.tinydataset_op(batch_range)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 train_batch = pylearn.dataset_ops.image_patches.image_patches(
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
107 s_idx=(batch_idx * batchsize + numpy.arange(batchsize)),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
108 dims=(n_patches, R, C),
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109 center=True,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 unitvar=True,
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
111 dtype=theano.config.floatX,
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 rasterized=True)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 if not as_unittest:
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
115 imgs_fn = theano.function([batch_idx], outputs=train_batch)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
117 trainer = trainer_alloc(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
118 rbm_alloc(n_I=n_vis),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
119 train_batch,
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
120 batchsize,
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
121 initial_lr_per_example=lr_per_example,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
122 l1_penalty=l1_penalty,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
123 l1_penalty_start=l1_penalty_start,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
124 persistent_chains=persistent_chains)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
125 rbm = trainer.rbm
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
126
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
127 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
128 grads = trainer.contrastive_grads()
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
129 learn_fn = theano.function([batch_idx],
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
130 outputs=[grads[0].norm(2), grads[0].norm(2), grads[1].norm(2)],
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
131 updates=trainer.cd_updates())
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
132 else:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
133 learn_fn = theano.function([batch_idx], outputs=[],
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
134 updates=trainer.cd_updates())
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
135
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
136 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
137 smplr = trainer.sampler
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
138 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
139 smplr = trainer._last_cd1_sampler
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
140
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
141 if dataset == 'cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
142 cPickle.dump(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
143 pylearn.dataset_ops.cifar10.random_cifar_patches_pca(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
144 n_vis, None, 'float32', n_patches, R, C,),
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
145 open('test_mcRBM.pca.pkl', 'w'))
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 print "Learning..."
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148 last_epoch = -1
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149 for jj in xrange(n_train_iters):
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
150 epoch = jj * batchsize / epoch_size
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
151
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152 print_jj = epoch != last_epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 last_epoch = epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 if as_unittest and epoch == 5:
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
156 U = rbm.U.get_value(borrow=True)
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
157 W = rbm.W.get_value(borrow=True)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
158
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
159 def allclose(a, b):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
160 return numpy.allclose(a, b, rtol=1.01, atol=1e-3)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
161 print ""
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162 print "--------------"
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
163 print "assert allclose(l2(U), %f)" % l2(U)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
164 print "assert allclose(l2(W), %f)" % l2(W)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
165 print "assert allclose(U.min(), %f)" % U.min()
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
166 print "assert allclose(U.max(), %f)" % U.max()
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
167 print "assert allclose(W.min(),%f)" % W.min()
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
168 print "assert allclose(W.max(), %f)" % W.max()
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 print "--------------"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
170
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
171 assert allclose(l2(U), 21.351664)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
172 assert allclose(l2(W), 6.275828)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 assert allclose(U.min(), -1.176703)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 assert allclose(U.max(), 0.859802)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
175 assert allclose(W.min(), -0.223128)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
176 assert allclose(W.max(), 0.227558)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178 break
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
179
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
180 if print_jj:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
181 if not as_unittest:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
182 tile(imgs_fn(jj), "imgs_%06i.png" % jj)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
183 if persistent_chains:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
184 tile(smplr.positions.value, "sample_%06i.png" % jj)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
185 tile(rbm.U.value.T, "U_%06i.png" % jj)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
186 tile(rbm.W.value.T, "W_%06i.png" % jj)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
187
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
188 print 'saving samples', jj, 'epoch', jj / (epoch_size / batchsize)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
189
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
190 print 'l2(U)', l2(rbm.U.get_value(borrow=True)),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
191 print 'l2(W)', l2(rbm.W.get_value(borrow=True)),
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
192 print 'l1_penalty',
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
193 try:
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
194 print trainer.effective_l1_penalty.get_value(borrow=True)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
195 except:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
196 print trainer.effective_l1_penalty
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
197
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
198 print 'U min max', rbm.U.get_value(borrow=True).min(), rbm.U.get_value(borrow=True).max(),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
199 print 'W min max', rbm.W.get_value(borrow=True).min(), rbm.W.get_value(borrow=True).max(),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
200 print 'a min max', rbm.a.get_value(borrow=True).min(), rbm.a.get_value(borrow=True).max(),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
201 print 'b min max', rbm.b.get_value(borrow=True).min(), rbm.b.get_value(borrow=True).max(),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
202 print 'c min max', rbm.c.get_value(borrow=True).min(), rbm.c.get_value(borrow=True).max()
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
203
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
204 if persistent_chains:
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
205 print 'parts min', smplr.positions.get_value(borrow=True).min(),
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
206 print 'max', smplr.positions.get_value(borrow=True).max(),
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
207 print 'HMC step', smplr.stepsize.get_value(borrow=True),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
208 print 'arate', smplr.avg_acceptance_rate.get_value(borrow=True)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
209
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
210 l2_of_Ugrad = learn_fn(jj)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
211
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
212 if persistent_chains and print_jj:
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
213 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
214 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
215 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
216 #print 'FE+', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
217 #print 'FE+[0]', float(l2_of_Ugrad[3]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
218 #print 'FE+[1]', float(l2_of_Ugrad[4]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219 #print 'FE+[2]', float(l2_of_Ugrad[5]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220 #print 'FE+[3]', float(l2_of_Ugrad[6])
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
222 if not as_unittest:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
223 if jj % 2000 == 0:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
224 print ''
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
225 print 'Saving rbm...'
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
226 cPickle.dump(rbm, open('mcRBM.rbm.%06i.pkl' % jj, 'w'), -1)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
227 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
228 print 'Saving sampler...'
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
229 cPickle.dump(smplr, open(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
230 'mcRBM.smplr.%06i.pkl' % jj, 'w'), -1)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
231
1273
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
232 if not as_unittest:
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
233 return rbm, smplr
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
234
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
235
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
236 def run_classif_experiment(checkpoint):
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
237
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
238 R, C = 8, 8
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
239 n_vis = 74
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
240 # PRETRAIN
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
241 #
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
242 # extract 1 million 8x8 patches from TinyImages
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
243 # pre-process them the right way
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
244 # find 74 dims of PCA
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
245 # filter patches through PCA
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
246 whitened_patches, pca_dct = pylearn.dataset_ops.tinyimages.main(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
247 n_imgs=100000, max_components=n_vis, seed=234)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
248 #
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
249 # Set up mcRBM Trainer
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
250 # Initialize P using topological 3x3 overlapping patches thing
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
251 # start learning P matrix after 2 passes through dataset
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
252 #
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
253 rbm_filename = 'mcRBM.rbm.%06i.pkl' % 46000
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
254 try:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
255 open(rbm_filename).close()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
256 load_mcrbm = True
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
257 except:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
258 load_mcrbm = False
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
259
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
260 if load_mcrbm:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
261 print 'loading mcRBM from file', rbm_filename
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
262 rbm = cPickle.load(open(rbm_filename))
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
263
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
264 else:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
265 print "Training mcRBM"
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
266 batchsize = 128
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
267 epoch_size = len(whitened_patches)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
268 tile = pylearn.dataset_ops.tinyimages.save_filters
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
269 train_batch = theano.tensor.matrix()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
270 trainer = mcRBMTrainer.alloc_for_P(
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
271 rbm=mcRBM_withP.alloc_topo_P(n_I=n_vis, n_J=81),
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
272 visible_batch=train_batch,
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
273 batchsize=batchsize,
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
274 initial_lr_per_example=0.05,
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
275 l1_penalty=1e-3,
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
276 l1_penalty_start=sys.maxint,
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
277 p_training_start=2 * epoch_size // batchsize,
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
278 persistent_chains=False)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
279 rbm = trainer.rbm
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
280 learn_fn = theano.function([train_batch], outputs=[],
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
281 updates=trainer.cd_updates())
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
282 smplr = trainer._last_cd1_sampler
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
283
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
284 ii = 0
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
285 for i_epoch in range(6):
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
286 for i_batch in xrange(epoch_size // batchsize):
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
287 batch_vals = whitened_patches[i_batch *
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
288 batchsize:(i_batch + 1) * batchsize]
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
289 learn_fn(batch_vals)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
290
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
291 if (ii % 1000) == 0:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
292 #tile(imgs_fn(ii), "imgs_%06i.png"%ii)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
293 tile(rbm.U.value.T, "U_%06i.png" % ii)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
294 tile(rbm.W.value.T, "W_%06i.png" % ii)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
295
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
296 print 'saving samples', ii, 'epoch', i_epoch, i_batch
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
297
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
298 print 'l2(U)', l2(rbm.U.get_value(borrow=True)),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
299 print 'l2(W)', l2(rbm.W.get_value(borrow=True)),
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
300 print 'l1_penalty',
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
301 try:
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
302 print trainer.effective_l1_penalty.get_value(borrow=True)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
303 except:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
304 print trainer.effective_l1_penalty
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
305
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
306 print 'U min max', rbm.U.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
307 borrow=True).min(), rbm.U.get_value(borrow=True).max(),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
308 print 'W min max', rbm.W.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
309 borrow=True).min(), rbm.W.get_value(borrow=True).max(),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
310 print 'a min max', rbm.a.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
311 borrow=True).min(), rbm.a.get_value(borrow=True).max(),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
312 print 'b min max', rbm.b.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
313 borrow=True).min(), rbm.b.get_value(borrow=True).max(),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
314 print 'c min max', rbm.c.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
315 borrow=True).min(), rbm.c.get_value(borrow=True).max()
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
316
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
317 print 'HMC step', smplr.stepsize.get_value(borrow=True),
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
318 print 'arate', smplr.avg_acceptance_rate.get_value(borrow=True)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
319 print 'P min max', rbm.P.get_value(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
320 borrow=True).min(), rbm.P.get_value(borrow=True).max(),
1512
7f166d01bf8e Remove deprecation warning.
Frederic Bastien <nouiz@nouiz.org>
parents: 1509
diff changeset
321 print 'P_lr', trainer.p_lr.get_value(borrow=True)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
322 print ''
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
323 print 'Saving rbm...'
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
324 cPickle.dump(rbm, open('mcRBM.rbm.%06i.pkl' % ii, 'w'), -1)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
325
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
326 ii += 1
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
327
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
328 # extract convolutional features from the CIFAR10 data
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
329 feat_filename = 'mcrbm_features.npy'
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
330 feat_filename = 'cifar10.features.46000.npy'
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
331 try:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
332 open(feat_filename).close()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
333 load_features = True
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
334 except:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
335 load_features = False
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
336
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
337 if load_features:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
338 print 'Loading features from', feat_filename
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
339 all_features = numpy.load(feat_filename, mmap_mode='r')
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
340 else:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
341 batchsize = 100
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
342 feat_idx = tensor.lscalar()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
343 feat_idx_range = feat_idx * batchsize + tensor.arange(batchsize)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
344 train_batch_x, train_batch_y = pylearn.dataset_ops.cifar10.cifar10(
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
345 feat_idx_range,
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
346 split='all',
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
347 dtype='uint8',
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
348 rasterized=False,
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
349 color='rgb')
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
350
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
351 WINDOW_SIZE = 8
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
352 WINDOW_STRIDE = 4
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
353
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
354 # put these into shared vars because support for big matrix
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
355 # constants is bad, (comparing them is slow)
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
356 pca_eigvecs = theano.shared(pca_dct['eig_vecs'].astype('float32'))
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
357 pca_eigvals = theano.shared(pca_dct['eig_vals'].astype('float32'))
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
358 pca_mean = theano.shared(pca_dct['mean'].astype('float32'))
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
359
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
360 def theano_pca_whiten(X):
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
361 #copying preprepcessing.pca.pca_whiten
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
362 return tensor.true_div(
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
363 tensor.dot(X - pca_mean, pca_eigvecs),
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
364 tensor.sqrt(pca_eigvals) + 1e-8)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
365
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
366 h_list = []
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
367 g_list = []
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
368 for r_offset in range(0, 32 - WINDOW_SIZE + 1, WINDOW_STRIDE):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
369 for c_offset in range(0, 32 - WINDOW_SIZE + 1, WINDOW_STRIDE):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
370 window = train_batch_x[:, r_offset:r_offset + WINDOW_SIZE,
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
371 c_offset:c_offset + WINDOW_SIZE, :]
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
372 assert window.dtype == 'uint8'
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
373
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
374 #rasterize the patches
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
375 raster_window = tensor.flatten(tensor.cast(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
376 window, 'float32'), 2)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
377
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
378 #subtract off the mean of each image
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
379 raster_window = raster_window - raster_window.mean(axis=1).reshape((batchsize,1))
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
380
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
381 h, g = rbm.expected_h_g_given_v(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
382 theano_pca_whiten(raster_window))
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
383
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
384 h_list.append(h)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
385 g_list.append(g)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
386
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
387 hg = tensor.concatenate(h_list + g_list, axis=1)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
388
1507
2a6a6f16416c fix import.
Frederic Bastien <nouiz@nouiz.org>
parents: 1335
diff changeset
389 feat_fn = theano.function([feat_idx], hg)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
390 features = numpy.empty((60000, 11025), dtype='float32')
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
391 for i in xrange(60000 // batchsize):
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
392 if i % 100 == 0:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
393 print("feature batch %i" % i)
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
394 features[i * batchsize:(i + 1) * batchsize] = feat_fn(i)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
395
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
396 print("saving features to %s" % feat_filename)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
397 numpy.save(feat_filename, features)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
398 all_features = features
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
399 del features
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
400
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
401 # CLASSIFY FEATURES
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
402 if 0:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
403 # nothing to load
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
404 pass
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
405 else:
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
406 batchsize = 100
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
407
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
408 if feat_filename.startswith('cifar'):
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
409 learnrate = 0.002
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
410 l1_regularization = 0.004
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
411 anneal_epoch = 100
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
412 n_epochs = 500
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
413 else:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
414 learnrate = 0.005
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
415 l1_regularization = 0.004
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
416 n_epochs = 100
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
417 anneal_epoch = 20
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
418
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
419 x_i = tensor.matrix()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
420 y_i = tensor.ivector()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
421 lr = tensor.scalar()
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
422 #l1_regularization = float(sys.argv[1]) #1.e-3
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
423 #l2_regularization = float(sys.argv[2]) #1.e-3*0
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
424
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
425 feature_logreg = LogisticRegression.new(x_i,
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
426 n_in=11025, n_out=10,
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
427 dtype=x_i.dtype)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
428
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
429 # marc'aurelio does this...
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
430 feature_logreg.w.value = numpy.random.RandomState(44).randn(11025,
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
431 10) * .02
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
432
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
433 traincost = feature_logreg.nll(y_i).sum()
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
434 traincost = traincost + abs(feature_logreg.w).sum() * l1_regularization
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
435 #traincost = traincost + (feature_logreg.w**2).sum() * l2_regularization
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
436 train_logreg_fn = theano.function([x_i, y_i, lr],
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
437 [feature_logreg.nll(y_i).mean(),
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
438 feature_logreg.errors(y_i).mean()],
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
439 updates=pylearn.gd.sgd.sgd_updates(
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
440 params=feature_logreg.params,
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
441 grads=tensor.grad(traincost, feature_logreg.params),
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
442 stepsizes=[lr, lr / 10.]))
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
443
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
444 all_labels = pylearn.dataset_ops.cifar10.all_data_labels('uint8')[1]
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
445 pylearn.dataset_ops.cifar10.all_data_labels.forget()
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
446 # clear memo cache
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
447 assert len(all_labels) == 60000
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
448 if 0:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
449 print "Using validation set"
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
450 train_labels = all_labels[:40000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
451 valid_labels = all_labels[40000:50000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
452 test_labels = all_labels[50000:60000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
453 train_features = all_features[:40000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
454 valid_features = all_features[40000:50000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
455 test_features = all_features[50000:60000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
456 else:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
457 print "NOT USING validation set"
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
458 train_labels = all_labels[:50000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
459 valid_labels = None
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
460 test_labels = all_labels[50000:60000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
461 train_features = all_features[:50000]
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
462 valid_features = None
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
463 test_features = all_features[50000:60000]
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
464
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
465 if 1:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
466 print "Computing mean and std.dev"
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
467 train_mean = train_features.mean(axis=0)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
468 train_std = train_features.std(axis=0) + 1e-4
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
469 preproc = lambda x: (x - train_mean) / (0.1 + train_std)
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
470 else:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
471 print "Not centering data"
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
472 preproc = lambda x: x
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
473
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
474 for epoch in xrange(n_epochs):
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
475 print 'epoch', epoch
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
476 # validate
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
477 # Marc'Aurelio, you crazy!!
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
478 # the division by batchsize is done in the cost function
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
479 e_lr = learnrate / (batchsize * max(1.0, numpy.floor(
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
480 max(1., epoch / float(anneal_epoch)) - 2)))
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
481
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
482 if valid_features is not None:
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
483 l01s = []
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
484 nlls = []
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
485 for i in xrange(10000 / batchsize):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
486 x_i = valid_features[i * batchsize:(i + 1) * batchsize]
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
487 y_i = valid_labels[i * batchsize:(i + 1) * batchsize]
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
488
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
489 #lr=0.0 -> no learning, safe for validation set
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
490 nll, l01 = train_logreg_fn(preproc(x_i), y_i, 0.0)
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
491 nlls.append(nll)
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
492 l01s.append(l01)
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
493 print 'validate log_reg', numpy.mean(nlls), numpy.mean(l01s)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
494
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
495 # test
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
496
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
497 l01s = []
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
498 nlls = []
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
499 for i in xrange(len(test_features) // batchsize):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
500 x_i = test_features[i * batchsize:(i + 1) * batchsize]
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
501 y_i = test_labels[i * batchsize:(i + 1) * batchsize]
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
502
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
503 #lr=0.0 -> no learning, safe for validation set
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
504 nll, l01 = train_logreg_fn(preproc(x_i), y_i, 0.0)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
505 nlls.append(nll)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
506 l01s.append(l01)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
507 print 'test log_reg', numpy.mean(nlls), numpy.mean(l01s)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
508
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
509 #train
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
510 l01s = []
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
511 nlls = []
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
512 for i in xrange(len(train_features) // batchsize):
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
513 x_i = train_features[i * batchsize:(i + 1) * batchsize]
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
514 y_i = train_labels[i * batchsize:(i + 1) * batchsize]
1333
c7b2da4e2df6 modifs to test_mcRBM to reproduce mcRBM classif results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1323
diff changeset
515 nll, l01 = train_logreg_fn(preproc(x_i), y_i, e_lr)
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
516 nlls.append(nll)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
517 l01s.append(l01)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
518 print 'train log_reg', numpy.mean(nlls), numpy.mean(l01s)
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
519
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
520
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
521 import pickle as cPickle
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
522 #import cPickle
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
523 if __name__ == '__main__':
1509
b709f6b53b17 auto fix white space.
Frederic Bastien <nouiz@nouiz.org>
parents: 1508
diff changeset
524 if 0:
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
525 #learning 16 x 16 pinwheel filters from official cifar patches (MAR)
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
526 rbm, smplr = test_reproduce_ranzato_hinton_2010(
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
527 as_unittest=False,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
528 n_train_iters=5000,
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
529 rbm_alloc=lambda n_I: mcRBM_withP.alloc_topo_P(n_I, n_J=81),
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
530 trainer_alloc=mcRBMTrainer.alloc_for_P,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
531 dataset='MAR'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
532 )
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
533
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
534 if 0:
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
535 # pretraining settings
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
536 rbm, smplr = test_reproduce_ranzato_hinton_2010(
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
537 as_unittest=False,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
538 n_train_iters=60000,
1524
9d21919e2332 autopep8
Frederic Bastien <nouiz@nouiz.org>
parents: 1512
diff changeset
539 rbm_alloc=lambda n_I: mcRBM_withP.alloc_topo_P(n_I, n_J=81),
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
540 trainer_alloc=mcRBMTrainer.alloc_for_P,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
541 lr_per_example=0.05,
1286
8905186b176c test_mcRBM - added code to iterate over tinyimages
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1284
diff changeset
542 dataset='tinyimages_patches',
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
543 l1_penalty=1e-3,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
544 l1_penalty_start=30000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
545 #l1_penalty_start=350, #DEBUG
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
546 persistent_chains=False
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
547 )
1323
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
548
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
549 if 1:
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
550 def checkpoint():
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
551 return checkpoint
d6726417cf57 adding training script for test_mcRBM to reproduce classification results
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1286
diff changeset
552 run_classif_experiment(checkpoint=checkpoint)