Mercurial > pylearn

annotate pylearn/algorithms/tests/test_mcRBM.py @ 1284:1817485d586d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mcRBM - many changes incl. adding support for pooling matrix
author James Bergstra <bergstrj@iro.umontreal.ca>
date Wed, 15 Sep 2010 17:49:21 -0400
parents 7bb5dd98e671
children 8905186b176c
rev   line source
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 from pylearn.algorithms.mcRBM import *
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
2 import pylearn.datasets.cifar10
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
3
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
4 import pylearn.dataset_ops.cifar10
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
5
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
6 def _mar_train_patches(dtype):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
7 R,C=16,16
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
8 train_data = pylearn.dataset_ops.cifar10.train_data_labels(dtype)[0][:40000]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
9 #train_data shape is (40000, 3072)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
10 train_data = train_data.reshape((40000,3,32,32)).transpose([0,2,3,1])
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
11 patches = train_data[:, :R, :C, :].reshape((40000, 3*R*C))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
12 patches -= patches.mean(axis=0)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
13 wpatches = numpy.dot(patches, d['pcatransf'].T)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
14 return wpatches
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
15
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
16 def mar_centered(s_idx, split, dtype='float64', rasterized=False, color='grey'):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
17 """
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
18 Returns a pair (img, label) of theano expressions for cifar-10 samples
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
19
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
20 :param s_idx: the indexes
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
21
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
22 :param split:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
23
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
24 :param dtype:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
25
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
26 :param rasterized: return examples as vectors (True) or 28x28 matrices (False)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
27
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
28 :param color: control how to deal with the color in the images'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
29 - grey greyscale (with luminance weighting)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
30 - rgb add a trailing dimension of length 3 with rgb colour channels
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
31
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
32 """
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
33
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
34 split_options = {'train':(train_data, train_labels),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
35 'valid': (valid_data, valid_labels),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
36 'test': (test_data, test_labels)}
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
37
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
38 if split not in split_options:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
39 raise ValueError('invalid split option', (split, split_options.keys()))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
40
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
41 color_options = ('grey', 'rgb')
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
42 if color not in color_options:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
43 raise ValueError('invalid color option', (color, color_options))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
44
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
45 x_fn, y_fn = split_options[split]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
46
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
47 x_op = TensorFnDataset(dtype, (False,), (x_fn, (dtype,)), (3072,))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
48 y_op = TensorFnDataset('int32', (), y_fn)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
49
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
50 x = x_op(s_idx)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
51 y = y_op(s_idx)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
52
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
53 # Y = 0.3R + 0.59G + 0.11B from
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
54 # http://gimp-savvy.com/BOOK/index.html?node54.html
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
55 rgb_dtype = 'float32'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
56 if dtype == 'float64':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
57 rgb_dtype = dtype
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
58 r = numpy.asarray(.3, dtype=rgb_dtype)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
59 g = numpy.asarray(.59, dtype=rgb_dtype)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
60 b = numpy.asarray(.11, dtype=rgb_dtype)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
62 if x.ndim == 1:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
63 if rasterized:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
64 if color=='grey':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
65 x = r * x[:1024] + g * x[1024:2048] + b * x[2048:]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
66 if dtype=='uint8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
67 x = theano.tensor.cast(x, 'uint8')
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
68 elif color=='rgb':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
69 # the strides aren't what you'd expect between channels,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
70 # but theano is all about weird strides
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
71 x = x.reshape((3,32*32)).T
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
72 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
73 raise NotImplemented('color', color)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
74 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
75 if color=='grey':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
76 x = r * x[:1024] + g * x[1024:2048] + b * x[2048:]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
77 if dtype=='uint8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
78 x = theano.tensor.cast(x, 'uint8')
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
79 x = x.reshape((32,32))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
80 elif color=='rgb':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
81 # the strides aren't what you'd expect between channels,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
82 # but theano is all about weird strides
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
83 x = x.reshape((3,32,32)).dimshuffle(1, 2, 0)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
84 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
85 raise NotImplemented('color', color)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
86 elif x.ndim == 2:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
87 N = x.shape[0] # symbolic
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
88 if rasterized:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
89 if color=='grey':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
90 x = r * x[:,:1024] + g * x[:,1024:2048] + b * x[:,2048:]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
91 if dtype=='uint8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
92 x = theano.tensor.cast(x, 'uint8')
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
93 elif color=='rgb':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
94 # the strides aren't what you'd expect between channels,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
95 # but theano is all about weird strides
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
96 x = x.reshape((N, 3,32*32)).dimshuffle(0, 2, 1)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
97 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
98 raise NotImplemented('color', color)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
99 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
100 if color=='grey':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
101 x = r * x[:,:1024] + g * x[:,1024:2048] + b * x[:,2048:]
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
102 if dtype=='uint8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
103 x = theano.tensor.cast(x, 'uint8')
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
104 x.reshape((N, 32, 32))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
105 elif color=='rgb':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
106 # the strides aren't what you'd expect between channels,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
107 # but theano is all about weird strides
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
108 x = x.reshape((N,3,32,32)).dimshuffle(0, 2, 3, 1)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
109 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
110 raise NotImplemented('color', color)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
111 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
112 raise ValueError('x has too many dimensions', x.ndim)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
113
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
114 return x, y
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
115
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
116
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
117 def _default_rbm_alloc(n_I, n_K=256, n_J=100):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
118 return mcRBM.alloc(n_I, n_K, n_J)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
119
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
120 def _default_trainer_alloc(rbm, train_batch, batchsize, l1_penalty, l1_penalty_start):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
121 return mcRBMTrainer.alloc(rbm, train_batch, batchsize, l1_penalty=l1_penalty,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
122 l1_penalty_start=l1_penalty_start,persistent_chains=persistent_chains)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
123
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
124
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
125 def test_reproduce_ranzato_hinton_2010(dataset='MAR', as_unittest=True, n_train_iters=5000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
126 rbm_alloc=_default_rbm_alloc, trainer_alloc=_default_trainer_alloc,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
127 lr_per_example=.075,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
128 l1_penalty=1e-3,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
129 l1_penalty_start=1000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
130 persistent_chains=True,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
131 ):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
132
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
133 batchsize = 128
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
134
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 n_vis=105
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 n_patches=10240
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
138 epoch_size=n_patches
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
139 elif dataset=='cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
140 R,C= 8,8 # the size of image patches
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
141 n_vis=96 # pca components
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
142 epoch_size=batchsize*500
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
143 n_patches=epoch_size*20
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 R,C= 16,16 # the size of image patches
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 n_vis=R*C
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 n_patches=100000
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
148 epoch_size=n_patches
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
150 def l2(X):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
151 return numpy.sqrt((X**2).sum())
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
152
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154 tile = pylearn.dataset_ops.image_patches.save_filters_of_ranzato_hinton_2010
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
155 elif dataset == 'cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
156 def tile(X, fname):
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
157 _img = pylearn.datasets.cifar10.tile_rasterized_examples(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
158 pylearn.preprocessing.pca.pca_whiten_inverse(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
159 pylearn.dataset_ops.cifar10.random_cifar_patches_pca(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
160 n_vis, None, 'float32', n_patches, R, C,),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
161 X),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
162 img_shape=(R,C))
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
163 image_tiling.save_tiled_raster_images(_img, fname)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 def tile(X, fname):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166 _img = image_tiling.tile_raster_images(X,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 img_shape=(R,C),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 min_dynamic_range=1e-2)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 image_tiling.save_tiled_raster_images(_img, fname)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
170
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
171 batch_idx = TT.iscalar()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
172
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 if dataset == 'MAR':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 train_batch = pylearn.dataset_ops.image_patches.ranzato_hinton_2010_op(batch_idx * batchsize + np.arange(batchsize))
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
175 elif dataset == 'cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
176 train_batch = pylearn.dataset_ops.cifar10.cifar10_patches(batch_idx * batchsize +
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
177 np.arange(batchsize), 'train', n_patches=n_patches, patch_size=(R,C),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
178 pca_components=n_vis)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
179 else:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
180 train_batch = pylearn.dataset_ops.image_patches.image_patches(
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
181 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 dims = (n_patches,R,C),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 center=True,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
184 unitvar=True,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
185 dtype=floatX,
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
186 rasterized=True)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
187
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
188 if not as_unittest:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
189 imgs_fn = function([batch_idx], outputs=train_batch)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
190
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
191 trainer = trainer_alloc(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
192 rbm_alloc(n_I=n_vis),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
193 train_batch,
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
194 batchsize,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
195 initial_lr_per_example=lr_per_example,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
196 l1_penalty=l1_penalty,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
197 l1_penalty_start=l1_penalty_start,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
198 persistent_chains=persistent_chains)
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
199 rbm=trainer.rbm
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
200
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
201 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
202 grads = trainer.contrastive_grads()
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
203 learn_fn = function([batch_idx],
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
204 outputs=[grads[0].norm(2), grads[0].norm(2), grads[1].norm(2)],
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
205 updates=trainer.cd_updates())
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
206 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
207 learn_fn = function([batch_idx], outputs=[], updates=trainer.cd_updates())
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
208
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
209 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
210 smplr = trainer.sampler
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
211 else:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
212 smplr = trainer._last_cd1_sampler
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
213
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
214 if dataset == 'cifar10patches8x8':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
215 cPickle.dump(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
216 pylearn.dataset_ops.cifar10.random_cifar_patches_pca(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
217 n_vis, None, 'float32', n_patches, R, C,),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
218 open('test_mcRBM.pca.pkl','w'))
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220 print "Learning..."
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221 last_epoch = -1
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
222 for jj in xrange(n_train_iters):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
223 epoch = jj*batchsize / epoch_size
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
224
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
225 print_jj = epoch != last_epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
226 last_epoch = epoch
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
227
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
228 if as_unittest and epoch == 5:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
229 U = rbm.U.value
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
230 W = rbm.W.value
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
231 def allclose(a,b):
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
232 return numpy.allclose(a,b,rtol=1.01,atol=1e-3)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
233 print ""
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
234 print "--------------"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
235 print "assert allclose(l2(U), %f)"%l2(U)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
236 print "assert allclose(l2(W), %f)"%l2(W)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
237 print "assert allclose(U.min(), %f)"%U.min()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
238 print "assert allclose(U.max(), %f)"%U.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
239 print "assert allclose(W.min(),%f)"%W.min()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
240 print "assert allclose(W.max(), %f)"%W.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
241 print "--------------"
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
242
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
243 assert allclose(l2(U), 21.351664)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
244 assert allclose(l2(W), 6.275828)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
245 assert allclose(U.min(), -1.176703)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
246 assert allclose(U.max(), 0.859802)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
247 assert allclose(W.min(),-0.223128)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
248 assert allclose(W.max(), 0.227558 )
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
249
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
250 break
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
251
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
252 if print_jj:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
253 if not as_unittest:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
254 tile(imgs_fn(jj), "imgs_%06i.png"%jj)
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
255 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
256 tile(smplr.positions.value, "sample_%06i.png"%jj)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
257 tile(rbm.U.value.T, "U_%06i.png"%jj)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
258 tile(rbm.W.value.T, "W_%06i.png"%jj)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
259
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
260 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
261
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
262 print 'l2(U)', l2(rbm.U.value),
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
263 print 'l2(W)', l2(rbm.W.value),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
264 print 'l1_penalty',
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
265 try:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
266 print trainer.effective_l1_penalty.value
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
267 except:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
268 print trainer.effective_l1_penalty
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
269
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
270 print 'U min max', rbm.U.value.min(), rbm.U.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
271 print 'W min max', rbm.W.value.min(), rbm.W.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
272 print 'a min max', rbm.a.value.min(), rbm.a.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
273 print 'b min max', rbm.b.value.min(), rbm.b.value.max(),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
274 print 'c min max', rbm.c.value.min(), rbm.c.value.max()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
275
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
276 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
277 print 'parts min', smplr.positions.value.min(),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
278 print 'max',smplr.positions.value.max(),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
279 print 'HMC step', smplr.stepsize.value,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
280 print 'arate', smplr.avg_acceptance_rate.value
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
281
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
282
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
283 l2_of_Ugrad = learn_fn(jj)
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
284
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
285 if persistent_chains and print_jj:
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
286 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
287 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
288 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
289 #print 'FE+', float(l2_of_Ugrad[2]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
290 #print 'FE+[0]', float(l2_of_Ugrad[3]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
291 #print 'FE+[1]', float(l2_of_Ugrad[4]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
292 #print 'FE+[2]', float(l2_of_Ugrad[5]),
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
293 #print 'FE+[3]', float(l2_of_Ugrad[6])
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
294
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
295 if not as_unittest:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
296 if jj % 2000 == 0:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
297 print ''
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
298 print 'Saving rbm...'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
299 cPickle.dump(rbm, open('mcRBM.rbm.%06i.pkl'%jj, 'w'), -1)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
300 if persistent_chains:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
301 print 'Saving sampler...'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
302 cPickle.dump(smplr, open('mcRBM.smplr.%06i.pkl'%jj, 'w'), -1)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
303
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
304
1273
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
305 if not as_unittest:
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
306 return rbm, smplr
1284
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
307
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
308 import pickle as cPickle
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
309 #import cPickle
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
310 if __name__ == '__main__':
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
311 if 0:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
312 #learning 16 x 16 pinwheel filters from official cifar patches (MAR)
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
313 rbm,smplr = test_reproduce_ranzato_hinton_2010(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
314 as_unittest=False,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
315 n_train_iters=5000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
316 rbm_alloc=lambda n_I : mcRBM_withP.alloc_topo_P(n_I, n_J=81),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
317 trainer_alloc=mcRBMTrainer.alloc_for_P,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
318 dataset='MAR'
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
319 )
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
320
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
321 if 1:
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
322 rbm,smplr = test_reproduce_ranzato_hinton_2010(
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
323 as_unittest=False,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
324 n_train_iters=60000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
325 rbm_alloc=lambda n_I : mcRBM_withP.alloc_topo_P(n_I, n_J=81),
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
326 trainer_alloc=mcRBMTrainer.alloc_for_P,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
327 lr_per_example=0.05,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
328 dataset='cifar10patches8x8',
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
329 l1_penalty=1e-3,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
330 l1_penalty_start=30000,
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
331 #l1_penalty_start=350, #DEBUG
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
332 persistent_chains=False
1817485d586d mcRBM - many changes incl. adding support for pooling matrix
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
333 )