Mercurial > ift6266

annotate deep/crbm/mnist_crbm.py @ 350:625c0c3fcbdb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Amelioration de l'efficacite de la sauvegarde des parametres
author SylvainPL <sylvain.pannetier.lebeuf@umontreal.ca>
date Tue, 20 Apr 2010 16:38:13 -0400
parents 8d116d4a7593
children ffbf0e41bcee
rev   line source
337
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
1 #!/usr/bin/python
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
2
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
3 import sys
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
4 import os, os.path
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
5
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
6 import numpy as N
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
7
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
8 import theano
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
9 import theano.tensor as T
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
10
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
11 from crbm import CRBM, ConvolutionParams
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
12
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
13 from pylearn.datasets import MNIST
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
14 from pylearn.io.image_tiling import tile_raster_images
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
15
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
16 import Image
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
17
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
18 from pylearn.io.seriestables import *
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
19 import tables
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
20
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
21 IMAGE_OUTPUT_DIR = 'img/'
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
22
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
23 REDUCE_EVERY = 100
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
24
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
25 def filename_from_time(suffix):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
26 import datetime
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
27 return str(datetime.datetime.now()) + suffix + ".png"
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
28
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
29 # Just a shortcut for a common case where we need a few
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
30 # related Error (float) series
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
31 def get_accumulator_series_array( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
32 hdf5_file, group_name, series_names,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
33 reduce_every,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
34 index_names=('epoch','minibatch'),
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
35 stdout_too=True,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
36 skip_hdf5_append=False):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
37 all_series = []
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
38
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
39 hdf5_file.createGroup('/', group_name)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
40
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
41 other_targets = []
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
42 if stdout_too:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
43 other_targets = [StdoutAppendTarget()]
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
44
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
45 for sn in series_names:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
46 series_base = \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
47 ErrorSeries(error_name=sn,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
48 table_name=sn,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
49 hdf5_file=hdf5_file,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
50 hdf5_group='/'+group_name,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
51 index_names=index_names,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
52 other_targets=other_targets,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
53 skip_hdf5_append=skip_hdf5_append)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
54
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
55 all_series.append( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
56 AccumulatorSeriesWrapper( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
57 base_series=series_base,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
58 reduce_every=reduce_every))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
59
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
60 ret_wrapper = SeriesArrayWrapper(all_series)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
61
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
62 return ret_wrapper
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
63
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
64 class MnistCrbm(object):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
65 def __init__(self):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
66 self.mnist = MNIST.full()#first_10k()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
67
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
68 self.cp = ConvolutionParams( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
69 num_filters=40,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
70 num_input_planes=1,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
71 height_filters=12,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
72 width_filters=12)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
73
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
74 self.image_size = (28,28)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
75
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
76 self.minibatch_size = 10
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
77
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
78 self.lr = 0.01
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
79 self.sparsity_lambda = 1.0
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
80 # about 1/num_filters, so only one filter active at a time
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
81 # 40 * 0.05 = ~2 filters active for any given pixel
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
82 self.sparsity_p = 0.05
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
83
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
84 self.crbm = CRBM( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
85 minibatch_size=self.minibatch_size,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
86 image_size=self.image_size,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
87 conv_params=self.cp,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
88 learning_rate=self.lr,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
89 sparsity_lambda=self.sparsity_lambda,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
90 sparsity_p=self.sparsity_p)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
91
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
92 self.num_epochs = 10
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
93
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
94 self.init_series()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
95
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
96 def init_series(self):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
97
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
98 series = {}
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
99
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
100 basedir = os.getcwd()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
101
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
102 h5f = tables.openFile(os.path.join(basedir, "series.h5"), "w")
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
103
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
104 cd_series_names = self.crbm.cd_return_desc
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
105 series['cd'] = \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
106 get_accumulator_series_array( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
107 h5f, 'cd', cd_series_names,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
108 REDUCE_EVERY,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
109 stdout_too=True)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
110
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
111 sparsity_series_names = self.crbm.sparsity_return_desc
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
112 series['sparsity'] = \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
113 get_accumulator_series_array( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
114 h5f, 'sparsity', sparsity_series_names,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
115 REDUCE_EVERY,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
116 stdout_too=True)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
117
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
118 # so first we create the names for each table, based on
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
119 # position of each param in the array
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
120 params_stdout = StdoutAppendTarget("\n------\nParams")
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
121 series['params'] = SharedParamsStatisticsWrapper(
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
122 new_group_name="params",
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
123 base_group="/",
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
124 arrays_names=['W','b_h','b_x'],
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
125 hdf5_file=h5f,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
126 index_names=('epoch','minibatch'),
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
127 other_targets=[params_stdout])
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
128
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
129 self.series = series
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
130
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
131 def train(self):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
132 num_minibatches = len(self.mnist.train.x) / self.minibatch_size
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
133
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
134 print_every = 1000
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
135 visualize_every = 5000
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
136 gibbs_steps_from_random = 1000
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
137
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
138 for epoch in xrange(self.num_epochs):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
139 for mb_index in xrange(num_minibatches):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
140 mb_x = self.mnist.train.x \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
141 [mb_index : mb_index+self.minibatch_size]
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
142 mb_x = mb_x.reshape((self.minibatch_size, 1, 28, 28))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
143
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
144 #E_h = crbm.E_h_given_x_func(mb_x)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
145 #print "Shape of E_h", E_h.shape
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
146
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
147 cd_return = self.crbm.CD_step(mb_x)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
148 sp_return = self.crbm.sparsity_step(mb_x)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
149
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
150 self.series['cd'].append( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
151 (epoch, mb_index), cd_return)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
152 self.series['sparsity'].append( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
153 (epoch, mb_index), sp_return)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
154
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
155 total_idx = epoch*num_minibatches + mb_index
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
156
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
157 if (total_idx+1) % REDUCE_EVERY == 0:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
158 self.series['params'].append( \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
159 (epoch, mb_index), self.crbm.params)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
160
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
161 if total_idx % visualize_every == 0:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
162 self.visualize_gibbs_result(\
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
163 mb_x, gibbs_steps_from_random)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
164 self.visualize_gibbs_result(mb_x, 1)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
165 self.visualize_filters()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
166
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
167 def visualize_gibbs_result(self, start_x, gibbs_steps):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
168 # Run minibatch_size chains for gibbs_steps
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
169 x_samples = None
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
170 if not start_x is None:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
171 x_samples = self.crbm.gibbs_samples_from(start_x, gibbs_steps)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
172 else:
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
173 x_samples = self.crbm.random_gibbs_samples(gibbs_steps)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
174 x_samples = x_samples.reshape((self.minibatch_size, 28*28))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
175
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
176 tile = tile_raster_images(x_samples, self.image_size,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
177 (1, self.minibatch_size), output_pixel_vals=True)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
178
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
179 filepath = os.path.join(IMAGE_OUTPUT_DIR,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
180 filename_from_time("gibbs"))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
181 img = Image.fromarray(tile)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
182 img.save(filepath)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
183
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
184 print "Result of running Gibbs", \
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
185 gibbs_steps, "times outputed to", filepath
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
186
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
187 def visualize_filters(self):
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
188 cp = self.cp
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
189
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
190 # filter size
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
191 fsz = (cp.height_filters, cp.width_filters)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
192 tile_shape = (cp.num_filters, cp.num_input_planes)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
193
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
194 filters_flattened = self.crbm.W.value.reshape(
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
195 (tile_shape[0]*tile_shape[1],
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
196 fsz[0]*fsz[1]))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
197
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
198 tile = tile_raster_images(filters_flattened, fsz,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
199 tile_shape, output_pixel_vals=True)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
200
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
201 filepath = os.path.join(IMAGE_OUTPUT_DIR,
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
202 filename_from_time("filters"))
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
203 img = Image.fromarray(tile)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
204 img.save(filepath)
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
205
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
206 print "Filters (as images) outputed to", filepath
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
207 print "b_h is", self.crbm.b_h.value
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
208
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
209
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
210
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
211
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
212 if __name__ == '__main__':
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
213 mc = MnistCrbm()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
214 mc.train()
8d116d4a7593 Added convolutional RBM (ala Lee09) code, imported from my working dir elsewhere. Seems to work for one layer. No subsampling yet.
fsavard
parents:
diff changeset
215