annotate pylearn/algorithms/mcRBM.py @ 997:71b0132b694a

mcRBM - removed container logic that was redundant with global methods
author James Bergstra <bergstrj@iro.umontreal.ca>
date Tue, 24 Aug 2010 16:00:08 -0400
parents 60f279ec0f7f
children 8ba8b08e0442
rev   line source
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
8 and performs one of the experiments on CIFAR-10 discussed in that paper. There are some minor
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
9 discrepancies between the paper and the accompanying code (train_mcRBM.py), and the
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
10 accompanying code has been taken to be correct in those cases because I couldn't get things to
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
11 work otherwise.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
30 Version in paper
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
31 ----------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
32
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
33 Full Energy of the Mean and Covariance RBM, with
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
40 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
41 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i (U_{if} v_i) / |U_{.f}|*|v| )^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
42 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
43 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
44 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
45 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
46
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
47 For the energy function to correspond to a probability distribution, P must be non-positive. P
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
48 is initialized to be a diagonal, and in our experience it can be left as such because even in
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
49 the paper it has a very low learning rate, and is only allowed to be updated after the filters
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
50 in U are learned (in effect).
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
51
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
52 Version in published train_mcRBM code
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
53 -------------------------------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
54
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
55 The train_mcRBM file implements learning in a similar but technically different Energy function:
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 E (v, h, g) =
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
58 - 0.5 \sum_f \sum_k P_{fk} h_k (\sum_i U_{if} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
64 There are two differences with respect to the paper:
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
65
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
66 - 'v' is not normalized by its length, but rather it is normalized to have length close to
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
67 the square root of the number of its components. The variable called 'small' that
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
68 "avoids division by zero" is orders larger than machine precision, and is on the order of
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
69 the normalized sum-of-squares, so I've included it in the Energy function.
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
70
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
71 - 'U' is also not normalized by its length. U is initialized to have columns that are
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
72 shorter than unit-length (approximately 0.2 with the 105 principle components in the
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
73 train_mcRBM data). During training, the columns of U are constrained manually to have
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
74 equal lengths (see the use of normVF), but Euclidean norm is allowed to change. During
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
75 learning it quickly converges towards 1 and then exceeds 1. It does not seem like this
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
76 column-wise normalization of U is justified by maximum-likelihood, I have no intuition
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
77 for why it is used.
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
78
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
80 Version in this code
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
81 --------------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
82
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
83 This file implements the same algorithm as the train_mcRBM code, except that the P matrix is
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
84 omitted for clarity, and replaced analytically with a negative identity matrix.
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
85
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
86 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
87 + 0.5 \sum_k h_k (\sum_i U_{ik} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
88 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
89 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
90 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
91 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
92
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
93
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
94
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
115 Matrices are generally layed out and accessed according to a C-order convention.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
119 #
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
120 # WORKING NOTES
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
121 # THIS DERIVATION IS BASED ON THE ** PAPER ** ENERGY FUNCTION
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
122 # NOT THE ENERGY FUNCTION IN THE CODE!!!
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
123 #
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127 #
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
128 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
129 # E (v, h, g) =
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
130 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
131 # - \sum_k b_k h_k
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
132 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
133 # - \sum_j \sum_i W_{ij} g_j v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
134 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
135 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
136 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
137 #
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 # - \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
148 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
149 # - \sum_i a_i v_i ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
150 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
151 # Get rid of double negs in exp
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
156 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
157 # + \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
158 # + \sum_j c_j g_j))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
159 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
160 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
161 # Break up log
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
163 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
170 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
171 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
172 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(..
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|* |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
175 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
176 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
179 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
180 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
181 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
184 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
185 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
186 # For negative-one-diagonal P this gives:
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
187 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
188 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (|U_{*k}|*|v|)))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
189 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
190 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
191 # - \sum_i a_i v_i
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
192
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
193 import sys
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
194 import logging
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
195 import numpy as np
973
aa201f357d7b mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 972
diff changeset
196 import numpy
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
197
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
198 import theano
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
199 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
200 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
201 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
202
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
203 import pylearn
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
204 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
205 from pylearn.io import image_tiling
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
206
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
207 #TODO: This should be in the datasets folder
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
208 import pylearn.datasets.config
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
209 from pylearn.dataset_ops.protocol import TensorFnDataset
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
210 from pylearn.dataset_ops.memo import memo
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
211 import pylearn
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
212 import scipy.io
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
213 import os
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
214
973
aa201f357d7b mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 972
diff changeset
215
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
216 #TODO: This should be in the nnet part of the library
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
217 def sgd_updates(params, grads, lr):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
218 try:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219 float(lr)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220 lr = [lr for p in params]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221 except TypeError:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
222 pass
974
f2cdcc71ece1 mcRBM - added L1 penalties and normal sign convention to contrastive grad
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 973
diff changeset
223 updates = [(p, p - plr * gp) for (plr, p, gp) in zip(lr, params, grads)]
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
224 return updates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
225
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
226 @memo
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
227 def load_mcRBM_demo_patches():
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
228 d = scipy.io.loadmat(os.path.join(pylearn.datasets.config.data_root(),'image_patches', 'mcRBM', 'training_colorpatches_16x16_demo.mat'))
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
229 totnumcases = d["whitendata"].shape[0]
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
230 #d = d["whitendata"][0:np.floor(totnumcases/batch_size)*batch_size,:].copy()
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
231 d = d["whitendata"].copy()
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
232 return d
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
233
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
234
989
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
235 def hidden_cov_units_preactivation_given_v(rbm, v, small=0.5):
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
236 """Return argument to the sigmoid that would give mean of covariance hid units
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
237
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
238 See the math at the top of this file for what 'adjusted' means.
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
239
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
240 return b - 0.5 * dot(adjusted(v), U)**2
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
241 """
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
242 (U,W,a,b,c) = rbm
989
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
243 unit_v = v / (TT.sqrt(TT.mean(v**2, axis=1)+small)).dimshuffle(0,'x') # adjust row norm
9e753ddcc320 mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 988
diff changeset
244 return b - 0.5 * dot(unit_v, U)**2
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
245
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
246 def free_energy_terms_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
247 """Returns theano expression for the terms that are added to form the free energy of
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
248 visible vector `v` in an mcRBM.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
249
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
250 1. Free energy related to covariance hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
251 2. Free energy related to mean hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
252 3. Free energy related to L2-Norm of `v`
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
253 4. Free energy related to projection of `v` onto biases `a`
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
254 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
255 U, W, a, b, c = rbm
975
38e66e0da66a mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 974
diff changeset
256 t0 = -TT.sum(TT.nnet.softplus(hidden_cov_units_preactivation_given_v(rbm, v)),axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 974
diff changeset
257 t1 = -TT.sum(TT.nnet.softplus(c + dot(v,W)), axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 974
diff changeset
258 t2 = 0.5 * TT.sum(v**2, axis=1)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
259 t3 = -TT.dot(v, a)
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
260 return [t0, t1, t2, t3]
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
261
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
262 def free_energy_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
263 """Returns theano expression for free energy of visible vector `v` in an mcRBM
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
264 """
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
265 return sum(free_energy_terms_given_v(rbm,v))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
266
990
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
267 def contrastive_gradient(rbm, pos_v, neg_v, U_l1_penalty=0, W_l1_penalty=0):
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
268 """Return a list of gradient expressions for the rbm parameters
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
269
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
270 :param pos_v: positive-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
271 :param neg_v: negative-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
272 :param U_l1_penalty: a scalar-valued multiplier on the L1 penalty on U
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
273 :param W_l1_penalty: a scalar-valued multiplier on the L1 penalty on W
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
274 """
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
275 U, W, a, b, c = rbm
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
276 pos_FE = free_energy_given_v(rbm, pos_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
277 neg_FE = free_energy_given_v(rbm, neg_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
278 c0 = (pos_FE - neg_FE).sum()
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
279 c1 = abs(U).sum()*U_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
280 c2 = abs(W).sum()*W_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
281 cost = c0 + c1 + c2
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
282 rval = theano.tensor.grad(cost, list(rbm))
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
283 return rval
e70e74464170 mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 989
diff changeset
284
991
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
285 def expected_h_g_given_v(rbm, v):
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
286 """Returns tuple (`h`, `g`) of theano expression conditional expectations in an mcRBM.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
287
991
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
288 `h` is the conditional on the covariance units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
289 `g` is the conditional on the mean units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
290
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
291 """
991
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
292 (U, W, a, b, c) = rbm
d68828c98c38 mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 990
diff changeset
293 h = TT.nnet.sigmoid(hidden_cov_units_preactivation_given_v(rbm, v))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
294 g = nnet.sigmoid(c + dot(v,W))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
295 return (h, g)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
296
996
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
297 def n_visible_units(rbm):
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
298 """Return the number of visible units of this RBM
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
299
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
300 For an RBM made from shared variables, this will return an integer,
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
301 for a purely symbolic RBM this will return a theano expression.
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
302
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
303 """
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
304 W = rbm[1]
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
305 try:
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
306 return W.value.shape[0]
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
307 except AttributeError:
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
308 return W.shape[0]
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
309
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
310
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
311 def sampler(rbm, n_particles, n_visible=None, rng=7823748):
996
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
312 """Return an `HMC_sampler` that will draw samples from the distribution over visible
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
313 units specified by this RBM.
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
314
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
315 :param n_particles: this many parallel chains will be simulated.
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
316 :param rng: seed or numpy RandomState object to initialize particles, and to drive the simulation.
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
317 """
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
318 if not hasattr(rng, 'randn'):
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
319 rng = np.random.RandomState(rng)
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
320 if n_visible is None:
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
321 n_visible = n_visible_units(rbm)
996
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
322 rval = HMC_sampler(
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
323 positions = [shared(
996
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
324 rng.randn(
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
325 n_particles,
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
326 n_visible).astype(floatX),
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
327 name='particles')],
996
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
328 energy_fn = lambda p : free_energy_given_v(rbm, p[0]),
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
329 seed=int(rng.randint(2**30)))
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
330 return rval
60f279ec0f7f mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 995
diff changeset
331
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
332 class MeanCovRBM(object):
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
333 """Container for mcRBM parameters
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
334
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
335 It provides parameter lookup by name, as well as a heuristic for initializing the
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
336 parameters for effective learning.
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
337
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
338 Attributes:
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
339
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
340 - U - the covariance filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
341 - W - the mean filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
342 - a - the visible bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
343 - b - the covariance bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
344 - c - the mean bias (theano shared variable)
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
345
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
346 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
347
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
348 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
349
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
350 n_visible = property(lambda s: s.W.value.shape[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
351
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
352 def __init__(self, U, W, a, b, c):
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
353 self.__dict__.update(locals())
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
354 del self.__dict__['self']
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
355
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
356 def __getitem__(self, idx):
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
357 # support unpacking of this container as if it were a tuple
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
358 return self.params[idx]
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
359
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
360 @classmethod
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
361 def new_from_dims(cls, n_I, n_K, n_J, rng = 8923402190):
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
362 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
363 Return a MeanCovRBM instance with randomly-initialized parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
364
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
365 :param n_I: input dimensionality
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
366 :param n_K: number of covariance hidden units
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
367 :param n_J: number of mean filters (linear)
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
368 :param rng: seed or numpy RandomState object to initialize params
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
369 """
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
370 if not hasattr(rng, 'randn'):
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
371 rng = np.random.RandomState(rng)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
372
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
373 def shrd(X,name):
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
374 return shared(X.astype(floatX), name=name)
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
375
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
376 # initialization taken from train_mcRBM.py
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
377 return cls(
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
378 U = shrd(0.02 * rng.randn(n_I, n_K),'U'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
379 W = shrd(0.05 * rng.randn(n_I, n_J),'W'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
380 a = shrd(np.ones(n_I)*(0),'a'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
381 b = shrd(np.ones(n_K)*2,'b'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
382 c = shrd(np.ones(n_J)*(-2),'c'))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
383
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
384 def __getstate__(self):
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
385 # unpack shared containers, which may have references to Theano stuff and are not a
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
386 # long-term stable data type.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
387 return dict(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
388 U = self.U.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
389 W = self.W.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
390 b = self.b.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
391 c = self.c.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
392
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
393 def __setstate__(self, dct):
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
394 d = dict(dct)
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
395 for key in ['U', 'W', 'a', 'b', 'c']:
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
396 d[key] = shared(d[key], name=key)
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
397 self.__init__(**d)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
398
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
399 if __name__ == '__main__':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
400
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
401 print >> sys.stderr, "TODO: use P matrix (aka FH matrix)"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
402
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
403 dataset='MAR'
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
404 if dataset == 'MAR':
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
405 R,C= 21,5
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
406 n_patches=10240
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
407 demodata = scipy.io.loadmat(os.path.join(pylearn.datasets.config.data_root(),'image_patches', 'mcRBM', 'training_colorpatches_16x16_demo.mat'))
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
408 else:
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
409 R,C= 16,16 # the size of image patches
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
410 n_patches=100000
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
411
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
412 n_train_iters=5000
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
413
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
414 n_burnin_steps=10000
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
415
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
416 l1_penalty=1e-3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
417 no_l1_epochs = 10
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
418 effective_l1_penalty=0.0
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
419
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
420 epoch_size=n_patches
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
421 batchsize = 128
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
422 lr = 0.075 / batchsize
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
423 s_lr = TT.scalar()
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
424 s_l1_penalty=TT.scalar()
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
425 n_K=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
426 n_F=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
427 n_J=100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
428
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
429 rbm = MeanCovRBM.new_from_dims(n_I=R*C, n_K=n_K, n_J=n_J)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
430
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
431 sampler = sampler(rbm, n_particles=batchsize)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
432
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
433 def l2(X):
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
434 return numpy.sqrt((X**2).sum())
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
435 def tile(X, fname):
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
436 if dataset == 'MAR':
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
437 X = np.dot(X, demodata['invpcatransf'].T)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
438 R=16
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
439 C=16
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
440 #X = X.reshape((X.shape[0], 3, 16, 16)).transpose([0,2,3,1]).copy()
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
441 X = (X[:,:256], X[:,256:512], X[:,512:], None)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
442 _img = image_tiling.tile_raster_images(X,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
443 img_shape=(R,C),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
444 min_dynamic_range=1e-2)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
445 image_tiling.save_tiled_raster_images(_img, fname)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
446 #print "Burning in..."
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
447 #for burnin in xrange(n_burnin_steps):
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
448 #sampler.simulate()
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
449
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
450 if 0:
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
451 print "Just SAMPLING..."
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
452 for jj in xrange(n_burnin_steps):
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
453 if 0 == jj % 100:
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
454 tile(sampler.positions[0].value, "sampler_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
455 tile(numpy.random.randn(100, 105), "random_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
456 print "burning in... ", jj
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
457 sys.stdout.flush()
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
458 sampler.simulate()
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
459
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
460 sys.exit()
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
461
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
462 batch_idx = TT.iscalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
463
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
464 if dataset == 'MAR':
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
465 op = TensorFnDataset(floatX,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
466 bcast=(False,),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
467 fn=load_mcRBM_demo_patches,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
468 single_shape=(105,))
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
469 train_batch = op((batch_idx * batchsize + np.arange(batchsize))%n_patches)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
470 else:
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
471 from pylearn.dataset_ops import image_patches
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
472 train_batch = image_patches.image_patches(
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
473 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
474 dims = (n_patches,R,C),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
475 center=True,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
476 unitvar=True,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
477 dtype=floatX,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
478 rasterized=True)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
479
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
480 imgs_fn = function([batch_idx], outputs=train_batch)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
481
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
482 grads = contrastive_gradient(rbm,
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
483 pos_v=train_batch,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
484 neg_v=sampler.positions[0],
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
485 U_l1_penalty=s_l1_penalty,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
486 W_l1_penalty=s_l1_penalty)
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
487 sgd_ups = sgd_updates(
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
488 rbm.params,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
489 grads,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
490 lr=[2*s_lr, .2*s_lr, .02*s_lr, .1*s_lr, .02*s_lr ])
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
491
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
492 learn_fn = function([batch_idx, s_lr, s_l1_penalty],
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
493 outputs=[
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
494 grads[0].norm(2),
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
495 (sgd_ups[0][1] - sgd_ups[0][0]).norm(2),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
496 (sgd_ups[1][1] - sgd_ups[1][0]).norm(2),
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
497 ],
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
498 updates = sgd_ups)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
499 theano.printing.pydotprint(function([batch_idx, s_l1_penalty], grads[0]), 'grads0.png')
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
500
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
501 print "Learning..."
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
502 normVF=1
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
503 last_epoch = -1
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
504 for jj in xrange(n_train_iters):
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
505 epoch = jj*batchsize / epoch_size
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
506
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
507 print_jj = epoch != last_epoch
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
508 last_epoch = epoch
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
509
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
510 if epoch > 10:
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
511 break
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
512
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
513 if print_jj:
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
514 tile(imgs_fn(jj), "imgs_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
515 tile(sampler.positions[0].value, "sample_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
516 tile(rbm.U.value.T, "U_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
517 tile(rbm.W.value.T, "W_%06i.png"%jj)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
518
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
519 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
520
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
521 print 'l2(U)', l2(rbm.U.value),
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
522 print 'l2(W)', l2(rbm.W.value)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
523
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
524 print 'U min max', rbm.U.value.min(), rbm.U.value.max(),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
525 print 'W min max', rbm.W.value.min(), rbm.W.value.max(),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
526 print 'a min max', rbm.a.value.min(), rbm.a.value.max(),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
527 print 'b min max', rbm.b.value.min(), rbm.b.value.max(),
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
528 print 'c min max', rbm.c.value.min(), rbm.c.value.max()
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
529
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
530 print 'parts min', sampler.positions[0].value.min(),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
531 print 'max',sampler.positions[0].value.max(),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
532 print 'HMC step', sampler.stepsize,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
533 print 'arate', sampler.avg_acceptance_rate
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
534
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
535 sampler.simulate()
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
536
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
537 l2_of_Ugrad = learn_fn(jj,
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
538 lr/max(1, jj/(20*epoch_size/batchsize)),
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
539 effective_l1_penalty)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
540
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
541 if print_jj:
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
542 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
543 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
544 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
545 #print 'FE+', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
546 #print 'FE+[0]', float(l2_of_Ugrad[3]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
547 #print 'FE+[1]', float(l2_of_Ugrad[4]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
548 #print 'FE+[2]', float(l2_of_Ugrad[5]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
549 #print 'FE+[3]', float(l2_of_Ugrad[6])
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
550
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
551 if jj == no_l1_epochs * epoch_size/batchsize:
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
552 print "Activating L1 weight decay"
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
553 effective_l1_penalty = 1e-3
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
554
992
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
555 # weird normalization technique...
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
556 # It constrains all the columns of the matrix to have the same length
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
557 # But the matrix itself is re-scaled to have an arbitrary abslute size.
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
558 U = rbm.U.value
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
559 U_norms = np.sqrt((U*U).sum(axis=0))
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
560 assert len(U_norms) == n_F
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
561 normVF = .95 * normVF + .05 * np.mean(U_norms)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 991
diff changeset
562 rbm.U.value = rbm.U.value * normVF/U_norms
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
563