Mercurial > pylearn

annotate pylearn/algorithms/mcRBM.py @ 1275:f0129e37a8ef

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
mcRBM - changed params from lambda to method for pickling
author James Bergstra <bergstrj@iro.umontreal.ca>
date Wed, 08 Sep 2010 13:18:13 -0400
parents 7bb5dd98e671
children 1817485d586d
rev   line source
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
8 and performs one of the experiments on CIFAR-10 discussed in that paper. There are some minor
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
9 discrepancies between the paper and the accompanying code (train_mcRBM.py), and the
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
10 accompanying code has been taken to be correct in those cases because I couldn't get things to
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
11 work otherwise.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
30 Version in paper
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
31 ----------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
32
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
33 Full Energy of the Mean and Covariance RBM, with
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
40 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
41 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i (U_{if} v_i) / |U_{.f}|*|v| )^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
42 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
43 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
44 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
45 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
46
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
47 For the energy function to correspond to a probability distribution, P must be non-positive. P
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
48 is initialized to be a diagonal, and in our experience it can be left as such because even in
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
49 the paper it has a very low learning rate, and is only allowed to be updated after the filters
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
50 in U are learned (in effect).
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
51
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
52 Version in published train_mcRBM code
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
53 -------------------------------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
54
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
55 The train_mcRBM file implements learning in a similar but technically different Energy function:
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 E (v, h, g) =
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
58 - 0.5 \sum_f \sum_k P_{fk} h_k (\sum_i U_{if} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
64 There are two differences with respect to the paper:
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
65
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
66 - 'v' is not normalized by its length, but rather it is normalized to have length close to
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
67 the square root of the number of its components. The variable called 'small' that
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
68 "avoids division by zero" is orders larger than machine precision, and is on the order of
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
69 the normalized sum-of-squares, so I've included it in the Energy function.
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
70
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
71 - 'U' is also not normalized by its length. U is initialized to have columns that are
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
72 shorter than unit-length (approximately 0.2 with the 105 principle components in the
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
73 train_mcRBM data). During training, the columns of U are constrained manually to have
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
74 equal lengths (see the use of normVF), but Euclidean norm is allowed to change. During
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
75 learning it quickly converges towards 1 and then exceeds 1. It does not seem like this
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
76 column-wise normalization of U is justified by maximum-likelihood, I have no intuition
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
77 for why it is used.
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
78
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
80 Version in this code
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
81 --------------------
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
82
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
83 This file implements the same algorithm as the train_mcRBM code, except that the P matrix is
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
84 omitted for clarity, and replaced analytically with a negative identity matrix.
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
85
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
86 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
87 + 0.5 \sum_k h_k (\sum_i U_{ik} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
88 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
89 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
90 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
91 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
92
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
93
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
94
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
115 Matrices are generally layed out and accessed according to a C-order convention.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118
984
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
119 #
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
120 # WORKING NOTES
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
121 # THIS DERIVATION IS BASED ON THE ** PAPER ** ENERGY FUNCTION
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
122 # NOT THE ENERGY FUNCTION IN THE CODE!!!
5badf36a6daf mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 979
diff changeset
123 #
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127 #
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
128 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
129 # E (v, h, g) =
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
130 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
131 # - \sum_k b_k h_k
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
132 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
133 # - \sum_j \sum_i W_{ij} g_j v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
134 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
135 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
136 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
137 #
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 # - \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
148 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
149 # - \sum_i a_i v_i ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
150 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
151 # Get rid of double negs in exp
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
156 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
157 # + \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
158 # + \sum_j c_j g_j))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
159 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
160 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
161 # Break up log
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
163 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
170 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
171 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
172 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(..
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|* |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
175 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
176 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
179 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
180 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
181 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
184 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
185 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
186 # For negative-one-diagonal P this gives:
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
187 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
188 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (|U_{*k}|*|v|)))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
189 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
190 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
191 # - \sum_i a_i v_i
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
192
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
193 import sys, os, logging
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
194 import numpy as np
973
aa201f357d7b mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 972
diff changeset
195 import numpy
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
196
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
197 import theano
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
198 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
199 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
200 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
201
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
202 sharedX = lambda X, name : shared(numpy.asarray(X, dtype=floatX), name=name)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
203
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
204 import pylearn
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
205 #TODO: clean up the HMC_sampler code
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
206 #TODO: think of naming convention for acronyms + suffix?
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
207 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
208 from pylearn.io import image_tiling
999
c6d08a760960 added sgd_updates to gd/sgd.py. Modif mcRBM to use it.
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 998
diff changeset
209 from pylearn.gd.sgd import sgd_updates
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
210 import pylearn.dataset_ops.image_patches
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
211
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
212 ###########################################
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
213 #
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
214 # Candidates for factoring
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
215 #
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
216 ###########################################
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
217
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
218 def l1(X):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
219 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
220 :param X: TensorType variable
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
221
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
222 :rtype: TensorType scalar
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
223
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
224 :returns: the sum of absolute values of the terms in X
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
225
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
226 :math: \sum_i |X_i|
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
227
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
228 Where i is an appropriately dimensioned index.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
229
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
230 """
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
231 return abs(X).sum()
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
232
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
233 def l2(X):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
234 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
235 :param X: TensorType variable
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
236
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
237 :rtype: TensorType scalar
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
238
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
239 :returns: the sum of absolute values of the terms in X
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
240
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
241 :math: \sqrt{ \sum_i X_i^2 }
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
242
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
243 Where i is an appropriately dimensioned index.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
244
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
245 """
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
246 return TT.sqrt((X**2).sum())
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
247
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
248 def contrastive_cost(free_energy_fn, pos_v, neg_v):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
249 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
250 :param free_energy_fn: lambda (TensorType matrix MxN) -> TensorType vector of M free energies
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
251 :param pos_v: TensorType matrix MxN of M "positive phase" particles
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
252 :param neg_v: TensorType matrix MxN of M "negative phase" particles
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
253
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
254 :returns: TensorType scalar that's the sum of the difference of free energies
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
255
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
256 :math: \sum_i free_energy(pos_v[i]) - free_energy(neg_v[i])
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
257
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
258 """
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
259 return (free_energy_fn(pos_v) - free_energy_fn(neg_v)).sum()
988
fd243cb2bf0b mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 987
diff changeset
260
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
261 def contrastive_grad(free_energy_fn, pos_v, neg_v, wrt, other_cost=0):
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
262 """
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
263 :param free_energy_fn: lambda (TensorType matrix MxN) -> TensorType vector of M free energies
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
264 :param pos_v: positive-phase sample of visible units
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
265 :param neg_v: negative-phase sample of visible units
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
266 :param wrt: TensorType variables with respect to which we want gradients (similar to the
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
267 'wrt' argument to tensor.grad)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
268 :param other_cost: TensorType scalar
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
269
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
270 :returns: TensorType variables for the gradient on each of the 'wrt' arguments
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
271
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
273 :math: Cost = other_cost + \sum_i free_energy(pos_v[i]) - free_energy(neg_v[i])
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
274 :math: d Cost / dW for W in `wrt`
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
275
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
276
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
277 This function is similar to tensor.grad - it returns the gradient[s] on a cost with respect
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
278 to one or more parameters. The difference between tensor.grad and this function is that
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
279 the negative phase term (`neg_v`) is considered constant, i.e. d `Cost` / d `neg_v` = 0.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
280 This is desirable because `neg_v` might be the result of a sampling expression involving
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
281 some of the parameters, but the contrastive divergence algorithm does not call for
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
282 backpropagating through the sampling procedure.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
283
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
284 Warning - if other_cost depends on pos_v or neg_v and you *do* want to backpropagate from
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
285 the `other_cost` through those terms, then this function is inappropriate. In that case,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
286 you should call tensor.grad separately for the other_cost and add the gradient expressions
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
287 you get from ``contrastive_grad(..., other_cost=0)``
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
288
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
289 """
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
290 cost=contrastive_cost(free_energy_fn, pos_v, neg_v)
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
291 if other_cost:
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
292 cost = cost + other_cost
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
293 return theano.tensor.grad(cost,
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
294 wrt=wrt,
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
295 consider_constant=[neg_v])
973
aa201f357d7b mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 972
diff changeset
296
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
297 ###########################################
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
298 #
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
299 # Expressions that are mcRBM-specific
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
300 #
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
301 ###########################################
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
302
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
303 class mcRBM(object):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
304 """Light-weight class that provides the math related to inference
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
305
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
306 Attributes:
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
307
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
308 - U - the covariance filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
309 - W - the mean filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
310 - a - the visible bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
311 - b - the covariance bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
312 - c - the mean bias (theano shared variable)
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
313
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
314 """
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
315 def __init__(self, U, W, a, b, c):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
316 self.U = U
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
317 self.W = W
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
318 self.a = a
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
319 self.b = b
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
320 self.c = c
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
321
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
322 def hidden_cov_units_preactivation_given_v(self, v, small=0.5):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
323 """Return argument to the sigmoid that would give mean of covariance hid units
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
324
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
325 See the math at the top of this file for what 'adjusted' means.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
326
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
327 return b - 0.5 * dot(adjusted(v), U)**2
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
328 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
329 unit_v = v / (TT.sqrt(TT.mean(v**2, axis=1)+small)).dimshuffle(0,'x') # adjust row norm
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
330 return self.b - 0.5 * dot(unit_v, self.U)**2
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
331
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
332 def free_energy_terms_given_v(self, v):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
333 """Returns theano expression for the terms that are added to form the free energy of
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
334 visible vector `v` in an mcRBM.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
335
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
336 1. Free energy related to covariance hiddens
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
337 2. Free energy related to mean hiddens
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
338 3. Free energy related to L2-Norm of `v`
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
339 4. Free energy related to projection of `v` onto biases `a`
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
340 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
341 t0 = -TT.sum(TT.nnet.softplus(self.hidden_cov_units_preactivation_given_v(v)),axis=1)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
342 t1 = -TT.sum(TT.nnet.softplus(self.c + dot(v,self.W)), axis=1)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
343 t2 = 0.5 * TT.sum(v**2, axis=1)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
344 t3 = -TT.dot(v, self.a)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
345 return [t0, t1, t2, t3]
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
346
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
347 def free_energy_given_v(self, v):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
348 """Returns theano expression for free energy of visible vector `v` in an mcRBM
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
349 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
350 return TT.add(*self.free_energy_terms_given_v(v))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
351
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
352 def expected_h_g_given_v(self, v):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
353 """Returns tuple (`h`, `g`) of theano expression conditional expectations in an mcRBM.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
354
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
355 `h` is the conditional on the covariance units.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
356 `g` is the conditional on the mean units.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
357
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
358 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
359 h = TT.nnet.sigmoid(self.hidden_cov_units_preactivation_given_v(v))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
360 g = nnet.sigmoid(self.c + dot(v,self.W))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
361 return (h, g)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
362
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
363 def n_visible_units(self):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
364 """Return the number of visible units of this RBM
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
365
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
366 For an RBM made from shared variables, this will return an integer,
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
367 for a purely symbolic RBM this will return a theano expression.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
368
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
369 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
370 try:
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
371 return self.W.value.shape[0]
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
372 except AttributeError:
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
373 return self.W.shape[0]
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
374
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
375 def sampler(self, n_particles, n_visible=None, rng=7823748):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
376 """Return an `HMC_sampler` that will draw samples from the distribution over visible
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
377 units specified by this RBM.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
378
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
379 :param n_particles: this many parallel chains will be simulated.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
380 :param rng: seed or numpy RandomState object to initialize particles, and to drive the simulation.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
381 """
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
382 if not hasattr(rng, 'randn'):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
383 rng = np.random.RandomState(rng)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
384 if n_visible is None:
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
385 n_visible = self.n_visible_units()
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
386 rval = HMC_sampler.new_from_shared_positions(
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
387 shared_positions = sharedX(
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
388 rng.randn(
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
389 n_particles,
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
390 n_visible),
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
391 name='particles'),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
392 energy_fn=self.free_energy_given_v,
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
393 seed=int(rng.randint(2**30)))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
394 return rval
997
71b0132b694a mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 996
diff changeset
395
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
396 def as_feedforward_layer(self, v):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
397 """Return a dictionary with keys: inputs, outputs and params
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
398
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
399 The inputs is [v]
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
400
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
401 The outputs is :math:`[E[h|v], E[g|v]]` where `h` is the covariance hidden units and `g` is
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
402 the mean hidden units.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
403
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
404 The params are ``[U, W, b, c]``, the model parameters that enter into the conditional
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
405 expectations.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
406
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
407 :TODO: add an optional parameter to return only one of the expections.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
408
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
409 """
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
410 return dict(
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
411 inputs = [v],
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
412 outputs = list(self.expected_h_g_given_v(v)),
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
413 params = [self.U, self.W, self.b, self.c],
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
414 )
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
415
1275
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
416 def params(self):
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
417 """Return the elements of [U,W,a,b,c] that are shared variables
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
418
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
419 WRITEME : a *prescriptive* definition of this method suitable for mention in the API
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
420 doc.
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
421
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
422 """
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
423 return list(self._params)
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
424
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
425 @classmethod
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
426 def alloc(cls, n_I, n_K, n_J, rng = 8923402190,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
427 U_range=0.02,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
428 W_range=0.05,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
429 a_ival=0,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
430 b_ival=2,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
431 c_ival=-2):
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
432 """
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
433 Return a MeanCovRBM instance with randomly-initialized shared variable parameters.
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
434
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
435 :param n_I: input dimensionality
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
436 :param n_K: number of covariance hidden units
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
437 :param n_J: number of mean filters (linear)
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
438 :param rng: seed or numpy RandomState object to initialize parameters
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
439
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
440 :note:
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
441 Constants for initial ranges and values taken from train_mcRBM.py.
995
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
442 """
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
443 if not hasattr(rng, 'randn'):
68ca3ea34e72 mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 994
diff changeset
444 rng = np.random.RandomState(rng)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
445
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
446 rval = cls(
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
447 U = sharedX(U_range * rng.randn(n_I, n_K),'U'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
448 W = sharedX(W_range * rng.randn(n_I, n_J),'W'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
449 a = sharedX(np.ones(n_I)*a_ival,'a'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
450 b = sharedX(np.ones(n_K)*b_ival,'b'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
451 c = sharedX(np.ones(n_J)*c_ival,'c'),)
1275
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
452 rval._params = [rval.U, rval.W, rval.a, rval.b, rval.c]
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
453 return rval
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
454
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
455 class mcRBMTrainer(object):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
456 """Light-weight class encapsulating math for mcRBM training
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
457
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
458 Attributes:
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
459 - rbm - an mcRBM instance
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
460 - sampler - an HMC_sampler instance
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
461 - normVF - geometrically updated norm of U matrix columns (shared var)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
462 - learn_rate - SGD learning rate [un-annealed]
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
463 - learn_rate_multipliers - the learning rates for each of the parameters of the rbm (in
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
464 order corresponding to what's returned by ``rbm.params()``)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
465 - l1_penalty - float or TensorType scalar to modulate l1 penalty of rbm.U and rbm.W
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
466 - iter - number of cd_updates (shared var) - used to anneal the effective learn_rate
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
467 - lr_anneal_start - scalar or TensorType scalar - iter at which time to start decreasing
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
468 the learning rate proportional to 1/iter
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
469
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
470 """
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
471 # TODO: accept a GD algo as an argument?
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
472 @classmethod
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
473 def alloc(cls, rbm, visible_batch, batchsize, initial_lr=0.075, rng=234,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
474 l1_penalty=0,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
475 learn_rate_multipliers=[2, .2, .02, .1, .02],
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
476 lr_anneal_start=2000,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
477 ):
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
478
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
479 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
480 :param rbm: mcRBM instance to train
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
481 :param visible_batch: TensorType variable for training data
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
482 :param batchsize: the number of rows in visible_batch
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
483 :param initial_lr: the learning rate (may be annealed)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
484 :param rng: seed or RandomState to initialze PCD sampler
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
485 :param l1_penalty: see class doc
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
486 :param learn_rate_multipliers: see class doc
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
487 :param lr_anneal_start: see class doc
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
488 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
489 #TODO: :param lr_anneal_iter: the iteration at which 1/t annealing will begin
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
490
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
491 #TODO: get batchsize from visible_batch??
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
492 # allocates shared var for negative phase particles
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
493
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
494
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
495 # TODO: should normVF be initialized to match the size of rbm.U ?
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
496
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
497 return cls(
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
498 rbm=rbm,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
499 visible_batch=visible_batch,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
500 sampler=rbm.sampler(batchsize, rng=rng),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
501 normVF=sharedX(1.0, 'normVF'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
502 learn_rate=sharedX(initial_lr/batchsize, 'learn_rate'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
503 iter=sharedX(0, 'iter'),
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
504 l1_penalty=l1_penalty,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
505 learn_rate_multipliers=learn_rate_multipliers,
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
506 lr_anneal_start=lr_anneal_start)
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
507
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
508 def __init__(self, **kwargs):
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
509 self.__dict__.update(kwargs)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
510
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
511 def normalize_U(self, new_U):
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
512 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
513 :param new_U: a proposed new value for rbm.U
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
514
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
515 :returns: a pair of TensorType variables:
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
516 a corrected new value for U, and a new value for self.normVF
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
517
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
518 This is a weird normalization procedure, but the sample code for the paper has it, and
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
519 it seems to be important.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
520 """
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
521 U_norms = TT.sqrt((new_U**2).sum(axis=0))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
522 new_normVF = .95 * self.normVF + .05 * TT.mean(U_norms)
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
523 return (new_U * new_normVF / U_norms), new_normVF
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
524
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
525 def contrastive_grads(self):
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
526 """Return the contrastive divergence gradients on the parameters of self.rbm """
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
527 return contrastive_grad(
1270
d38cb039c662 debugging mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1267
diff changeset
528 free_energy_fn=self.rbm.free_energy_given_v,
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
529 pos_v=self.visible_batch,
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
530 neg_v=self.sampler.positions,
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
531 wrt = self.rbm.params(),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
532 other_cost=(l1(self.rbm.U)+l1(self.rbm.W)) * self.l1_penalty)
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
533
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
534 def cd_updates(self):
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
535 """
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
536 Return a dictionary of shared variable updates that implements contrastive divergence
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
537 learning by stochastic gradient descent with an annealed learning rate.
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
538 """
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
539
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
540 grads = self.contrastive_grads()
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
541
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
542 # contrastive divergence updates
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
543 # TODO: sgd_updates is a particular optization algo (others are possible)
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
544 # parametrize so that algo is plugin
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
545 # the normalization normVF might be sgd-specific though...
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
546
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
547 # TODO: when sgd has an annealing schedule, this should
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
548 # go through that mechanism.
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
549
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
550 lr = TT.clip(
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
551 self.learn_rate * TT.cast(self.lr_anneal_start / (self.iter+1), floatX),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
552 0.0, #min
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
553 self.learn_rate) #max
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
554
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
555 ups = dict(sgd_updates(
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
556 self.rbm.params(),
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
557 grads,
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
558 stepsizes=[a*lr for a in self.learn_rate_multipliers]))
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
559
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
560 ups[self.iter] = self.iter + 1
979
2a53384d9742 mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 978
diff changeset
561
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
562 # sampler updates
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
563 ups.update(dict(self.sampler.updates()))
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
564
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
565 # add trainer updates (replace CD update of U)
1272
ba25c6e4f55d mcRBM working with whole learning algo in theano
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1270
diff changeset
566 ups[self.rbm.U], ups[self.normVF] = self.normalize_U(ups[self.rbm.U])
1267
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
567
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
568 return ups
075c193afd1b refactoring mcRBM
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1000
diff changeset
569
1000
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
570 if __name__ == '__main__':
d4a14c6c36e0 mcRBM - post code-review #1 with Guillaume
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 999
diff changeset
571 import pylearn.algorithms.tests.test_mcRBM
1273
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
572 rbm,smplr = pylearn.algorithms.tests.test_mcRBM.test_reproduce_ranzato_hinton_2010(
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
573 as_unittest=False,
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
574 n_train_iters=10)
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
575 import cPickle
1275
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
576 print ''
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
577 print 'Saving rbm...'
1273
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
578 cPickle.dump(rbm, open('mcRBM.rbm.pkl', 'w'), -1)
1275
f0129e37a8ef mcRBM - changed params from lambda to method for pickling
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1273
diff changeset
579 print 'Saving sampler...'
1273
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
580 cPickle.dump(smplr, open('mcRBM.smplr.pkl', 'w'), -1)
7bb5dd98e671 mcRBM - added hack to main script to pickle final model and sampler
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1272
diff changeset
581