annotate pylearn/algorithms/mcRBM.py @ 972:0b392d1401c5

mcRBM - adding math and comments
author James Bergstra <bergstrj@iro.umontreal.ca>
date Mon, 23 Aug 2010 15:59:21 -0400
parents 90e11d5d0a41
children aa201f357d7b
rev   line source
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8 and performs one of the experiments on CIFAR-10 discussed in that paper.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 Full Energy of mean and Covariance RBM, with
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 :
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 E (v, h, g) =
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
43 For the energy function to correspond to a probability distribution, P must be non-positive.
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
44
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 - `P`, a matrix (probably sparse) of pooling (F x K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67 Matrices are generally layed out according to a C-order convention.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 #
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
75 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
76 # E (v, h, g) =
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
77 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
78 # - \sum_k b_k h_k
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
79 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
80 # - \sum_j \sum_i W_{ij} g_j v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
81 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
82 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
83 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
84 #
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94 # - \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
95 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
96 # - \sum_i a_i v_i ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
97 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
98 # Get rid of double negs in exp
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 # + \sum_j \sum_i W_{ij} g_j v_i
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
105 # + \sum_j c_j g_j))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
106 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
107 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
108 # Break up log
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
117 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
118 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
119 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(..
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|* |v|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
123 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
126 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
127 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
128 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)))
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
129 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
130 # + 0.5 \sum_i v_i^2
972
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
131 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
132 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
133 # For negative-one-diagonal P this gives:
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
134 #
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
135 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (|U_{*k}|*|v|)))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
136 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
137 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 967
diff changeset
138 # - \sum_i a_i v_i
967
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 import sys
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 import logging
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 import numpy as np
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 import theano.sparse #installs the sparse shared var handler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149 from pylearn.io import image_tiling
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
150
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
151 from sparse_coding import numpy_project_onto_ball
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 #TODO: This should be in the nnet part of the library
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154 def sgd_updates(params, grads, lr):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 try:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
156 float(lr)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
157 lr = [lr for p in params]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
158 except TypeError:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
159 pass
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
160 updates = [(p, p + plr * gp) for (plr, p, gp) in zip(lr, params, grads)]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
161 return updates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
163 def as_shared(x, name=None, dtype=floatX):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164 if hasattr(x, 'type'):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 return x
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 if 'float' in str(x.dtype):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 return shared(x.astype(floatX), name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
170 return shared(x, name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
171
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
172 def hidden_cov_units_preactivation_given_v(rbm, v, small=1e-8):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 (U,W,a,b,c) = rbm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 unit_v = v / (TT.sqrt(TT.sum(v**2, axis=1))+small).dimshuffle(0,'x') # unit rows
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
175 unit_U = U # assuming unit cols!
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
176 #unit_U = U / (TT.sqrt(TT.sum(U**2, axis=0))+small) #unit cols
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177 return b - 0.5 * dot(unit_v, unit_U)**2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
179 def free_energy_given_v(rbm, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
180 """Returns theano expression for free energy of visible vector `v` in an mcRBM
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
181
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 An mcRBM is parametrized
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 by `U`, `W`, `b`, `c`.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
184 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
185
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
186
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
187 The free energy of v is what we need for learning and hybrid Monte-carlo negative-phase
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
188 sampling.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
189
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
190 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
191 U, W, a, b, c = rbm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
192
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
193 t0 = -TT.sum(TT.log1p(TT.exp(hidden_cov_units_preactivation_given_v(rbm, v))),axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
194 t1 = -TT.sum(TT.log1p(TT.exp(c + dot(v,W))), axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
195 t2 = 0.5 * TT.sum(v**2, axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
196 t3 = -TT.dot(v, a)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
197 return t0 + t1 + t2 + t3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
198
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
199 def expected_h_g_given_v(P, U, W, b, c, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
200 """Returns theano expression conditional expectations (`h`, `g`) in an mcRBM.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
201
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
202 An mcRBM is parametrized
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
203 by `U`, `W`, `b`, `c`.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
204 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
205
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
206
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
207 The conditional E[h, g | v] is what we need to classify images.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
208 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
209 raise NotImplementedError()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
210
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
211 #TODO: check to see if these args should be negated?
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
212
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
213 if P is None:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
214 h = nnet.sigmoid(b + 0.5 * cosines(v,U))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
215 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
216 h = nnet.sigmoid(b + 0.5 * dot(cosines(v,U), P))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
217 g = nnet.sigmoid(c + dot(v,W))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
218 return (h, g)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220 class MeanCovRBM(object):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221 """Container for mcRBM parameters that gives more convenient access to mcRBM methods.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
222 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
223
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
224 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
225
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
226 n_visible = property(lambda s: s.W.value.shape[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
227
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
228 def __init__(self, U, W, a, b, c):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
229 self.U = as_shared(U, 'U')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
230 self.W = as_shared(W, 'W')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
231 self.a = as_shared(a, 'a')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
232 self.b = as_shared(b, 'b')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
233 self.c = as_shared(c, 'c')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
234
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
235 assert self.b.type.dtype == 'float32'
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
236
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
237 @classmethod
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
238 def new_from_dims(cls,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
239 n_I, # input dimensionality
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
240 n_K, # number of covariance hidden units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
241 n_F, # number of covariance filters (squared)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
242 n_J, # number of mean filters (linear)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
243 seed = 8923402190,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
244 ):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
245 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
246 Return a MeanCovRBM instance with randomly-initialized parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
247 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
248
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
249
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
250 if 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
251 if P_init == 'diag':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
252 if n_K != n_F:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
253 raise ValueError('cannot use diagonal initialization of non-square P matrix')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
254 import scipy.sparse
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
255 P = -scipy.sparse.identity(n_K).tocsr()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
256 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
257 raise NotImplementedError()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
258
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
259 rng = np.random.RandomState(seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
260
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
261 # initialization taken from Marc'Aurelio
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
262
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
263 return cls(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
264 U = numpy_project_onto_ball(rng.randn(n_I, n_F).T).T,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
265 W = rng.randn(n_I, n_J)/np.sqrt((n_I+n_J)/2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
266 a = np.ones(n_I)*(-2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
267 b = np.ones(n_K)*2,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
268 c = np.zeros(n_J),)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
269
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
270 def __getstate__(self):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
271 # unpack shared containers, which may have references to Theano stuff
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
272 # and are not a long-term stable data type.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
273 return dict(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
274 U = self.U.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
275 W = self.W.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
276 b = self.b.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
277 c = self.c.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
278
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
279 def __setstate__(self, dct):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
280 self.__init__(**dct) # calls as_shared on pickled arrays
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
281
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
282 def hmc_sampler(self, n_particles=100, seed=7823748):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
283 return HMC_sampler(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
284 positions = [as_shared(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
285 np.random.RandomState(seed^20893).rand(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
286 n_particles,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
287 self.n_visible ))],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
288 energy_fn = lambda p : self.free_energy_given_v(p[0]),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
289 seed=seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
290
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
291 def free_energy_given_v(self, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
292 return free_energy_given_v(self.params, v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
293
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
294 def contrastive_gradient(self, pos_v, neg_v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
295 """Return a list of gradient expressions for self.params
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
296
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
297 :param pos_v: positive-phase sample of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
298 :param neg_v: negative-phase sample of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
299 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
300 pos_FE = self.free_energy_given_v(pos_v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
301 neg_FE = self.free_energy_given_v(neg_v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
302
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
303 gpos_FE = theano.tensor.grad(pos_FE.sum(), self.params)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
304 gneg_FE = theano.tensor.grad(neg_FE.sum(), self.params)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
305 return [ gn - gp for (gp,gn) in zip(gpos_FE, gneg_FE)]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
306
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
307 if __name__ == '__main__':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
308
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
309 print >> sys.stderr, "TODO: use P matrix (aka FH matrix)"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
310
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
311 R,C= 8,8 # the size of image patches
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
312 l1_penalty=1e-3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
313 no_l1_epochs = 10
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
314
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
315 epoch_size=50000
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
316 batchsize = 128
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
317 lr = 0.075 / batchsize
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
318 s_lr = TT.scalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
319 n_K=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
320 n_F=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
321 n_J=100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
322
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
323 rbm = MeanCovRBM.new_from_dims(n_I=R*C,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
324 n_K=n_K,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
325 n_J=n_J,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
326 n_F=n_F,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
327 )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
328
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
329 sampler = rbm.hmc_sampler(n_particles=100)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
330
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
331 from pylearn.dataset_ops import image_patches
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
332
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
333 batch_idx = TT.iscalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
334 train_batch = image_patches.image_patches(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
335 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
336 dims = (1000,R,C),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
337 dtype=floatX,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
338 rasterized=True)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
339
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
340 grads = rbm.contrastive_gradient(pos_v=train_batch, neg_v=sampler.positions[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
341
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
342 learn_fn = function([batch_idx, s_lr],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
343 outputs=[
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
344 grads[0].norm(2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
345 rbm.U.norm(2)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
346 ],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
347 updates = sgd_updates(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
348 rbm.params,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
349 grads,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
350 lr=[2*s_lr, .2*s_lr, .02*s_lr, .1*s_lr, .02*s_lr ]))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
351
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
352 for jj in xrange(10000):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
353 sampler.simulate()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
354 l2_of_Ugrad = learn_fn(jj, lr/max(1, jj/(20*epoch_size/batchsize)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
355
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
356 if jj > no_l1_epochs * epoch_size/batchsize:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
357 rbm.U.value -= l1_penalty * np.sign(rbm.U.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
358 rbm.W.value -= l1_penalty * np.sign(rbm.W.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
359
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
360 if jj % 5 == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
361 rbm.U.value = numpy_project_onto_ball(rbm.U.value.T).T
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
362
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
363 if ((jj < 10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
364 or (jj < 100 and 0==jj%10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
365 or (jj < 1000 and 0==jj%100)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
366 or (jj < 10000 and 0==jj%1000)):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
367 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize), l2_of_Ugrad
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
368 print 'neg particles', sampler.positions[0].value.min(), sampler.positions[0].value.max()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
369 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
370 image_tiling.tile_raster_images(sampler.positions[0].value, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
371 "sample_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
372 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
373 image_tiling.tile_raster_images(rbm.U.value.T, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
374 "U_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
375 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
376 image_tiling.tile_raster_images(rbm.W.value.T, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
377 "W_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
378
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
379
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
380
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
381 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
382 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
383 # Marc'Aurelio Ranzato's code
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
384 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
385 ######################################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
386 # compute the value of the free energy at a given input
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
387 # F = - sum log(1+exp(- .5 FH (VF data/norm(data))^2 + bias_cov)) +...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
388 # - sum log(1+exp(w_mean data + bias_mean)) + ...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
389 # - bias_vis data + 0.5 data^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
390 # NOTE: FH is constrained to be positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
391 # (in the paper the sign is negative but the sign in front of it is also flipped)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
392 def compute_energy_mcRBM(data,normdata,vel,energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,t1,t2,t6,feat,featsq,feat_mean,length,lengthsq,normcoeff,small,num_vis):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
393 # normalize input data vectors
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
394 data.mult(data, target = t6) # DxP (nr input dims x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
395 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
396 lengthsq.mult(0.5, target = energy) # energy of quadratic regularization term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
397 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
398
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
399 lengthsq.add(small) # small prevents division by 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
400 # energy_j = \sum_i 0.5 data_ij ^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
401 # lengthsq_j = 1/ (\sum_i data_ij ^2 + small)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
402 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
403 # length_j = sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
404 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
405 # normcoef_j = 1/sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
406 data.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
407 # normdata is like data, but cols have unit L2 norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
408
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
409 ## potential
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
410 # covariance contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
411 cmt.dot(VF.T, normdata, target = feat) # HxP (nr factors x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
412 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
413
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
414 # featsq is the squared cosines (VF with data)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
415 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
416 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
417 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
418 cmt.exp(t1) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
419 t1.add(1, target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
420 cmt.log(t2)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
421 t2.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
422 energy.add_sums(t2, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
423 # mean contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
424 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
425 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
426 cmt.exp(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
427 feat_mean.add(1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
428 cmt.log(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
429 feat_mean.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
430 energy.add_sums(feat_mean, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
431 # visible bias term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
432 data.mult_by_col(bias_vis, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
433 t6.mult(-1) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
434 energy.add_sums(t6, axis=0) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
435 # kinetic
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
436 vel.mult(vel, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
437 energy.add_sums(t6, axis = 0, mult = .5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
438
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
439 ######################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
440 # mcRBM trainer: sweeps over the training set.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
441 # For each batch of samples compute derivatives to update the parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
442 # at the training samples and at the negative samples drawn calling HMC sampler.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
443 def train_mcRBM():
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
444
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
445 config = ConfigParser()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
446 config.read('input_configuration')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
447
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
448 verbose = config.getint('VERBOSITY','verbose')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
449
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
450 num_epochs = config.getint('MAIN_PARAMETER_SETTING','num_epochs')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
451 batch_size = config.getint('MAIN_PARAMETER_SETTING','batch_size')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
452 startFH = config.getint('MAIN_PARAMETER_SETTING','startFH')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
453 startwd = config.getint('MAIN_PARAMETER_SETTING','startwd')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
454 doPCD = config.getint('MAIN_PARAMETER_SETTING','doPCD')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
455
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
456 # model parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
457 num_fac = config.getint('MODEL_PARAMETER_SETTING','num_fac')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
458 num_hid_cov = config.getint('MODEL_PARAMETER_SETTING','num_hid_cov')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
459 num_hid_mean = config.getint('MODEL_PARAMETER_SETTING','num_hid_mean')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
460 apply_mask = config.getint('MODEL_PARAMETER_SETTING','apply_mask')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
461
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
462 # load data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
463 data_file_name = config.get('DATA','data_file_name')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
464 d = loadmat(data_file_name) # input in the format PxD (P vectorized samples with D dimensions)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
465 totnumcases = d["whitendata"].shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
466 d = d["whitendata"][0:floor(totnumcases/batch_size)*batch_size,:].copy()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
467 totnumcases = d.shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
468 num_vis = d.shape[1]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
469 num_batches = int(totnumcases/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
470 dev_dat = cmt.CUDAMatrix(d.T) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
471
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
472 # training parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
473 epsilon = config.getfloat('OPTIMIZER_PARAMETERS','epsilon')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
474 epsilonVF = 2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
475 epsilonFH = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
476 epsilonb = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
477 epsilonw_mean = 0.2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
478 epsilonb_mean = 0.1*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
479 weightcost_final = config.getfloat('OPTIMIZER_PARAMETERS','weightcost_final')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
480
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
481 # HMC setting
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
482 hmc_step_nr = config.getint('HMC_PARAMETERS','hmc_step_nr')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
483 hmc_step = 0.01
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
484 hmc_target_ave_rej = config.getfloat('HMC_PARAMETERS','hmc_target_ave_rej')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
485 hmc_ave_rej = hmc_target_ave_rej
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
486
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
487 # initialize weights
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
488 VF = cmt.CUDAMatrix(np.array(0.02 * np.random.randn(num_vis, num_fac), dtype=np.float32, order='F')) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
489 if apply_mask == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
490 FH = cmt.CUDAMatrix( np.array( np.eye(num_fac,num_hid_cov), dtype=np.float32, order='F') ) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
491 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
492 dd = loadmat('your_FHinit_mask_file.mat') # see CVPR2010paper_material/topo2D_3x3_stride2_576filt.mat for an example
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
493 FH = cmt.CUDAMatrix( np.array( dd["FH"], dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
494 bias_cov = cmt.CUDAMatrix( np.array(2.0*np.ones((num_hid_cov, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
495 bias_vis = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
496 w_mean = cmt.CUDAMatrix( np.array( 0.05 * np.random.randn(num_vis, num_hid_mean), dtype=np.float32, order='F') ) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
497 bias_mean = cmt.CUDAMatrix( np.array( -2.0*np.ones((num_hid_mean,1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
498
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
499 # initialize variables to store derivatives
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
500 VFinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
501 FHinc = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
502 bias_covinc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_cov, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
503 bias_visinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
504 w_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_hid_mean)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
505 bias_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_mean, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
506
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
507 # initialize temporary storage
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
508 data = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
509 normdata = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
510 negdataini = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
511 feat = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
512 featsq = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
513 negdata = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
514 old_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
515 new_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
516 gradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
517 normgradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
518 thresh = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
519 feat_mean = cmt.CUDAMatrix( np.array(np.empty((num_hid_mean, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
520 vel = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
521 length = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
522 lengthsq = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
523 normcoeff = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
524 if apply_mask==1: # this used to constrain very large FH matrices only allowing to change values in a neighborhood
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
525 dd = loadmat('your_FHinit_mask_file.mat')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
526 mask = cmt.CUDAMatrix( np.array(dd["mask"], dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
527 normVF = 1
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
528 small = 0.5
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
529
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
530 # other temporary vars
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
531 t1 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
532 t2 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
533 t3 = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
534 t4 = cmt.CUDAMatrix( np.array(np.empty((1,batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
535 t5 = cmt.CUDAMatrix( np.array(np.empty((1,1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
536 t6 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
537 t7 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
538 t8 = cmt.CUDAMatrix( np.array(np.empty((num_vis, num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
539 t9 = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
540 t10 = cmt.CUDAMatrix( np.array(np.empty((1,num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
541 t11 = cmt.CUDAMatrix( np.array(np.empty((1,num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
542
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
543 # start training
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
544 for epoch in range(num_epochs):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
545
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
546 print "Epoch " + str(epoch + 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
547
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
548 # anneal learning rates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
549 epsilonVFc = epsilonVF/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
550 epsilonFHc = epsilonFH/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
551 epsilonbc = epsilonb/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
552 epsilonw_meanc = epsilonw_mean/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
553 epsilonb_meanc = epsilonb_mean/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
554 weightcost = weightcost_final
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
555
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
556 if epoch <= startFH:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
557 epsilonFHc = 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
558 if epoch <= startwd:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
559 weightcost = 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
560
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
561 for batch in range(num_batches):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
562
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
563 # get current minibatch
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
564 data = dev_dat.slice(batch*batch_size,(batch + 1)*batch_size) # DxP (nr dims x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
565
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
566 # normalize input data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
567 data.mult(data, target = t6) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
568 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
569 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
570 lengthsq.add(small) # small avoids division by 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
571 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
572 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
573 data.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
574 ## compute positive sample derivatives
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
575 # covariance part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
576 cmt.dot(VF.T, normdata, target = feat) # HxP (nr facs x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
577 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
578 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
579 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
580 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
581 t1.apply_sigmoid(target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
582 cmt.dot(featsq, t2.T, target = FHinc) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
583 cmt.dot(FH,t2, target = t3) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
584 t3.mult(feat)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
585 cmt.dot(normdata, t3.T, target = VFinc) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
586 t2.sum(axis = 1, target = bias_covinc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
587 bias_covinc.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
588 # visible bias
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
589 data.sum(axis = 1, target = bias_visinc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
590 bias_visinc.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
591 # mean part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
592 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
593 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
594 feat_mean.apply_sigmoid() # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
595 feat_mean.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
596 cmt.dot(data, feat_mean.T, target = w_meaninc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
597 feat_mean.sum(axis = 1, target = bias_meaninc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
598
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
599 # HMC sampling: draw an approximate sample from the model
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
600 if doPCD == 0: # CD-1 (set negative data to current training samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
601 hmc_step, hmc_ave_rej = draw_HMC_samples(data,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
602 else: # PCD-1 (use previous negative data as starting point for chain)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
603 negdataini.assign(negdata)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
604 hmc_step, hmc_ave_rej = draw_HMC_samples(negdataini,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
605
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
606 # compute derivatives at the negative samples
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
607 # normalize input data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
608 negdata.mult(negdata, target = t6) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
609 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
610 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
611 lengthsq.add(small)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
612 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
613 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
614 negdata.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
615 # covariance part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
616 cmt.dot(VF.T, normdata, target = feat) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
617 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
618 cmt.dot(FH.T,featsq, target = t1) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
619 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
620 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
621 t1.apply_sigmoid(target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
622 FHinc.subtract_dot(featsq, t2.T) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
623 FHinc.mult(0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
624 cmt.dot(FH,t2, target = t3) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
625 t3.mult(feat)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
626 VFinc.subtract_dot(normdata, t3.T) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
627 bias_covinc.add_sums(t2, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
628 # visible bias
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
629 bias_visinc.add_sums(negdata, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
630 # mean part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
631 cmt.dot(w_mean.T, negdata, target = feat_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
632 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
633 feat_mean.apply_sigmoid() # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
634 w_meaninc.add_dot(negdata, feat_mean.T)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
635 bias_meaninc.add_sums(feat_mean, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
636
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
637 # update parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
638 VFinc.add_mult(VF.sign(), weightcost) # L1 regularization
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
639 VF.add_mult(VFinc, -epsilonVFc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
640 # normalize columns of VF: normalize by running average of their norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
641 VF.mult(VF, target = t8)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
642 t8.sum(axis = 0, target = t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
643 cmt.sqrt(t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
644 t10.sum(axis=1,target = t5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
645 t5.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
646 normVF = .95*normVF + (.05/num_fac) * t5.numpy_array[0,0] # estimate norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
647 t10.reciprocal()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
648 VF.mult_by_row(t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
649 VF.mult(normVF)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
650 bias_cov.add_mult(bias_covinc, -epsilonbc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
651 bias_vis.add_mult(bias_visinc, -epsilonbc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
652
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
653 if epoch > startFH:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
654 FHinc.add_mult(FH.sign(), weightcost) # L1 regularization
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
655 FH.add_mult(FHinc, -epsilonFHc/batch_size) # update
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
656 # set to 0 negative entries in FH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
657 FH.greater_than(0, target = t9)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
658 FH.mult(t9)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
659 if apply_mask==1:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
660 FH.mult(mask)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
661 # normalize columns of FH: L1 norm set to 1 in each column
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
662 FH.sum(axis = 0, target = t11)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
663 t11.reciprocal()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
664 FH.mult_by_row(t11)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
665 w_meaninc.add_mult(w_mean.sign(),weightcost)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
666 w_mean.add_mult(w_meaninc, -epsilonw_meanc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
667 bias_mean.add_mult(bias_meaninc, -epsilonb_meanc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
668
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
669 if verbose == 1:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
670 print "VF: " + '%3.2e' % VF.euclid_norm() + ", DVF: " + '%3.2e' % (VFinc.euclid_norm()*(epsilonVFc/batch_size)) + ", FH: " + '%3.2e' % FH.euclid_norm() + ", DFH: " + '%3.2e' % (FHinc.euclid_norm()*(epsilonFHc/batch_size)) + ", bias_cov: " + '%3.2e' % bias_cov.euclid_norm() + ", Dbias_cov: " + '%3.2e' % (bias_covinc.euclid_norm()*(epsilonbc/batch_size)) + ", bias_vis: " + '%3.2e' % bias_vis.euclid_norm() + ", Dbias_vis: " + '%3.2e' % (bias_visinc.euclid_norm()*(epsilonbc/batch_size)) + ", wm: " + '%3.2e' % w_mean.euclid_norm() + ", Dwm: " + '%3.2e' % (w_meaninc.euclid_norm()*(epsilonw_meanc/batch_size)) + ", bm: " + '%3.2e' % bias_mean.euclid_norm() + ", Dbm: " + '%3.2e' % (bias_meaninc.euclid_norm()*(epsilonb_meanc/batch_size)) + ", step: " + '%3.2e' % hmc_step + ", rej: " + '%3.2e' % hmc_ave_rej
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
671 sys.stdout.flush()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
672 # back-up every once in a while
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
673 if np.mod(epoch,10) == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
674 VF.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
675 FH.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
676 bias_cov.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
677 w_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
678 bias_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
679 bias_vis.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
680 savemat("ws_temp", {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array,'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
681 # final back-up
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
682 VF.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
683 FH.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
684 bias_cov.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
685 bias_vis.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
686 w_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
687 bias_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
688 savemat("ws_fac" + str(num_fac) + "_cov" + str(num_hid_cov) + "_mean" + str(num_hid_mean), {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array, 'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})