Mercurial > pylearn
annotate pylearn/algorithms/mcRBM.py @ 999:c6d08a760960
added sgd_updates to gd/sgd.py. Modif mcRBM to use it.
author | James Bergstra <bergstrj@iro.umontreal.ca> |
---|---|
date | Tue, 24 Aug 2010 17:01:09 -0400 |
parents | 8ba8b08e0442 |
children | d4a14c6c36e0 |
rev | line source |
---|---|
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
1 """ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
2 This file implements the Mean & Covariance RBM discussed in |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
3 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
4 Ranzato, M. and Hinton, G. E. (2010) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
5 Modeling pixel means and covariances using factored third-order Boltzmann machines. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
6 IEEE Conference on Computer Vision and Pattern Recognition. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
7 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
8 and performs one of the experiments on CIFAR-10 discussed in that paper. There are some minor |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
9 discrepancies between the paper and the accompanying code (train_mcRBM.py), and the |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
10 accompanying code has been taken to be correct in those cases because I couldn't get things to |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
11 work otherwise. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
12 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
13 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
14 Math |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
15 ==== |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
16 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
17 Energy of "covariance RBM" |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
18 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
19 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
20 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
21 "vector element f" "vector element f" |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
22 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
23 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
24 diagonal entries, so it is helpful to see this as a simpler equation: |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
25 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
26 E = \sum_f h_f ( \sum_i C_{if} v_i )^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
27 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
28 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
29 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
30 Version in paper |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
31 ---------------- |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
32 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
33 Full Energy of the Mean and Covariance RBM, with |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
34 :math:`h_k = h_k^{(c)}`, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
35 :math:`g_j = h_j^{(m)}`, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
36 :math:`b_k = b_k^{(c)}`, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
37 :math:`c_j = b_j^{(m)}`, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
38 :math:`U_{if} = C_{if}`, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
39 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
40 E (v, h, g) = |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
41 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i (U_{if} v_i) / |U_{.f}|*|v| )^2 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
42 - \sum_k b_k h_k |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
43 + 0.5 \sum_i v_i^2 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
44 - \sum_j \sum_i W_{ij} g_j v_i |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
45 - \sum_j c_j g_j |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
46 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
47 For the energy function to correspond to a probability distribution, P must be non-positive. P |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
48 is initialized to be a diagonal, and in our experience it can be left as such because even in |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
49 the paper it has a very low learning rate, and is only allowed to be updated after the filters |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
50 in U are learned (in effect). |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
51 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
52 Version in published train_mcRBM code |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
53 ------------------------------------- |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
54 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
55 The train_mcRBM file implements learning in a similar but technically different Energy function: |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
56 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
57 E (v, h, g) = |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
58 - 0.5 \sum_f \sum_k P_{fk} h_k (\sum_i U_{if} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
59 - \sum_k b_k h_k |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
60 + 0.5 \sum_i v_i^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
61 - \sum_j \sum_i W_{ij} g_j v_i |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
62 - \sum_j c_j g_j |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
63 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
64 There are two differences with respect to the paper: |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
65 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
66 - 'v' is not normalized by its length, but rather it is normalized to have length close to |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
67 the square root of the number of its components. The variable called 'small' that |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
68 "avoids division by zero" is orders larger than machine precision, and is on the order of |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
69 the normalized sum-of-squares, so I've included it in the Energy function. |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
70 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
71 - 'U' is also not normalized by its length. U is initialized to have columns that are |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
72 shorter than unit-length (approximately 0.2 with the 105 principle components in the |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
73 train_mcRBM data). During training, the columns of U are constrained manually to have |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
74 equal lengths (see the use of normVF), but Euclidean norm is allowed to change. During |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
75 learning it quickly converges towards 1 and then exceeds 1. It does not seem like this |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
76 column-wise normalization of U is justified by maximum-likelihood, I have no intuition |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
77 for why it is used. |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
78 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
79 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
80 Version in this code |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
81 -------------------- |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
82 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
83 This file implements the same algorithm as the train_mcRBM code, except that the P matrix is |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
84 omitted for clarity, and replaced analytically with a negative identity matrix. |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
85 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
86 E (v, h, g) = |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
87 + 0.5 \sum_k h_k (\sum_i U_{ik} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
88 - \sum_k b_k h_k |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
89 + 0.5 \sum_i v_i^2 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
90 - \sum_j \sum_i W_{ij} g_j v_i |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
91 - \sum_j c_j g_j |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
92 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
93 |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
94 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
95 Conventions in this file |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
96 ======================== |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
97 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
98 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
99 more convenient. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
100 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
101 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
102 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
103 Suppose we have |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
104 I input dimensions, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
105 F squared filters, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
106 J mean variables, and |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
107 K covariance variables. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
108 The mcRBM is parametrized by 5 variables: |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
109 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
110 - `U`, a matrix whose rows are visible covariance directions (I x F) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
111 - `W`, a matrix whose rows are visible mean directions (I x J) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
112 - `b`, a vector of hidden covariance biases (K) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
113 - `c`, a vector of hidden mean biases (J) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
114 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
115 Matrices are generally layed out and accessed according to a C-order convention. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
116 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
117 """ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
118 |
984
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
119 # |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
120 # WORKING NOTES |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
121 # THIS DERIVATION IS BASED ON THE ** PAPER ** ENERGY FUNCTION |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
122 # NOT THE ENERGY FUNCTION IN THE CODE!!! |
5badf36a6daf
mcRBM - added notes to leading comment
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
979
diff
changeset
|
123 # |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
124 # Free energy is the marginal energy of visible units |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
125 # Recall: |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
126 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
127 # |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
128 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
129 # E (v, h, g) = |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
130 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2 |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
131 # - \sum_k b_k h_k |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
132 # + 0.5 \sum_i v_i^2 |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
133 # - \sum_j \sum_i W_{ij} g_j v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
134 # - \sum_j c_j g_j |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
135 # - \sum_i a_i v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
136 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
137 # |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
138 # Derivation, in which partition functions are ignored. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
139 # |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
140 # E(v) = -\log(Q(v)) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
141 # = -\log( \sum_{h,g} Q(v,h,g)) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
142 # = -\log( \sum_{h,g} exp(-E(v,h,g))) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
143 # = -\log( \sum_{h,g} exp(- |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
144 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
145 # - \sum_k b_k h_k |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
146 # + 0.5 \sum_i v_i^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
147 # - \sum_j \sum_i W_{ij} g_j v_i |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
148 # - \sum_j c_j g_j |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
149 # - \sum_i a_i v_i )) |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
150 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
151 # Get rid of double negs in exp |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
152 # = -\log( \sum_{h} exp( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
153 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
154 # + \sum_k b_k h_k |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
155 # - 0.5 \sum_i v_i^2 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
156 # ) * \sum_{g} exp( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
157 # + \sum_j \sum_i W_{ij} g_j v_i |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
158 # + \sum_j c_j g_j)) |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
159 # - \sum_i a_i v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
160 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
161 # Break up log |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
162 # = -\log( \sum_{h} exp( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
163 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
164 # + \sum_k b_k h_k |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
165 # )) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
166 # -\log( \sum_{g} exp( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
167 # + \sum_j \sum_i W_{ij} g_j v_i |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
168 # + \sum_j c_j g_j ))) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
169 # + 0.5 \sum_i v_i^2 |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
170 # - \sum_i a_i v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
171 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
172 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(.. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
173 # = -\log(\sum_{h} exp( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
174 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|* |v|) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
175 # + \sum_k b_k h_k |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
176 # )) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
177 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j )) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
178 # + 0.5 \sum_i v_i^2 |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
179 # - \sum_i a_i v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
180 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
181 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|))) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
182 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j )) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
183 # + 0.5 \sum_i v_i^2 |
972
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
184 # - \sum_i a_i v_i |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
185 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
186 # For negative-one-diagonal P this gives: |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
187 # |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
188 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (|U_{*k}|*|v|))) |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
189 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j )) |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
190 # + 0.5 \sum_i v_i^2 |
0b392d1401c5
mcRBM - adding math and comments
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
967
diff
changeset
|
191 # - \sum_i a_i v_i |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
192 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
193 import sys |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
194 import logging |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
195 import numpy as np |
973
aa201f357d7b
mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
972
diff
changeset
|
196 import numpy |
988
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
197 |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
198 import theano |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
199 from theano import function, shared, dot |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
200 from theano import tensor as TT |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
201 floatX = theano.config.floatX |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
202 |
988
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
203 import pylearn |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
204 from pylearn.sampling.hmc import HMC_sampler |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
205 from pylearn.io import image_tiling |
999
c6d08a760960
added sgd_updates to gd/sgd.py. Modif mcRBM to use it.
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
998
diff
changeset
|
206 from pylearn.gd.sgd import sgd_updates |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
207 |
988
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
208 #TODO: This should be in the datasets folder |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
209 import pylearn.datasets.config |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
210 import pylearn.dataset_ops.image_patches |
988
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
211 from pylearn.dataset_ops.protocol import TensorFnDataset |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
212 from pylearn.dataset_ops.memo import memo |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
213 import pylearn |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
214 import scipy.io |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
215 import os |
fd243cb2bf0b
mcRBM - moved some things to the top of the file
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
987
diff
changeset
|
216 |
973
aa201f357d7b
mcRBM - added numpy import
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
972
diff
changeset
|
217 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
218 #TODO: This should be in the nnet part of the library |
989
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
219 def hidden_cov_units_preactivation_given_v(rbm, v, small=0.5): |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
220 """Return argument to the sigmoid that would give mean of covariance hid units |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
221 |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
222 See the math at the top of this file for what 'adjusted' means. |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
223 |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
224 return b - 0.5 * dot(adjusted(v), U)**2 |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
225 """ |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
226 (U,W,a,b,c) = rbm |
989
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
227 unit_v = v / (TT.sqrt(TT.mean(v**2, axis=1)+small)).dimshuffle(0,'x') # adjust row norm |
9e753ddcc320
mcRBM - cleaned up the preactivation function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
988
diff
changeset
|
228 return b - 0.5 * dot(unit_v, U)**2 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
229 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
230 def free_energy_terms_given_v(rbm, v): |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
231 """Returns theano expression for the terms that are added to form the free energy of |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
232 visible vector `v` in an mcRBM. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
233 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
234 1. Free energy related to covariance hiddens |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
235 2. Free energy related to mean hiddens |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
236 3. Free energy related to L2-Norm of `v` |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
237 4. Free energy related to projection of `v` onto biases `a` |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
238 """ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
239 U, W, a, b, c = rbm |
975
38e66e0da66a
mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
974
diff
changeset
|
240 t0 = -TT.sum(TT.nnet.softplus(hidden_cov_units_preactivation_given_v(rbm, v)),axis=1) |
38e66e0da66a
mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
974
diff
changeset
|
241 t1 = -TT.sum(TT.nnet.softplus(c + dot(v,W)), axis=1) |
38e66e0da66a
mcRBM - put softplus in directly for num. stability
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
974
diff
changeset
|
242 t2 = 0.5 * TT.sum(v**2, axis=1) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
243 t3 = -TT.dot(v, a) |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
244 return [t0, t1, t2, t3] |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
245 |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
246 def free_energy_given_v(rbm, v): |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
247 """Returns theano expression for free energy of visible vector `v` in an mcRBM |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
248 """ |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
249 return sum(free_energy_terms_given_v(rbm,v)) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
250 |
990
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
251 def contrastive_gradient(rbm, pos_v, neg_v, U_l1_penalty=0, W_l1_penalty=0): |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
252 """Return a list of gradient expressions for the rbm parameters |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
253 |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
254 :param pos_v: positive-phase sample of visible units |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
255 :param neg_v: negative-phase sample of visible units |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
256 :param U_l1_penalty: a scalar-valued multiplier on the L1 penalty on U |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
257 :param W_l1_penalty: a scalar-valued multiplier on the L1 penalty on W |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
258 """ |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
259 U, W, a, b, c = rbm |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
260 pos_FE = free_energy_given_v(rbm, pos_v) |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
261 neg_FE = free_energy_given_v(rbm, neg_v) |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
262 c0 = (pos_FE - neg_FE).sum() |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
263 c1 = abs(U).sum()*U_l1_penalty |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
264 c2 = abs(W).sum()*W_l1_penalty |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
265 cost = c0 + c1 + c2 |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
266 rval = theano.tensor.grad(cost, list(rbm)) |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
267 return rval |
e70e74464170
mcRBM - moved contrastive_gradient to global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
989
diff
changeset
|
268 |
991
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
269 def expected_h_g_given_v(rbm, v): |
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
270 """Returns tuple (`h`, `g`) of theano expression conditional expectations in an mcRBM. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
271 |
991
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
272 `h` is the conditional on the covariance units. |
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
273 `g` is the conditional on the mean units. |
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
274 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
275 """ |
991
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
276 (U, W, a, b, c) = rbm |
d68828c98c38
mcRBM - cleaned up expected hidden unit function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
990
diff
changeset
|
277 h = TT.nnet.sigmoid(hidden_cov_units_preactivation_given_v(rbm, v)) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
278 g = nnet.sigmoid(c + dot(v,W)) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
279 return (h, g) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
280 |
996
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
281 def n_visible_units(rbm): |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
282 """Return the number of visible units of this RBM |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
283 |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
284 For an RBM made from shared variables, this will return an integer, |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
285 for a purely symbolic RBM this will return a theano expression. |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
286 |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
287 """ |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
288 W = rbm[1] |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
289 try: |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
290 return W.value.shape[0] |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
291 except AttributeError: |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
292 return W.shape[0] |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
293 |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
294 |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
295 def sampler(rbm, n_particles, n_visible=None, rng=7823748): |
996
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
296 """Return an `HMC_sampler` that will draw samples from the distribution over visible |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
297 units specified by this RBM. |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
298 |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
299 :param n_particles: this many parallel chains will be simulated. |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
300 :param rng: seed or numpy RandomState object to initialize particles, and to drive the simulation. |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
301 """ |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
302 if not hasattr(rng, 'randn'): |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
303 rng = np.random.RandomState(rng) |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
304 if n_visible is None: |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
305 n_visible = n_visible_units(rbm) |
996
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
306 rval = HMC_sampler( |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
307 positions = [shared( |
996
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
308 rng.randn( |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
309 n_particles, |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
310 n_visible).astype(floatX), |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
311 name='particles')], |
996
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
312 energy_fn = lambda p : free_energy_given_v(rbm, p[0]), |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
313 seed=int(rng.randint(2**30))) |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
314 return rval |
60f279ec0f7f
mcRBM - made sampler a global function
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
995
diff
changeset
|
315 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
316 class MeanCovRBM(object): |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
317 """Container for mcRBM parameters |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
318 |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
319 It provides parameter lookup by name, as well as a heuristic for initializing the |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
320 parameters for effective learning. |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
321 |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
322 Attributes: |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
323 |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
324 - U - the covariance filters (theano shared variable) |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
325 - W - the mean filters (theano shared variable) |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
326 - a - the visible bias (theano shared variable) |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
327 - b - the covariance bias (theano shared variable) |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
328 - c - the mean bias (theano shared variable) |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
329 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
330 """ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
331 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
332 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c]) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
333 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
334 n_visible = property(lambda s: s.W.value.shape[0]) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
335 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
336 def __init__(self, U, W, a, b, c): |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
337 self.__dict__.update(locals()) |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
338 del self.__dict__['self'] |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
339 |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
340 def __getitem__(self, idx): |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
341 # support unpacking of this container as if it were a tuple |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
342 return self.params[idx] |
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
343 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
344 @classmethod |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
345 def new_from_dims(cls, n_I, n_K, n_J, rng = 8923402190): |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
346 """ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
347 Return a MeanCovRBM instance with randomly-initialized parameters. |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
348 |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
349 :param n_I: input dimensionality |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
350 :param n_K: number of covariance hidden units |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
351 :param n_J: number of mean filters (linear) |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
352 :param rng: seed or numpy RandomState object to initialize params |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
353 """ |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
354 if not hasattr(rng, 'randn'): |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
355 rng = np.random.RandomState(rng) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
356 |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
357 def shrd(X,name): |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
358 return shared(X.astype(floatX), name=name) |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
359 |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
360 # initialization taken from train_mcRBM.py |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
361 return cls( |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
362 U = shrd(0.02 * rng.randn(n_I, n_K),'U'), |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
363 W = shrd(0.05 * rng.randn(n_I, n_J),'W'), |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
364 a = shrd(np.ones(n_I)*(0),'a'), |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
365 b = shrd(np.ones(n_K)*2,'b'), |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
366 c = shrd(np.ones(n_J)*(-2),'c')) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
367 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
368 def __getstate__(self): |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
369 # unpack shared containers, which may have references to Theano stuff and are not a |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
370 # long-term stable data type. |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
371 return dict( |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
372 U = self.U.value, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
373 W = self.W.value, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
374 b = self.b.value, |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
375 c = self.c.value) |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
376 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
377 def __setstate__(self, dct): |
995
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
378 d = dict(dct) |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
379 for key in ['U', 'W', 'a', 'b', 'c']: |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
380 d[key] = shared(d[key], name=key) |
68ca3ea34e72
mcRBM - cleaned up new_from_dims
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
994
diff
changeset
|
381 self.__init__(**d) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
382 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
383 if __name__ == '__main__': |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
384 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
385 dataset='MAR' |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
386 if dataset == 'MAR': |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
387 n_vis=105 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
388 n_patches=10240 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
389 else: |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
390 R,C= 16,16 # the size of image patches |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
391 n_vis=R*C |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
392 n_patches=100000 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
393 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
394 n_train_iters=5000 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
395 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
396 n_burnin_steps=10000 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
397 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
398 l1_penalty=1e-3 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
399 no_l1_epochs = 10 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
400 effective_l1_penalty=0.0 |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
401 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
402 epoch_size=n_patches |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
403 batchsize = 128 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
404 lr = 0.075 / batchsize |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
405 s_lr = TT.scalar() |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
406 s_l1_penalty=TT.scalar() |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
407 n_K=256 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
408 n_J=100 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
409 |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
410 rbm = MeanCovRBM.new_from_dims(n_I=n_vis, n_K=n_K, n_J=n_J) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
411 |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
412 smplr = sampler(rbm, n_particles=batchsize) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
413 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
414 def l2(X): |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
415 return numpy.sqrt((X**2).sum()) |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
416 if dataset == 'MAR': |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
417 tile = pylearn.dataset_ops.image_patches.save_filters_of_ranzato_hinton_2010 |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
418 else: |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
419 def tile(X, fname): |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
420 _img = image_tiling.tile_raster_images(X, |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
421 img_shape=(R,C), |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
422 min_dynamic_range=1e-2) |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
423 image_tiling.save_tiled_raster_images(_img, fname) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
424 |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
425 batch_idx = TT.iscalar() |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
426 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
427 if dataset == 'MAR': |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
428 train_batch = pylearn.dataset_ops.image_patches.ranzato_hinton_2010_op(batch_idx * batchsize + np.arange(batchsize)) |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
429 else: |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
430 train_batch = pylearn.dataset_ops.image_patches.image_patches( |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
431 s_idx = (batch_idx * batchsize + np.arange(batchsize)), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
432 dims = (n_patches,R,C), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
433 center=True, |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
434 unitvar=True, |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
435 dtype=floatX, |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
436 rasterized=True) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
437 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
438 imgs_fn = function([batch_idx], outputs=train_batch) |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
439 |
997
71b0132b694a
mcRBM - removed container logic that was redundant with global methods
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
996
diff
changeset
|
440 grads = contrastive_gradient(rbm, |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
441 pos_v=train_batch, |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
442 neg_v=smplr.positions[0], |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
443 U_l1_penalty=s_l1_penalty, |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
444 W_l1_penalty=s_l1_penalty) |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
445 sgd_ups = sgd_updates( |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
446 rbm.params, |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
447 grads, |
999
c6d08a760960
added sgd_updates to gd/sgd.py. Modif mcRBM to use it.
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
998
diff
changeset
|
448 stepsizes=[2*s_lr, .2*s_lr, .02*s_lr, .1*s_lr, .02*s_lr ]) |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
449 learn_fn = function([batch_idx, s_lr, s_l1_penalty], |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
450 outputs=[ |
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
451 grads[0].norm(2), |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
452 (sgd_ups[0][1] - sgd_ups[0][0]).norm(2), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
453 (sgd_ups[1][1] - sgd_ups[1][0]).norm(2), |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
454 ], |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
455 updates = sgd_ups) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
456 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
457 print "Learning..." |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
458 normVF=1 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
459 last_epoch = -1 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
460 for jj in xrange(n_train_iters): |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
461 epoch = jj*batchsize / epoch_size |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
462 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
463 print_jj = epoch != last_epoch |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
464 last_epoch = epoch |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
465 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
466 if epoch > 10: |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
467 break |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
468 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
469 if print_jj: |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
470 tile(imgs_fn(jj), "imgs_%06i.png"%jj) |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
471 tile(smplr.positions[0].value, "sample_%06i.png"%jj) |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
472 tile(rbm.U.value.T, "U_%06i.png"%jj) |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
473 tile(rbm.W.value.T, "W_%06i.png"%jj) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
474 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
475 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize) |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
476 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
477 print 'l2(U)', l2(rbm.U.value), |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
478 print 'l2(W)', l2(rbm.W.value) |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
479 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
480 print 'U min max', rbm.U.value.min(), rbm.U.value.max(), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
481 print 'W min max', rbm.W.value.min(), rbm.W.value.max(), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
482 print 'a min max', rbm.a.value.min(), rbm.a.value.max(), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
483 print 'b min max', rbm.b.value.min(), rbm.b.value.max(), |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
484 print 'c min max', rbm.c.value.min(), rbm.c.value.max() |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
485 |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
486 print 'parts min', smplr.positions[0].value.min(), |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
487 print 'max',smplr.positions[0].value.max(), |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
488 print 'HMC step', smplr.stepsize, |
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
489 print 'arate', smplr.avg_acceptance_rate |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
490 |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
491 smplr.simulate() |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
492 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
493 l2_of_Ugrad = learn_fn(jj, |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
494 lr/max(1, jj/(20*epoch_size/batchsize)), |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
495 effective_l1_penalty) |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
496 |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
497 if print_jj: |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
498 print 'l2(U_grad)', float(l2_of_Ugrad[0]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
499 print 'l2(U_inc)', float(l2_of_Ugrad[1]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
500 print 'l2(W_inc)', float(l2_of_Ugrad[2]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
501 #print 'FE+', float(l2_of_Ugrad[2]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
502 #print 'FE+[0]', float(l2_of_Ugrad[3]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
503 #print 'FE+[1]', float(l2_of_Ugrad[4]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
504 #print 'FE+[2]', float(l2_of_Ugrad[5]), |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
505 #print 'FE+[3]', float(l2_of_Ugrad[6]) |
979
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
506 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
507 if jj == no_l1_epochs * epoch_size/batchsize: |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
508 print "Activating L1 weight decay" |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
509 effective_l1_penalty = 1e-3 |
2a53384d9742
mcRBM - hacks to driver
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
978
diff
changeset
|
510 |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
511 # weird normalization technique... |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
512 # It constrains all the columns of the matrix to have the same length |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
513 # But the matrix itself is re-scaled to have an arbitrary abslute size. |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
514 U = rbm.U.value |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
515 U_norms = np.sqrt((U*U).sum(axis=0)) |
998
8ba8b08e0442
added the image_patches dataset used in RanzatoHinton2010
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
997
diff
changeset
|
516 assert len(U_norms) == n_K |
992
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
517 normVF = .95 * normVF + .05 * np.mean(U_norms) |
30b7c4defb6c
mcRBM - it works and committing it is taking forever... lets try this approach
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
991
diff
changeset
|
518 rbm.U.value = rbm.U.value * normVF/U_norms |
967
90e11d5d0a41
adding algorithms/mcRBM, but it is not done yet
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff
changeset
|
519 |