pylearn: pylearn/algorithms/mcRBM.py annotate

annotate pylearn/algorithms/mcRBM.py @ 992:30b7c4defb6c

mcRBM - it works and committing it is taking forever... lets try this approach

author	James Bergstra <bergstrj@iro.umontreal.ca>
date	Tue, 24 Aug 2010 14:52:09 -0400
parents	d68828c98c38
children	88107ec01ce8

rev	line source
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	7
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	8 and performs one of the experiments on CIFAR-10 discussed in that paper. There are some minor
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	9 discrepancies between the paper and the accompanying code (train_mcRBM.py), and the
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	10 accompanying code has been taken to be correct in those cases because I couldn't get things to
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	11 work otherwise.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	12
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	14 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	15 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	16
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	17 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	18
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	19 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	20 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	21 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	23 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	24 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	26 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	27
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	28
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	29
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	30 Version in paper
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	31 ----------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	32
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	33 Full Energy of the Mean and Covariance RBM, with
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	34 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	35 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	36 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	37 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	38 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	39
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	40 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	41 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i (U_{if} v_i) / \|U_{.f}\|*\|v\| )^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	42 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	43 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	44 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	45 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	46
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	47 For the energy function to correspond to a probability distribution, P must be non-positive. P
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	48 is initialized to be a diagonal, and in our experience it can be left as such because even in
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	49 the paper it has a very low learning rate, and is only allowed to be updated after the filters
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	50 in U are learned (in effect).
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	51
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	52 Version in published train_mcRBM code
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	53 -------------------------------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	54
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	55 The train_mcRBM file implements learning in a similar but technically different Energy function:
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	56
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	57 E (v, h, g) =
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	58 - 0.5 \sum_f \sum_k P_{fk} h_k (\sum_i U_{if} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	59 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	60 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	61 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	62 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	63
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	64 There are two differences with respect to the paper:
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	65
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	66 - 'v' is not normalized by its length, but rather it is normalized to have length close to
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	67 the square root of the number of its components. The variable called 'small' that
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	68 "avoids division by zero" is orders larger than machine precision, and is on the order of
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	69 the normalized sum-of-squares, so I've included it in the Energy function.
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	70
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	71 - 'U' is also not normalized by its length. U is initialized to have columns that are
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	72 shorter than unit-length (approximately 0.2 with the 105 principle components in the
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	73 train_mcRBM data). During training, the columns of U are constrained manually to have
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	74 equal lengths (see the use of normVF), but Euclidean norm is allowed to change. During
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	75 learning it quickly converges towards 1 and then exceeds 1. It does not seem like this
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	76 column-wise normalization of U is justified by maximum-likelihood, I have no intuition
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	77 for why it is used.
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	78
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	79
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	80 Version in this code
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	81 --------------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	82
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	83 This file implements the same algorithm as the train_mcRBM code, except that the P matrix is
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	84 omitted for clarity, and replaced analytically with a negative identity matrix.
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	85
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	86 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	87 + 0.5 \sum_k h_k (\sum_i U_{ik} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	88 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	89 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	90 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	91 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	92
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	93
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	94
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	95 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	96 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	97
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	98 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	99 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	101
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	102 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	103 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	104 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	105 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	106 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	107 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	108 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	109
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	110 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	111 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	112 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	113 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	114
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	115 Matrices are generally layed out and accessed according to a C-order convention.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	116
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	117 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	118
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	119 #
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	120 # WORKING NOTES
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	121 # THIS DERIVATION IS BASED ON THE PAPER ENERGY FUNCTION
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	122 # NOT THE ENERGY FUNCTION IN THE CODE!!!
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	123 #
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	124 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	125 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	126 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	127 #
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	128 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	129 # E (v, h, g) =
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	130 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / \|U_{*f}\|^2 \|v\|^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	131 # - \sum_k b_k h_k
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	132 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	133 # - \sum_j \sum_i W_{ij} g_j v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	134 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	135 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	136 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	137 #
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	138 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	139 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	140 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	141 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	142 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	143 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	144 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	145 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	146 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	147 # - \sum_j \sum_i W_{ij} g_j v_i
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	148 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	149 # - \sum_i a_i v_i ))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	150 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	151 # Get rid of double negs in exp
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	152 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	153 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	154 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	155 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	156 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	157 # + \sum_j \sum_i W_{ij} g_j v_i
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	158 # + \sum_j c_j g_j))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	159 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	160 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	161 # Break up log
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	162 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	163 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\|\|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	164 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	165 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	166 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	167 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	168 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	169 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	170 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	171 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	172 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(..
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	173 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	174 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	175 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	176 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	177 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	178 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	179 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	180 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	181 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (\|U_{f}\|\|v\|)))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	182 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	183 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	184 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	185 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	186 # For negative-one-diagonal P this gives:
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	187 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	188 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (\|U_{k}\|\|v\|)))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	189 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	190 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	191 # - \sum_i a_i v_i
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	192
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	193 import sys
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	194 import logging
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	195 import numpy as np
973 aa201f357d7b mcRBM - added numpy import James Bergstra <bergstrj@iro.umontreal.ca> parents: 972 diff changeset	196 import numpy
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	197
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	198 import theano
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	199 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	200 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	201 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	202
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	203 import pylearn
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	204 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	205 from pylearn.io import image_tiling
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	206
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	207 #TODO: This should be in the datasets folder
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	208 import pylearn.datasets.config
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	209 from pylearn.dataset_ops.protocol import TensorFnDataset
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	210 from pylearn.dataset_ops.memo import memo
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	211 import pylearn
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	212 import scipy.io
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	213 import os
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	214
973 aa201f357d7b mcRBM - added numpy import James Bergstra <bergstrj@iro.umontreal.ca> parents: 972 diff changeset	215
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	216 #TODO: This should be in the nnet part of the library
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	217 def sgd_updates(params, grads, lr):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	218 try:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	219 float(lr)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	220 lr = [lr for p in params]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	221 except TypeError:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	222 pass
974 f2cdcc71ece1 mcRBM - added L1 penalties and normal sign convention to contrastive grad James Bergstra <bergstrj@iro.umontreal.ca> parents: 973 diff changeset	223 updates = [(p, p - plr * gp) for (plr, p, gp) in zip(lr, params, grads)]
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	224 return updates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	225
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	226 @memo
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	227 def load_mcRBM_demo_patches():
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	228 d = scipy.io.loadmat(os.path.join(pylearn.datasets.config.data_root(),'image_patches', 'mcRBM', 'training_colorpatches_16x16_demo.mat'))
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	229 totnumcases = d["whitendata"].shape[0]
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	230 #d = d["whitendata"][0:np.floor(totnumcases/batch_size)*batch_size,:].copy()
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	231 d = d["whitendata"].copy()
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	232 return d
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	233
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	234
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	235
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	236
987 043aa1b748a7 mcRBM - comment to as_shared James Bergstra <bergstrj@iro.umontreal.ca> parents: 986 diff changeset	237 # this is a little hack, probably should be removed
043aa1b748a7 mcRBM - comment to as_shared James Bergstra <bergstrj@iro.umontreal.ca> parents: 986 diff changeset	238 # The logic about casting things to shared vars is busted anyway (wrt pickling)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	239 def as_shared(x, name=None, dtype=floatX):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	240 if hasattr(x, 'type'):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	241 return x
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	242 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	243 if 'float' in str(x.dtype):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	244 return shared(x.astype(floatX), name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	245 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	246 return shared(x, name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	247
989 9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	248 def hidden_cov_units_preactivation_given_v(rbm, v, small=0.5):
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	249 """Return argument to the sigmoid that would give mean of covariance hid units
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	250
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	251 See the math at the top of this file for what 'adjusted' means.
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	252
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	253 return b - 0.5 * dot(adjusted(v), U)**2
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	254 """
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	255 (U,W,a,b,c) = rbm
989 9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	256 unit_v = v / (TT.sqrt(TT.mean(v**2, axis=1)+small)).dimshuffle(0,'x') # adjust row norm
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	257 return b - 0.5 * dot(unit_v, U)**2
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	258
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	259 def free_energy_terms_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	260 """Returns theano expression for the terms that are added to form the free energy of
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	261 visible vector `v` in an mcRBM.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	262
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	263 1. Free energy related to covariance hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	264 2. Free energy related to mean hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	265 3. Free energy related to L2-Norm of `v`
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	266 4. Free energy related to projection of `v` onto biases `a`
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	267 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	268 U, W, a, b, c = rbm
975 38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	269 t0 = -TT.sum(TT.nnet.softplus(hidden_cov_units_preactivation_given_v(rbm, v)),axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	270 t1 = -TT.sum(TT.nnet.softplus(c + dot(v,W)), axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	271 t2 = 0.5 * TT.sum(v**2, axis=1)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	272 t3 = -TT.dot(v, a)
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	273 return [t0, t1, t2, t3]
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	274
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	275 def free_energy_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	276 """Returns theano expression for free energy of visible vector `v` in an mcRBM
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	277 """
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	278 return sum(free_energy_terms_given_v(rbm,v))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	279
990 e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	280 def contrastive_gradient(rbm, pos_v, neg_v, U_l1_penalty=0, W_l1_penalty=0):
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	281 """Return a list of gradient expressions for the rbm parameters
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	282
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	283 :param pos_v: positive-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	284 :param neg_v: negative-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	285 :param U_l1_penalty: a scalar-valued multiplier on the L1 penalty on U
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	286 :param W_l1_penalty: a scalar-valued multiplier on the L1 penalty on W
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	287 """
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	288 U, W, a, b, c = rbm
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	289 pos_FE = free_energy_given_v(rbm, pos_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	290 neg_FE = free_energy_given_v(rbm, neg_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	291 c0 = (pos_FE - neg_FE).sum()
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	292 c1 = abs(U).sum()*U_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	293 c2 = abs(W).sum()*W_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	294 cost = c0 + c1 + c2
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	295 rval = theano.tensor.grad(cost, list(rbm))
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	296 return rval
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	297
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	298 def expected_h_g_given_v(rbm, v):
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	299 """Returns tuple (`h`, `g`) of theano expression conditional expectations in an mcRBM.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	300
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	301 `h` is the conditional on the covariance units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	302 `g` is the conditional on the mean units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	303
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	304 """
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	305 (U, W, a, b, c) = rbm
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	306 h = TT.nnet.sigmoid(hidden_cov_units_preactivation_given_v(rbm, v))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	307 g = nnet.sigmoid(c + dot(v,W))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	308 return (h, g)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	309
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	310 class MeanCovRBM(object):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	311 """Container for mcRBM parameters that gives more convenient access to mcRBM methods.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	312 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	313
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	314 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	315
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	316 n_visible = property(lambda s: s.W.value.shape[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	317
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	318 def __init__(self, U, W, a, b, c):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	319 self.U = as_shared(U, 'U')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	320 self.W = as_shared(W, 'W')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	321 self.a = as_shared(a, 'a')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	322 self.b = as_shared(b, 'b')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	323 self.c = as_shared(c, 'c')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	324
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	325 assert self.b.type.dtype == 'float32'
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	326
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	327 @classmethod
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	328 def new_from_dims(cls,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	329 n_I, # input dimensionality
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	330 n_K, # number of covariance hidden units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	331 n_F, # number of covariance filters (squared)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	332 n_J, # number of mean filters (linear)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	333 seed = 8923402190,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	334 ):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	335 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	336 Return a MeanCovRBM instance with randomly-initialized parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	337 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	338 rng = np.random.RandomState(seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	339
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	340 # initialization taken from Marc'Aurelio
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	341
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	342 return cls(
986 bee0ca674b2b mcRBM - removed numpy_project_onto_ball James Bergstra <bergstrj@iro.umontreal.ca> parents: 985 diff changeset	343 U = 0.02 * rng.randn(n_I, n_F),
bee0ca674b2b mcRBM - removed numpy_project_onto_ball James Bergstra <bergstrj@iro.umontreal.ca> parents: 985 diff changeset	344 W = 0.05 * rng.randn(n_I, n_J),
bee0ca674b2b mcRBM - removed numpy_project_onto_ball James Bergstra <bergstrj@iro.umontreal.ca> parents: 985 diff changeset	345 #W = rng.randn(n_I, n_J)/np.sqrt((n_I+n_J)/2),
bee0ca674b2b mcRBM - removed numpy_project_onto_ball James Bergstra <bergstrj@iro.umontreal.ca> parents: 985 diff changeset	346 a = np.ones(n_I)*(0),
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	347 b = np.ones(n_K)*2,
986 bee0ca674b2b mcRBM - removed numpy_project_onto_ball James Bergstra <bergstrj@iro.umontreal.ca> parents: 985 diff changeset	348 c = np.ones(n_J)*(-2),)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	349
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	350 def __getstate__(self):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	351 # unpack shared containers, which may have references to Theano stuff
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	352 # and are not a long-term stable data type.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	353 return dict(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	354 U = self.U.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	355 W = self.W.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	356 b = self.b.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	357 c = self.c.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	358
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	359 def __setstate__(self, dct):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	360 self.__init__(**dct) # calls as_shared on pickled arrays
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	361
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	362 def hmc_sampler(self, n_particles=100, seed=7823748):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	363 return HMC_sampler(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	364 positions = [as_shared(
978 ab4bc97ca060 mcRBM - particles initialized w randn instead of rand() James Bergstra <bergstrj@iro.umontreal.ca> parents: 977 diff changeset	365 np.random.RandomState(seed^20893).randn(
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	366 n_particles,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	367 self.n_visible ))],
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	368 energy_fn = lambda p : free_energy_given_v(self.params, p[0]),
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	369 seed=seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	370
976 4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	371 def free_energy_given_v(self, v, extra=False):
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	372 assert 0
976 4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	373 rval = free_energy_given_v(self.params, v)
4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	374 if extra:
4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	375 return rval
4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	376 else:
4cbd65cf902d mcRBM - added extra free_energy param James Bergstra <bergstrj@iro.umontreal.ca> parents: 975 diff changeset	377 return rval[0]
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	378
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	379 def contrastive_gradient(self, args, *kwargs):
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	380 """Return a list of gradient expressions for self.params
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	381
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	382 :param pos_v: positive-phase sample of visible units
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	383 :param neg_v: negative-phase sample of visible units
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	384 """
990 e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	385 return contrastive_gradient(self.params, args, *kwargs)
974 f2cdcc71ece1 mcRBM - added L1 penalties and normal sign convention to contrastive grad James Bergstra <bergstrj@iro.umontreal.ca> parents: 973 diff changeset	386
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	387
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	388 if __name__ == '__main__':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	389
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	390 print >> sys.stderr, "TODO: use P matrix (aka FH matrix)"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	391
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	392 dataset='MAR'
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	393 if dataset == 'MAR':
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	394 R,C= 21,5
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	395 n_patches=10240
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	396 demodata = scipy.io.loadmat(os.path.join(pylearn.datasets.config.data_root(),'image_patches', 'mcRBM', 'training_colorpatches_16x16_demo.mat'))
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	397 else:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	398 R,C= 16,16 # the size of image patches
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	399 n_patches=100000
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	400
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	401 n_train_iters=5000
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	402
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	403 n_burnin_steps=10000
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	404
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	405 l1_penalty=1e-3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	406 no_l1_epochs = 10
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	407 effective_l1_penalty=0.0
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	408
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	409 epoch_size=n_patches
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	410 batchsize = 128
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	411 lr = 0.075 / batchsize
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	412 s_lr = TT.scalar()
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	413 s_l1_penalty=TT.scalar()
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	414 n_K=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	415 n_F=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	416 n_J=100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	417
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	418 rbm = MeanCovRBM.new_from_dims(n_I=R*C,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	419 n_K=n_K,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	420 n_J=n_J,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	421 n_F=n_F,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	422 )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	423
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	424 sampler = rbm.hmc_sampler(n_particles=batchsize)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	425
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	426 def l2(X):
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	427 return numpy.sqrt((X**2).sum())
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	428 def tile(X, fname):
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	429 if dataset == 'MAR':
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	430 X = np.dot(X, demodata['invpcatransf'].T)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	431 R=16
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	432 C=16
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	433 #X = X.reshape((X.shape[0], 3, 16, 16)).transpose([0,2,3,1]).copy()
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	434 X = (X[:,:256], X[:,256:512], X[:,512:], None)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	435 _img = image_tiling.tile_raster_images(X,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	436 img_shape=(R,C),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	437 min_dynamic_range=1e-2)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	438 image_tiling.save_tiled_raster_images(_img, fname)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	439 #print "Burning in..."
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	440 #for burnin in xrange(n_burnin_steps):
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	441 #sampler.simulate()
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	442
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	443 if 0:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	444 print "Just SAMPLING..."
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	445 for jj in xrange(n_burnin_steps):
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	446 if 0 == jj % 100:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	447 tile(sampler.positions[0].value, "sampler_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	448 tile(numpy.random.randn(100, 105), "random_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	449 print "burning in... ", jj
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	450 sys.stdout.flush()
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	451 sampler.simulate()
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	452
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	453 sys.exit()
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	454
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	455 batch_idx = TT.iscalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	456
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	457 if dataset == 'MAR':
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	458 op = TensorFnDataset(floatX,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	459 bcast=(False,),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	460 fn=load_mcRBM_demo_patches,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	461 single_shape=(105,))
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	462 train_batch = op((batch_idx * batchsize + np.arange(batchsize))%n_patches)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	463 else:
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	464 from pylearn.dataset_ops import image_patches
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	465 train_batch = image_patches.image_patches(
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	466 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	467 dims = (n_patches,R,C),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	468 center=True,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	469 unitvar=True,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	470 dtype=floatX,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	471 rasterized=True)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	472
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	473 imgs_fn = function([batch_idx], outputs=train_batch)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	474
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	475 grads = rbm.contrastive_gradient(
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	476 pos_v=train_batch,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	477 neg_v=sampler.positions[0],
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	478 U_l1_penalty=s_l1_penalty,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	479 W_l1_penalty=s_l1_penalty)
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	480 sgd_ups = sgd_updates(
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	481 rbm.params,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	482 grads,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	483 lr=[2s_lr, .2s_lr, .02s_lr, .1s_lr, .02*s_lr ])
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	484
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	485 learn_fn = function([batch_idx, s_lr, s_l1_penalty],
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	486 outputs=[
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	487 grads[0].norm(2),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	488 (sgd_ups[0][1] - sgd_ups[0][0]).norm(2),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	489 (sgd_ups[1][1] - sgd_ups[1][0]).norm(2),
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	490 ],
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	491 updates = sgd_ups)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	492 #rbm.free_energy_given_v(train_batch).sum(),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	493 #rbm.free_energy_given_v(train_batch,extra=1)[1][0].sum(),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	494 #rbm.free_energy_given_v(train_batch,extra=1)[1][1].sum(),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	495 #rbm.free_energy_given_v(train_batch,extra=1)[1][2].sum(),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	496 #rbm.free_energy_given_v(train_batch,extra=1)[1][3].sum(),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	497 theano.printing.pydotprint(function([batch_idx, s_l1_penalty], grads[0]), 'grads0.png')
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	498
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	499 print "Learning..."
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	500 normVF=1
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	501 last_epoch = -1
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	502 for jj in xrange(n_train_iters):
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	503 epoch = jj*batchsize / epoch_size
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	504
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	505 print_jj = epoch != last_epoch
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	506 last_epoch = epoch
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	507
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	508 if epoch > 10:
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	509 break
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	510
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	511 if print_jj:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	512 tile(imgs_fn(jj), "imgs_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	513 tile(sampler.positions[0].value, "sample_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	514 tile(rbm.U.value.T, "U_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	515 tile(rbm.W.value.T, "W_%06i.png"%jj)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	516
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	517 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	518
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	519 print 'l2(U)', l2(rbm.U.value),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	520 print 'l2(W)', l2(rbm.W.value)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	521
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	522 print 'U min max', rbm.U.value.min(), rbm.U.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	523 print 'W min max', rbm.W.value.min(), rbm.W.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	524 print 'a min max', rbm.a.value.min(), rbm.a.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	525 print 'b min max', rbm.b.value.min(), rbm.b.value.max(),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	526 print 'c min max', rbm.c.value.min(), rbm.c.value.max()
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	527
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	528 print 'parts min', sampler.positions[0].value.min(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	529 print 'max',sampler.positions[0].value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	530 print 'HMC step', sampler.stepsize,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	531 print 'arate', sampler.avg_acceptance_rate
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	532
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	533 sampler.simulate()
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	534
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	535 l2_of_Ugrad = learn_fn(jj,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	536 lr/max(1, jj/(20*epoch_size/batchsize)),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	537 effective_l1_penalty)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	538
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	539 if print_jj:
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	540 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	541 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	542 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	543 #print 'FE+', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	544 #print 'FE+[0]', float(l2_of_Ugrad[3]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	545 #print 'FE+[1]', float(l2_of_Ugrad[4]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	546 #print 'FE+[2]', float(l2_of_Ugrad[5]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	547 #print 'FE+[3]', float(l2_of_Ugrad[6])
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	548
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	549 if jj == no_l1_epochs * epoch_size/batchsize:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	550 print "Activating L1 weight decay"
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	551 effective_l1_penalty = 1e-3
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	552
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	553 # weird normalization technique...
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	554 # It constrains all the columns of the matrix to have the same length
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	555 # But the matrix itself is re-scaled to have an arbitrary abslute size.
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	556 U = rbm.U.value
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	557 U_norms = np.sqrt((U*U).sum(axis=0))
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	558 assert len(U_norms) == n_F
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	559 normVF = .95 * normVF + .05 * np.mean(U_norms)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	560 rbm.U.value = rbm.U.value * normVF/U_norms
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	561
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	562
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	563 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	564 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	565 # Marc'Aurelio Ranzato's code
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	566 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	567 ######################################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	568 # compute the value of the free energy at a given input
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	569 # F = - sum log(1+exp(- .5 FH (VF data/norm(data))^2 + bias_cov)) +...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	570 # - sum log(1+exp(w_mean data + bias_mean)) + ...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	571 # - bias_vis data + 0.5 data^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	572 # NOTE: FH is constrained to be positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	573 # (in the paper the sign is negative but the sign in front of it is also flipped)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	574 def compute_energy_mcRBM(data,normdata,vel,energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,t1,t2,t6,feat,featsq,feat_mean,length,lengthsq,normcoeff,small,num_vis):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	575 # normalize input data vectors
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	576 data.mult(data, target = t6) # DxP (nr input dims x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	577 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	578 lengthsq.mult(0.5, target = energy) # energy of quadratic regularization term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	579 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	580
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	581 lengthsq.add(small) # small prevents division by 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	582 # energy_j = \sum_i 0.5 data_ij ^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	583 # lengthsq_j = 1/ (\sum_i data_ij ^2 + small)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	584 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	585 # length_j = sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	586 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	587 # normcoef_j = 1/sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	588 data.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	589 # normdata is like data, but cols have unit L2 norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	590
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	591 ## potential
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	592 # covariance contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	593 cmt.dot(VF.T, normdata, target = feat) # HxP (nr factors x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	594 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	595
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	596 # featsq is the squared cosines (VF with data)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	597 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	598 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	599 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	600 cmt.exp(t1) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	601 t1.add(1, target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	602 cmt.log(t2)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	603 t2.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	604 energy.add_sums(t2, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	605 # mean contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	606 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	607 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	608 cmt.exp(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	609 feat_mean.add(1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	610 cmt.log(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	611 feat_mean.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	612 energy.add_sums(feat_mean, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	613 # visible bias term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	614 data.mult_by_col(bias_vis, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	615 t6.mult(-1) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	616 energy.add_sums(t6, axis=0) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	617 # kinetic
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	618 vel.mult(vel, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	619 energy.add_sums(t6, axis = 0, mult = .5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	620
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	621 ######################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	622 # mcRBM trainer: sweeps over the training set.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	623 # For each batch of samples compute derivatives to update the parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	624 # at the training samples and at the negative samples drawn calling HMC sampler.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	625 def train_mcRBM():
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	626
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	627 config = ConfigParser()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	628 config.read('input_configuration')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	629
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	630 verbose = config.getint('VERBOSITY','verbose')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	631
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	632 num_epochs = config.getint('MAIN_PARAMETER_SETTING','num_epochs')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	633 batch_size = config.getint('MAIN_PARAMETER_SETTING','batch_size')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	634 startFH = config.getint('MAIN_PARAMETER_SETTING','startFH')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	635 startwd = config.getint('MAIN_PARAMETER_SETTING','startwd')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	636 doPCD = config.getint('MAIN_PARAMETER_SETTING','doPCD')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	637
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	638 # model parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	639 num_fac = config.getint('MODEL_PARAMETER_SETTING','num_fac')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	640 num_hid_cov = config.getint('MODEL_PARAMETER_SETTING','num_hid_cov')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	641 num_hid_mean = config.getint('MODEL_PARAMETER_SETTING','num_hid_mean')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	642 apply_mask = config.getint('MODEL_PARAMETER_SETTING','apply_mask')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	643
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	644 # load data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	645 data_file_name = config.get('DATA','data_file_name')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	646 d = loadmat(data_file_name) # input in the format PxD (P vectorized samples with D dimensions)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	647 totnumcases = d["whitendata"].shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	648 d = d["whitendata"][0:floor(totnumcases/batch_size)*batch_size,:].copy()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	649 totnumcases = d.shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	650 num_vis = d.shape[1]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	651 num_batches = int(totnumcases/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	652 dev_dat = cmt.CUDAMatrix(d.T) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	653
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	654 # training parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	655 epsilon = config.getfloat('OPTIMIZER_PARAMETERS','epsilon')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	656 epsilonVF = 2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	657 epsilonFH = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	658 epsilonb = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	659 epsilonw_mean = 0.2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	660 epsilonb_mean = 0.1*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	661 weightcost_final = config.getfloat('OPTIMIZER_PARAMETERS','weightcost_final')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	662
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	663 # HMC setting
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	664 hmc_step_nr = config.getint('HMC_PARAMETERS','hmc_step_nr')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	665 hmc_step = 0.01
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	666 hmc_target_ave_rej = config.getfloat('HMC_PARAMETERS','hmc_target_ave_rej')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	667 hmc_ave_rej = hmc_target_ave_rej
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	668
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	669 # initialize weights
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	670 VF = cmt.CUDAMatrix(np.array(0.02 * np.random.randn(num_vis, num_fac), dtype=np.float32, order='F')) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	671 if apply_mask == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	672 FH = cmt.CUDAMatrix( np.array( np.eye(num_fac,num_hid_cov), dtype=np.float32, order='F') ) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	673 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	674 dd = loadmat('your_FHinit_mask_file.mat') # see CVPR2010paper_material/topo2D_3x3_stride2_576filt.mat for an example
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	675 FH = cmt.CUDAMatrix( np.array( dd["FH"], dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	676 bias_cov = cmt.CUDAMatrix( np.array(2.0*np.ones((num_hid_cov, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	677 bias_vis = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	678 w_mean = cmt.CUDAMatrix( np.array( 0.05 * np.random.randn(num_vis, num_hid_mean), dtype=np.float32, order='F') ) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	679 bias_mean = cmt.CUDAMatrix( np.array( -2.0*np.ones((num_hid_mean,1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	680
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	681 # initialize variables to store derivatives
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	682 VFinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	683 FHinc = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	684 bias_covinc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_cov, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	685 bias_visinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	686 w_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_hid_mean)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	687 bias_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_mean, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	688
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	689 # initialize temporary storage
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	690 data = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	691 normdata = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	692 negdataini = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	693 feat = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	694 featsq = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	695 negdata = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	696 old_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	697 new_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	698 gradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	699 normgradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	700 thresh = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	701 feat_mean = cmt.CUDAMatrix( np.array(np.empty((num_hid_mean, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	702 vel = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	703 length = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	704 lengthsq = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	705 normcoeff = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	706 if apply_mask==1: # this used to constrain very large FH matrices only allowing to change values in a neighborhood
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	707 dd = loadmat('your_FHinit_mask_file.mat')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	708 mask = cmt.CUDAMatrix( np.array(dd["mask"], dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	709 normVF = 1
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	710 small = 0.5
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	711
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	712 # other temporary vars
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	713 t1 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	714 t2 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	715 t3 = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	716 t4 = cmt.CUDAMatrix( np.array(np.empty((1,batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	717 t5 = cmt.CUDAMatrix( np.array(np.empty((1,1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	718 t6 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	719 t7 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	720 t8 = cmt.CUDAMatrix( np.array(np.empty((num_vis, num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	721 t9 = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	722 t10 = cmt.CUDAMatrix( np.array(np.empty((1,num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	723 t11 = cmt.CUDAMatrix( np.array(np.empty((1,num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	724
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	725 # start training
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	726 for epoch in range(num_epochs):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	727
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	728 print "Epoch " + str(epoch + 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	729
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	730 # anneal learning rates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	731 epsilonVFc = epsilonVF/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	732 epsilonFHc = epsilonFH/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	733 epsilonbc = epsilonb/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	734 epsilonw_meanc = epsilonw_mean/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	735 epsilonb_meanc = epsilonb_mean/max(1,epoch/20)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	736 weightcost = weightcost_final
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	737
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	738 if epoch <= startFH:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	739 epsilonFHc = 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	740 if epoch <= startwd:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	741 weightcost = 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	742
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	743 for batch in range(num_batches):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	744
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	745 # get current minibatch
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	746 data = dev_dat.slice(batchbatch_size,(batch + 1)batch_size) # DxP (nr dims x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	747
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	748 # normalize input data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	749 data.mult(data, target = t6) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	750 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	751 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	752 lengthsq.add(small) # small avoids division by 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	753 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	754 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	755 data.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	756 ## compute positive sample derivatives
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	757 # covariance part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	758 cmt.dot(VF.T, normdata, target = feat) # HxP (nr facs x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	759 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	760 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	761 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	762 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	763 t1.apply_sigmoid(target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	764 cmt.dot(featsq, t2.T, target = FHinc) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	765 cmt.dot(FH,t2, target = t3) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	766 t3.mult(feat)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	767 cmt.dot(normdata, t3.T, target = VFinc) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	768 t2.sum(axis = 1, target = bias_covinc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	769 bias_covinc.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	770 # visible bias
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	771 data.sum(axis = 1, target = bias_visinc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	772 bias_visinc.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	773 # mean part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	774 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	775 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	776 feat_mean.apply_sigmoid() # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	777 feat_mean.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	778 cmt.dot(data, feat_mean.T, target = w_meaninc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	779 feat_mean.sum(axis = 1, target = bias_meaninc)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	780
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	781 # HMC sampling: draw an approximate sample from the model
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	782 if doPCD == 0: # CD-1 (set negative data to current training samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	783 hmc_step, hmc_ave_rej = draw_HMC_samples(data,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	784 else: # PCD-1 (use previous negative data as starting point for chain)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	785 negdataini.assign(negdata)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	786 hmc_step, hmc_ave_rej = draw_HMC_samples(negdataini,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	787
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	788 # compute derivatives at the negative samples
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	789 # normalize input data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	790 negdata.mult(negdata, target = t6) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	791 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	792 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	793 lengthsq.add(small)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	794 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	795 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	796 negdata.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	797 # covariance part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	798 cmt.dot(VF.T, normdata, target = feat) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	799 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	800 cmt.dot(FH.T,featsq, target = t1) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	801 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	802 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	803 t1.apply_sigmoid(target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	804 FHinc.subtract_dot(featsq, t2.T) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	805 FHinc.mult(0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	806 cmt.dot(FH,t2, target = t3) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	807 t3.mult(feat)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	808 VFinc.subtract_dot(normdata, t3.T) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	809 bias_covinc.add_sums(t2, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	810 # visible bias
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	811 bias_visinc.add_sums(negdata, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	812 # mean part
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	813 cmt.dot(w_mean.T, negdata, target = feat_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	814 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	815 feat_mean.apply_sigmoid() # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	816 w_meaninc.add_dot(negdata, feat_mean.T)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	817 bias_meaninc.add_sums(feat_mean, axis = 1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	818
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	819 # update parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	820 VFinc.add_mult(VF.sign(), weightcost) # L1 regularization
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	821 VF.add_mult(VFinc, -epsilonVFc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	822 # normalize columns of VF: normalize by running average of their norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	823 VF.mult(VF, target = t8)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	824 t8.sum(axis = 0, target = t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	825 cmt.sqrt(t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	826 t10.sum(axis=1,target = t5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	827 t5.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	828 normVF = .95normVF + (.05/num_fac) t5.numpy_array[0,0] # estimate norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	829 t10.reciprocal()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	830 VF.mult_by_row(t10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	831 VF.mult(normVF)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	832 bias_cov.add_mult(bias_covinc, -epsilonbc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	833 bias_vis.add_mult(bias_visinc, -epsilonbc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	834
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	835 if epoch > startFH:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	836 FHinc.add_mult(FH.sign(), weightcost) # L1 regularization
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	837 FH.add_mult(FHinc, -epsilonFHc/batch_size) # update
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	838 # set to 0 negative entries in FH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	839 FH.greater_than(0, target = t9)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	840 FH.mult(t9)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	841 if apply_mask==1:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	842 FH.mult(mask)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	843 # normalize columns of FH: L1 norm set to 1 in each column
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	844 FH.sum(axis = 0, target = t11)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	845 t11.reciprocal()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	846 FH.mult_by_row(t11)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	847 w_meaninc.add_mult(w_mean.sign(),weightcost)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	848 w_mean.add_mult(w_meaninc, -epsilonw_meanc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	849 bias_mean.add_mult(bias_meaninc, -epsilonb_meanc/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	850
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	851 if verbose == 1:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	852 print "VF: " + '%3.2e' % VF.euclid_norm() + ", DVF: " + '%3.2e' % (VFinc.euclid_norm()(epsilonVFc/batch_size)) + ", FH: " + '%3.2e' % FH.euclid_norm() + ", DFH: " + '%3.2e' % (FHinc.euclid_norm()(epsilonFHc/batch_size)) + ", bias_cov: " + '%3.2e' % bias_cov.euclid_norm() + ", Dbias_cov: " + '%3.2e' % (bias_covinc.euclid_norm()(epsilonbc/batch_size)) + ", bias_vis: " + '%3.2e' % bias_vis.euclid_norm() + ", Dbias_vis: " + '%3.2e' % (bias_visinc.euclid_norm()(epsilonbc/batch_size)) + ", wm: " + '%3.2e' % w_mean.euclid_norm() + ", Dwm: " + '%3.2e' % (w_meaninc.euclid_norm()(epsilonw_meanc/batch_size)) + ", bm: " + '%3.2e' % bias_mean.euclid_norm() + ", Dbm: " + '%3.2e' % (bias_meaninc.euclid_norm()(epsilonb_meanc/batch_size)) + ", step: " + '%3.2e' % hmc_step + ", rej: " + '%3.2e' % hmc_ave_rej
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	853 sys.stdout.flush()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	854 # back-up every once in a while
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	855 if np.mod(epoch,10) == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	856 VF.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	857 FH.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	858 bias_cov.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	859 w_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	860 bias_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	861 bias_vis.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	862 savemat("ws_temp", {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array,'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	863 # final back-up
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	864 VF.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	865 FH.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	866 bias_cov.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	867 bias_vis.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	868 w_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	869 bias_mean.copy_to_host()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	870 savemat("ws_fac" + str(num_fac) + "_cov" + str(num_hid_cov) + "_mean" + str(num_hid_mean), {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array, 'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})

Mercurial > pylearn

annotate pylearn/algorithms/mcRBM.py @ 992:30b7c4defb6c