pylearn: pylearn/algorithms/mcRBM.py annotate

annotate pylearn/algorithms/mcRBM.py @ 998:8ba8b08e0442

added the image_patches dataset used in RanzatoHinton2010 modified mcRBM to use it.

author	James Bergstra <bergstrj@iro.umontreal.ca>
date	Tue, 24 Aug 2010 16:51:53 -0400
parents	71b0132b694a
children	c6d08a760960

rev	line source
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	7
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	8 and performs one of the experiments on CIFAR-10 discussed in that paper. There are some minor
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	9 discrepancies between the paper and the accompanying code (train_mcRBM.py), and the
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	10 accompanying code has been taken to be correct in those cases because I couldn't get things to
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	11 work otherwise.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	12
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	14 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	15 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	16
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	17 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	18
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	19 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	20 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	21 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	23 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	24 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	26 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	27
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	28
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	29
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	30 Version in paper
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	31 ----------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	32
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	33 Full Energy of the Mean and Covariance RBM, with
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	34 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	35 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	36 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	37 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	38 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	39
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	40 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	41 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i (U_{if} v_i) / \|U_{.f}\|*\|v\| )^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	42 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	43 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	44 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	45 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	46
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	47 For the energy function to correspond to a probability distribution, P must be non-positive. P
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	48 is initialized to be a diagonal, and in our experience it can be left as such because even in
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	49 the paper it has a very low learning rate, and is only allowed to be updated after the filters
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	50 in U are learned (in effect).
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	51
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	52 Version in published train_mcRBM code
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	53 -------------------------------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	54
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	55 The train_mcRBM file implements learning in a similar but technically different Energy function:
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	56
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	57 E (v, h, g) =
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	58 - 0.5 \sum_f \sum_k P_{fk} h_k (\sum_i U_{if} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	59 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	60 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	61 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	62 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	63
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	64 There are two differences with respect to the paper:
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	65
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	66 - 'v' is not normalized by its length, but rather it is normalized to have length close to
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	67 the square root of the number of its components. The variable called 'small' that
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	68 "avoids division by zero" is orders larger than machine precision, and is on the order of
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	69 the normalized sum-of-squares, so I've included it in the Energy function.
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	70
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	71 - 'U' is also not normalized by its length. U is initialized to have columns that are
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	72 shorter than unit-length (approximately 0.2 with the 105 principle components in the
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	73 train_mcRBM data). During training, the columns of U are constrained manually to have
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	74 equal lengths (see the use of normVF), but Euclidean norm is allowed to change. During
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	75 learning it quickly converges towards 1 and then exceeds 1. It does not seem like this
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	76 column-wise normalization of U is justified by maximum-likelihood, I have no intuition
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	77 for why it is used.
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	78
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	79
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	80 Version in this code
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	81 --------------------
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	82
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	83 This file implements the same algorithm as the train_mcRBM code, except that the P matrix is
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	84 omitted for clarity, and replaced analytically with a negative identity matrix.
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	85
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	86 E (v, h, g) =
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	87 + 0.5 \sum_k h_k (\sum_i U_{ik} v_i / sqrt(\sum_i v_i^2/I + 0.5))^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	88 - \sum_k b_k h_k
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	89 + 0.5 \sum_i v_i^2
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	90 - \sum_j \sum_i W_{ij} g_j v_i
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	91 - \sum_j c_j g_j
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	92
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	93
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	94
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	95 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	96 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	97
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	98 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	99 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	101
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	102 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	103 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	104 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	105 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	106 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	107 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	108 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	109
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	110 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	111 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	112 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	113 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	114
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	115 Matrices are generally layed out and accessed according to a C-order convention.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	116
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	117 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	118
984 5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	119 #
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	120 # WORKING NOTES
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	121 # THIS DERIVATION IS BASED ON THE PAPER ENERGY FUNCTION
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	122 # NOT THE ENERGY FUNCTION IN THE CODE!!!
5badf36a6daf mcRBM - added notes to leading comment James Bergstra <bergstrj@iro.umontreal.ca> parents: 979 diff changeset	123 #
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	124 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	125 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	126 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	127 #
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	128 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	129 # E (v, h, g) =
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	130 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / \|U_{*f}\|^2 \|v\|^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	131 # - \sum_k b_k h_k
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	132 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	133 # - \sum_j \sum_i W_{ij} g_j v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	134 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	135 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	136 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	137 #
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	138 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	139 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	140 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	141 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	142 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	143 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	144 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	145 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	146 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	147 # - \sum_j \sum_i W_{ij} g_j v_i
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	148 # - \sum_j c_j g_j
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	149 # - \sum_i a_i v_i ))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	150 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	151 # Get rid of double negs in exp
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	152 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	153 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	154 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	155 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	156 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	157 # + \sum_j \sum_i W_{ij} g_j v_i
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	158 # + \sum_j c_j g_j))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	159 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	160 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	161 # Break up log
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	162 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	163 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\|\|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	164 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	165 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	166 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	167 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	168 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	169 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	170 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	171 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	172 # Use domain h is binary to turn log(sum(exp(sum...))) into sum(log(..
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	173 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	174 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	175 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	176 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	177 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	178 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	179 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	180 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	181 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (\|U_{f}\|\|v\|)))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	182 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	183 # + 0.5 \sum_i v_i^2
972 0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	184 # - \sum_i a_i v_i
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	185 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	186 # For negative-one-diagonal P this gives:
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	187 #
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	188 # = - \sum_{k} \log(1 + exp(b_k - 0.5 \sum_i (U_{ik} v_i )^2 / (\|U_{k}\|\|v\|)))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	189 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	190 # + 0.5 \sum_i v_i^2
0b392d1401c5 mcRBM - adding math and comments James Bergstra <bergstrj@iro.umontreal.ca> parents: 967 diff changeset	191 # - \sum_i a_i v_i
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	192
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	193 import sys
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	194 import logging
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	195 import numpy as np
973 aa201f357d7b mcRBM - added numpy import James Bergstra <bergstrj@iro.umontreal.ca> parents: 972 diff changeset	196 import numpy
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	197
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	198 import theano
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	199 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	200 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	201 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	202
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	203 import pylearn
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	204 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	205 from pylearn.io import image_tiling
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	206
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	207 #TODO: This should be in the datasets folder
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	208 import pylearn.datasets.config
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	209 import pylearn.dataset_ops.image_patches
988 fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	210 from pylearn.dataset_ops.protocol import TensorFnDataset
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	211 from pylearn.dataset_ops.memo import memo
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	212 import pylearn
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	213 import scipy.io
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	214 import os
fd243cb2bf0b mcRBM - moved some things to the top of the file James Bergstra <bergstrj@iro.umontreal.ca> parents: 987 diff changeset	215
973 aa201f357d7b mcRBM - added numpy import James Bergstra <bergstrj@iro.umontreal.ca> parents: 972 diff changeset	216
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	217 #TODO: This should be in the nnet part of the library
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	218 def sgd_updates(params, grads, lr):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	219 try:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	220 float(lr)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	221 lr = [lr for p in params]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	222 except TypeError:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	223 pass
974 f2cdcc71ece1 mcRBM - added L1 penalties and normal sign convention to contrastive grad James Bergstra <bergstrj@iro.umontreal.ca> parents: 973 diff changeset	224 updates = [(p, p - plr * gp) for (plr, p, gp) in zip(lr, params, grads)]
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	225 return updates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	226
989 9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	227 def hidden_cov_units_preactivation_given_v(rbm, v, small=0.5):
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	228 """Return argument to the sigmoid that would give mean of covariance hid units
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	229
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	230 See the math at the top of this file for what 'adjusted' means.
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	231
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	232 return b - 0.5 * dot(adjusted(v), U)**2
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	233 """
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	234 (U,W,a,b,c) = rbm
989 9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	235 unit_v = v / (TT.sqrt(TT.mean(v**2, axis=1)+small)).dimshuffle(0,'x') # adjust row norm
9e753ddcc320 mcRBM - cleaned up the preactivation function James Bergstra <bergstrj@iro.umontreal.ca> parents: 988 diff changeset	236 return b - 0.5 * dot(unit_v, U)**2
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	237
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	238 def free_energy_terms_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	239 """Returns theano expression for the terms that are added to form the free energy of
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	240 visible vector `v` in an mcRBM.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	241
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	242 1. Free energy related to covariance hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	243 2. Free energy related to mean hiddens
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	244 3. Free energy related to L2-Norm of `v`
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	245 4. Free energy related to projection of `v` onto biases `a`
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	246 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	247 U, W, a, b, c = rbm
975 38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	248 t0 = -TT.sum(TT.nnet.softplus(hidden_cov_units_preactivation_given_v(rbm, v)),axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	249 t1 = -TT.sum(TT.nnet.softplus(c + dot(v,W)), axis=1)
38e66e0da66a mcRBM - put softplus in directly for num. stability James Bergstra <bergstrj@iro.umontreal.ca> parents: 974 diff changeset	250 t2 = 0.5 * TT.sum(v**2, axis=1)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	251 t3 = -TT.dot(v, a)
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	252 return [t0, t1, t2, t3]
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	253
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	254 def free_energy_given_v(rbm, v):
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	255 """Returns theano expression for free energy of visible vector `v` in an mcRBM
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	256 """
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	257 return sum(free_energy_terms_given_v(rbm,v))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	258
990 e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	259 def contrastive_gradient(rbm, pos_v, neg_v, U_l1_penalty=0, W_l1_penalty=0):
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	260 """Return a list of gradient expressions for the rbm parameters
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	261
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	262 :param pos_v: positive-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	263 :param neg_v: negative-phase sample of visible units
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	264 :param U_l1_penalty: a scalar-valued multiplier on the L1 penalty on U
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	265 :param W_l1_penalty: a scalar-valued multiplier on the L1 penalty on W
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	266 """
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	267 U, W, a, b, c = rbm
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	268 pos_FE = free_energy_given_v(rbm, pos_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	269 neg_FE = free_energy_given_v(rbm, neg_v)
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	270 c0 = (pos_FE - neg_FE).sum()
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	271 c1 = abs(U).sum()*U_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	272 c2 = abs(W).sum()*W_l1_penalty
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	273 cost = c0 + c1 + c2
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	274 rval = theano.tensor.grad(cost, list(rbm))
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	275 return rval
e70e74464170 mcRBM - moved contrastive_gradient to global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 989 diff changeset	276
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	277 def expected_h_g_given_v(rbm, v):
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	278 """Returns tuple (`h`, `g`) of theano expression conditional expectations in an mcRBM.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	279
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	280 `h` is the conditional on the covariance units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	281 `g` is the conditional on the mean units.
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	282
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	283 """
991 d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	284 (U, W, a, b, c) = rbm
d68828c98c38 mcRBM - cleaned up expected hidden unit function James Bergstra <bergstrj@iro.umontreal.ca> parents: 990 diff changeset	285 h = TT.nnet.sigmoid(hidden_cov_units_preactivation_given_v(rbm, v))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	286 g = nnet.sigmoid(c + dot(v,W))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	287 return (h, g)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	288
996 60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	289 def n_visible_units(rbm):
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	290 """Return the number of visible units of this RBM
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	291
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	292 For an RBM made from shared variables, this will return an integer,
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	293 for a purely symbolic RBM this will return a theano expression.
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	294
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	295 """
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	296 W = rbm[1]
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	297 try:
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	298 return W.value.shape[0]
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	299 except AttributeError:
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	300 return W.shape[0]
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	301
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	302
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	303 def sampler(rbm, n_particles, n_visible=None, rng=7823748):
996 60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	304 """Return an `HMC_sampler` that will draw samples from the distribution over visible
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	305 units specified by this RBM.
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	306
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	307 :param n_particles: this many parallel chains will be simulated.
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	308 :param rng: seed or numpy RandomState object to initialize particles, and to drive the simulation.
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	309 """
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	310 if not hasattr(rng, 'randn'):
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	311 rng = np.random.RandomState(rng)
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	312 if n_visible is None:
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	313 n_visible = n_visible_units(rbm)
996 60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	314 rval = HMC_sampler(
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	315 positions = [shared(
996 60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	316 rng.randn(
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	317 n_particles,
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	318 n_visible).astype(floatX),
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	319 name='particles')],
996 60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	320 energy_fn = lambda p : free_energy_given_v(rbm, p[0]),
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	321 seed=int(rng.randint(2**30)))
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	322 return rval
60f279ec0f7f mcRBM - made sampler a global function James Bergstra <bergstrj@iro.umontreal.ca> parents: 995 diff changeset	323
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	324 class MeanCovRBM(object):
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	325 """Container for mcRBM parameters
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	326
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	327 It provides parameter lookup by name, as well as a heuristic for initializing the
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	328 parameters for effective learning.
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	329
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	330 Attributes:
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	331
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	332 - U - the covariance filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	333 - W - the mean filters (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	334 - a - the visible bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	335 - b - the covariance bias (theano shared variable)
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	336 - c - the mean bias (theano shared variable)
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	337
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	338 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	339
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	340 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	341
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	342 n_visible = property(lambda s: s.W.value.shape[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	343
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	344 def __init__(self, U, W, a, b, c):
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	345 self.__dict__.update(locals())
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	346 del self.__dict__['self']
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	347
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	348 def __getitem__(self, idx):
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	349 # support unpacking of this container as if it were a tuple
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	350 return self.params[idx]
71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	351
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	352 @classmethod
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	353 def new_from_dims(cls, n_I, n_K, n_J, rng = 8923402190):
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	354 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	355 Return a MeanCovRBM instance with randomly-initialized parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	356
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	357 :param n_I: input dimensionality
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	358 :param n_K: number of covariance hidden units
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	359 :param n_J: number of mean filters (linear)
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	360 :param rng: seed or numpy RandomState object to initialize params
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	361 """
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	362 if not hasattr(rng, 'randn'):
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	363 rng = np.random.RandomState(rng)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	364
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	365 def shrd(X,name):
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	366 return shared(X.astype(floatX), name=name)
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	367
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	368 # initialization taken from train_mcRBM.py
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	369 return cls(
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	370 U = shrd(0.02 * rng.randn(n_I, n_K),'U'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	371 W = shrd(0.05 * rng.randn(n_I, n_J),'W'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	372 a = shrd(np.ones(n_I)*(0),'a'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	373 b = shrd(np.ones(n_K)*2,'b'),
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	374 c = shrd(np.ones(n_J)*(-2),'c'))
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	375
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	376 def __getstate__(self):
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	377 # unpack shared containers, which may have references to Theano stuff and are not a
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	378 # long-term stable data type.
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	379 return dict(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	380 U = self.U.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	381 W = self.W.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	382 b = self.b.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	383 c = self.c.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	384
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	385 def __setstate__(self, dct):
995 68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	386 d = dict(dct)
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	387 for key in ['U', 'W', 'a', 'b', 'c']:
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	388 d[key] = shared(d[key], name=key)
68ca3ea34e72 mcRBM - cleaned up new_from_dims James Bergstra <bergstrj@iro.umontreal.ca> parents: 994 diff changeset	389 self.__init__(**d)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	390
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	391 if __name__ == '__main__':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	392
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	393 dataset='MAR'
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	394 if dataset == 'MAR':
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	395 n_vis=105
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	396 n_patches=10240
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	397 else:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	398 R,C= 16,16 # the size of image patches
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	399 n_vis=R*C
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	400 n_patches=100000
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	401
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	402 n_train_iters=5000
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	403
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	404 n_burnin_steps=10000
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	405
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	406 l1_penalty=1e-3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	407 no_l1_epochs = 10
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	408 effective_l1_penalty=0.0
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	409
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	410 epoch_size=n_patches
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	411 batchsize = 128
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	412 lr = 0.075 / batchsize
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	413 s_lr = TT.scalar()
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	414 s_l1_penalty=TT.scalar()
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	415 n_K=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	416 n_J=100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	417
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	418 rbm = MeanCovRBM.new_from_dims(n_I=n_vis, n_K=n_K, n_J=n_J)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	419
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	420 smplr = sampler(rbm, n_particles=batchsize)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	421
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	422 def l2(X):
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	423 return numpy.sqrt((X**2).sum())
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	424 if dataset == 'MAR':
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	425 tile = pylearn.dataset_ops.image_patches.save_filters_of_ranzato_hinton_2010
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	426 else:
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	427 def tile(X, fname):
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	428 _img = image_tiling.tile_raster_images(X,
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	429 img_shape=(R,C),
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	430 min_dynamic_range=1e-2)
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	431 image_tiling.save_tiled_raster_images(_img, fname)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	432
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	433 batch_idx = TT.iscalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	434
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	435 if dataset == 'MAR':
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	436 train_batch = pylearn.dataset_ops.image_patches.ranzato_hinton_2010_op(batch_idx * batchsize + np.arange(batchsize))
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	437 else:
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	438 train_batch = pylearn.dataset_ops.image_patches.image_patches(
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	439 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	440 dims = (n_patches,R,C),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	441 center=True,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	442 unitvar=True,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	443 dtype=floatX,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	444 rasterized=True)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	445
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	446 imgs_fn = function([batch_idx], outputs=train_batch)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	447
997 71b0132b694a mcRBM - removed container logic that was redundant with global methods James Bergstra <bergstrj@iro.umontreal.ca> parents: 996 diff changeset	448 grads = contrastive_gradient(rbm,
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	449 pos_v=train_batch,
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	450 neg_v=smplr.positions[0],
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	451 U_l1_penalty=s_l1_penalty,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	452 W_l1_penalty=s_l1_penalty)
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	453 sgd_ups = sgd_updates(
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	454 rbm.params,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	455 grads,
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	456 lr=[2s_lr, .2s_lr, .02s_lr, .1s_lr, .02*s_lr ])
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	457 learn_fn = function([batch_idx, s_lr, s_l1_penalty],
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	458 outputs=[
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	459 grads[0].norm(2),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	460 (sgd_ups[0][1] - sgd_ups[0][0]).norm(2),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	461 (sgd_ups[1][1] - sgd_ups[1][0]).norm(2),
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	462 ],
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	463 updates = sgd_ups)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	464
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	465 print "Learning..."
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	466 normVF=1
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	467 last_epoch = -1
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	468 for jj in xrange(n_train_iters):
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	469 epoch = jj*batchsize / epoch_size
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	470
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	471 print_jj = epoch != last_epoch
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	472 last_epoch = epoch
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	473
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	474 if epoch > 10:
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	475 break
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	476
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	477 if print_jj:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	478 tile(imgs_fn(jj), "imgs_%06i.png"%jj)
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	479 tile(smplr.positions[0].value, "sample_%06i.png"%jj)
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	480 tile(rbm.U.value.T, "U_%06i.png"%jj)
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	481 tile(rbm.W.value.T, "W_%06i.png"%jj)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	482
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	483 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	484
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	485 print 'l2(U)', l2(rbm.U.value),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	486 print 'l2(W)', l2(rbm.W.value)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	487
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	488 print 'U min max', rbm.U.value.min(), rbm.U.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	489 print 'W min max', rbm.W.value.min(), rbm.W.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	490 print 'a min max', rbm.a.value.min(), rbm.a.value.max(),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	491 print 'b min max', rbm.b.value.min(), rbm.b.value.max(),
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	492 print 'c min max', rbm.c.value.min(), rbm.c.value.max()
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	493
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	494 print 'parts min', smplr.positions[0].value.min(),
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	495 print 'max',smplr.positions[0].value.max(),
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	496 print 'HMC step', smplr.stepsize,
8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	497 print 'arate', smplr.avg_acceptance_rate
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	498
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	499 smplr.simulate()
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	500
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	501 l2_of_Ugrad = learn_fn(jj,
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	502 lr/max(1, jj/(20*epoch_size/batchsize)),
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	503 effective_l1_penalty)
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	504
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	505 if print_jj:
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	506 print 'l2(U_grad)', float(l2_of_Ugrad[0]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	507 print 'l2(U_inc)', float(l2_of_Ugrad[1]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	508 print 'l2(W_inc)', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	509 #print 'FE+', float(l2_of_Ugrad[2]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	510 #print 'FE+[0]', float(l2_of_Ugrad[3]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	511 #print 'FE+[1]', float(l2_of_Ugrad[4]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	512 #print 'FE+[2]', float(l2_of_Ugrad[5]),
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	513 #print 'FE+[3]', float(l2_of_Ugrad[6])
979 2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	514
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	515 if jj == no_l1_epochs * epoch_size/batchsize:
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	516 print "Activating L1 weight decay"
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	517 effective_l1_penalty = 1e-3
2a53384d9742 mcRBM - hacks to driver James Bergstra <bergstrj@iro.umontreal.ca> parents: 978 diff changeset	518
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	519 # weird normalization technique...
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	520 # It constrains all the columns of the matrix to have the same length
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	521 # But the matrix itself is re-scaled to have an arbitrary abslute size.
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	522 U = rbm.U.value
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	523 U_norms = np.sqrt((U*U).sum(axis=0))
998 8ba8b08e0442 added the image_patches dataset used in RanzatoHinton2010 James Bergstra <bergstrj@iro.umontreal.ca> parents: 997 diff changeset	524 assert len(U_norms) == n_K
992 30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	525 normVF = .95 * normVF + .05 * np.mean(U_norms)
30b7c4defb6c mcRBM - it works and committing it is taking forever... lets try this approach James Bergstra <bergstrj@iro.umontreal.ca> parents: 991 diff changeset	526 rbm.U.value = rbm.U.value * normVF/U_norms
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	527

Mercurial > pylearn

annotate pylearn/algorithms/mcRBM.py @ 998:8ba8b08e0442