pylearn: pylearn/algorithms/mcRBM.py annotate

author	James Bergstra <bergstrj@iro.umontreal.ca>
date	Fri, 20 Aug 2010 13:58:56 -0400
parents
children	0b392d1401c5

rev	line source
967 90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	1 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	2 This file implements the Mean & Covariance RBM discussed in
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	4 Ranzato, M. and Hinton, G. E. (2010)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	5 Modeling pixel means and covariances using factored third-order Boltzmann machines.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	6 IEEE Conference on Computer Vision and Pattern Recognition.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	7
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	8 and performs one of the experiments on CIFAR-10 discussed in that paper.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	9
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	10
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	11 Math
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	12 ====
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	13
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	14 Energy of "covariance RBM"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	15
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	16 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	17 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	18 "vector element f" "vector element f"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	19
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	20 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	21 diagonal entries, so it is helpful to see this as a simpler equation:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	22
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	23 E = \sum_f h_f ( \sum_i C_{if} v_i )^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	24
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	25
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	26
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	27 Full Energy of mean and Covariance RBM, with
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	28 :math:`h_k = h_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	29 :math:`g_j = h_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	30 :math:`b_k = b_k^{(c)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	31 :math:`c_j = b_j^{(m)}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	32 :math:`U_{if} = C_{if}`,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	33
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	34 :
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	35
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	36 E (v, h, g) =
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	37 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / \|U_{*f}\|^2 \|v\|^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	38 - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	39 + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	40 - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	41 - \sum_j c_j g_j
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	42
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	43
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	44 Conventions in this file
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	45 ========================
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	46
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	47 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	48 more convenient.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	49
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	50
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	51 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	52 Suppose we have
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	53 I input dimensions,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	54 F squared filters,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	55 J mean variables, and
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	56 K covariance variables.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	57 The mcRBM is parametrized by 5 variables:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	58
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	59 - `P`, a matrix (probably sparse) of pooling (F x K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	60 - `U`, a matrix whose rows are visible covariance directions (I x F)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	61 - `W`, a matrix whose rows are visible mean directions (I x J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	62 - `b`, a vector of hidden covariance biases (K)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	63 - `c`, a vector of hidden mean biases (J)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	64
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	65 Matrices are generally layed out according to a C-order convention.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	66
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	67 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	68
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	69 # Free energy is the marginal energy of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	70 # Recall:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	71 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	72 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	73 # Derivation, in which partition functions are ignored.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	74 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	75 # E(v) = -\log(Q(v))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	76 # = -\log( \sum_{h,g} Q(v,h,g))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	77 # = -\log( \sum_{h,g} exp(-E(v,h,g)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	78 # = -\log( \sum_{h,g} exp(-
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	79 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	80 # - \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	81 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	82 # - \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	83 # - \sum_j c_j g_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	84 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	85 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	86 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	87 # - 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	88 # ) * \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	89 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	90 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	91 # = -\log( \sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	92 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\|\|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	93 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	94 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	95 # -\log( \sum_{g} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	96 # + \sum_j \sum_i W_{ij} g_j v_i
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	97 # + \sum_j c_j g_j )))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	98 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	99 # = -\log(\sum_{h} exp(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	100 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (\|U_{f}\| \|v\|)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	101 # + \sum_k b_k h_k
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	102 # ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	103 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	104 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	105 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (\|U_{f}\| # \|v\|)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	106 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	107 # + 0.5 \sum_i v_i^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	108
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	109 import sys
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	110 import logging
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	111 import numpy as np
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	112 from theano import function, shared, dot
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	113 from theano import tensor as TT
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	114 import theano.sparse #installs the sparse shared var handler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	115 floatX = theano.config.floatX
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	116
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	117 from pylearn.sampling.hmc import HMC_sampler
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	118 from pylearn.io import image_tiling
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	119
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	120 from sparse_coding import numpy_project_onto_ball
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	121
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	122 #TODO: This should be in the nnet part of the library
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	123 def sgd_updates(params, grads, lr):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	124 try:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	125 float(lr)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	126 lr = [lr for p in params]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	127 except TypeError:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	128 pass
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	129 updates = [(p, p + plr * gp) for (plr, p, gp) in zip(lr, params, grads)]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	130 return updates
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	131
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	132 def as_shared(x, name=None, dtype=floatX):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	133 if hasattr(x, 'type'):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	134 return x
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	135 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	136 if 'float' in str(x.dtype):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	137 return shared(x.astype(floatX), name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	138 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	139 return shared(x, name=name)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	140
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	141 def hidden_cov_units_preactivation_given_v(rbm, v, small=1e-8):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	142 (U,W,a,b,c) = rbm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	143 unit_v = v / (TT.sqrt(TT.sum(v**2, axis=1))+small).dimshuffle(0,'x') # unit rows
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	144 unit_U = U # assuming unit cols!
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	145 #unit_U = U / (TT.sqrt(TT.sum(U**2, axis=0))+small) #unit cols
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	146 return b - 0.5 * dot(unit_v, unit_U)**2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	147
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	148 def free_energy_given_v(rbm, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	149 """Returns theano expression for free energy of visible vector `v` in an mcRBM
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	150
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	151 An mcRBM is parametrized
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	152 by `U`, `W`, `b`, `c`.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	153 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	154
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	155
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	156 The free energy of v is what we need for learning and hybrid Monte-carlo negative-phase
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	157 sampling.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	158
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	159 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	160 U, W, a, b, c = rbm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	161
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	162 t0 = -TT.sum(TT.log1p(TT.exp(hidden_cov_units_preactivation_given_v(rbm, v))),axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	163 t1 = -TT.sum(TT.log1p(TT.exp(c + dot(v,W))), axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	164 t2 = 0.5 * TT.sum(v**2, axis=1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	165 t3 = -TT.dot(v, a)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	166 return t0 + t1 + t2 + t3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	167
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	168 def expected_h_g_given_v(P, U, W, b, c, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	169 """Returns theano expression conditional expectations (`h`, `g`) in an mcRBM.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	170
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	171 An mcRBM is parametrized
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	172 by `U`, `W`, `b`, `c`.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	173 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	174
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	175
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	176 The conditional E[h, g \| v] is what we need to classify images.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	177 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	178 raise NotImplementedError()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	179
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	180 #TODO: check to see if these args should be negated?
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	181
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	182 if P is None:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	183 h = nnet.sigmoid(b + 0.5 * cosines(v,U))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	184 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	185 h = nnet.sigmoid(b + 0.5 * dot(cosines(v,U), P))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	186 g = nnet.sigmoid(c + dot(v,W))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	187 return (h, g)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	188
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	189 class MeanCovRBM(object):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	190 """Container for mcRBM parameters that gives more convenient access to mcRBM methods.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	191 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	192
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	193 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	194
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	195 n_visible = property(lambda s: s.W.value.shape[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	196
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	197 def __init__(self, U, W, a, b, c):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	198 self.U = as_shared(U, 'U')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	199 self.W = as_shared(W, 'W')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	200 self.a = as_shared(a, 'a')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	201 self.b = as_shared(b, 'b')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	202 self.c = as_shared(c, 'c')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	203
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	204 assert self.b.type.dtype == 'float32'
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	205
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	206 @classmethod
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	207 def new_from_dims(cls,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	208 n_I, # input dimensionality
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	209 n_K, # number of covariance hidden units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	210 n_F, # number of covariance filters (squared)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	211 n_J, # number of mean filters (linear)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	212 seed = 8923402190,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	213 ):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	214 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	215 Return a MeanCovRBM instance with randomly-initialized parameters.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	216 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	217
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	218
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	219 if 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	220 if P_init == 'diag':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	221 if n_K != n_F:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	222 raise ValueError('cannot use diagonal initialization of non-square P matrix')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	223 import scipy.sparse
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	224 P = -scipy.sparse.identity(n_K).tocsr()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	225 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	226 raise NotImplementedError()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	227
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	228 rng = np.random.RandomState(seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	229
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	230 # initialization taken from Marc'Aurelio
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	231
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	232 return cls(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	233 U = numpy_project_onto_ball(rng.randn(n_I, n_F).T).T,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	234 W = rng.randn(n_I, n_J)/np.sqrt((n_I+n_J)/2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	235 a = np.ones(n_I)*(-2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	236 b = np.ones(n_K)*2,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	237 c = np.zeros(n_J),)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	238
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	239 def __getstate__(self):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	240 # unpack shared containers, which may have references to Theano stuff
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	241 # and are not a long-term stable data type.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	242 return dict(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	243 U = self.U.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	244 W = self.W.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	245 b = self.b.value,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	246 c = self.c.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	247
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	248 def __setstate__(self, dct):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	249 self.__init__(**dct) # calls as_shared on pickled arrays
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	250
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	251 def hmc_sampler(self, n_particles=100, seed=7823748):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	252 return HMC_sampler(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	253 positions = [as_shared(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	254 np.random.RandomState(seed^20893).rand(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	255 n_particles,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	256 self.n_visible ))],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	257 energy_fn = lambda p : self.free_energy_given_v(p[0]),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	258 seed=seed)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	259
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	260 def free_energy_given_v(self, v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	261 return free_energy_given_v(self.params, v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	262
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	263 def contrastive_gradient(self, pos_v, neg_v):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	264 """Return a list of gradient expressions for self.params
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	265
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	266 :param pos_v: positive-phase sample of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	267 :param neg_v: negative-phase sample of visible units
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	268 """
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	269 pos_FE = self.free_energy_given_v(pos_v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	270 neg_FE = self.free_energy_given_v(neg_v)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	271
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	272 gpos_FE = theano.tensor.grad(pos_FE.sum(), self.params)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	273 gneg_FE = theano.tensor.grad(neg_FE.sum(), self.params)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	274 return [ gn - gp for (gp,gn) in zip(gpos_FE, gneg_FE)]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	275
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	276 if __name__ == '__main__':
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	277
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	278 print >> sys.stderr, "TODO: use P matrix (aka FH matrix)"
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	279
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	280 R,C= 8,8 # the size of image patches
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	281 l1_penalty=1e-3
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	282 no_l1_epochs = 10
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	283
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	284 epoch_size=50000
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	285 batchsize = 128
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	286 lr = 0.075 / batchsize
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	287 s_lr = TT.scalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	288 n_K=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	289 n_F=256
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	290 n_J=100
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	291
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	292 rbm = MeanCovRBM.new_from_dims(n_I=R*C,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	293 n_K=n_K,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	294 n_J=n_J,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	295 n_F=n_F,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	296 )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	297
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	298 sampler = rbm.hmc_sampler(n_particles=100)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	299
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	300 from pylearn.dataset_ops import image_patches
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	301
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	302 batch_idx = TT.iscalar()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	303 train_batch = image_patches.image_patches(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	304 s_idx = (batch_idx * batchsize + np.arange(batchsize)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	305 dims = (1000,R,C),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	306 dtype=floatX,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	307 rasterized=True)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	308
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	309 grads = rbm.contrastive_gradient(pos_v=train_batch, neg_v=sampler.positions[0])
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	310
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	311 learn_fn = function([batch_idx, s_lr],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	312 outputs=[
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	313 grads[0].norm(2),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	314 rbm.U.norm(2)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	315 ],
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	316 updates = sgd_updates(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	317 rbm.params,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	318 grads,
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	319 lr=[2s_lr, .2s_lr, .02s_lr, .1s_lr, .02*s_lr ]))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	320
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	321 for jj in xrange(10000):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	322 sampler.simulate()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	323 l2_of_Ugrad = learn_fn(jj, lr/max(1, jj/(20*epoch_size/batchsize)))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	324
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	325 if jj > no_l1_epochs * epoch_size/batchsize:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	326 rbm.U.value -= l1_penalty * np.sign(rbm.U.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	327 rbm.W.value -= l1_penalty * np.sign(rbm.W.value)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	328
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	329 if jj % 5 == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	330 rbm.U.value = numpy_project_onto_ball(rbm.U.value.T).T
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	331
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	332 if ((jj < 10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	333 or (jj < 100 and 0==jj%10)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	334 or (jj < 1000 and 0==jj%100)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	335 or (jj < 10000 and 0==jj%1000)):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	336 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize), l2_of_Ugrad
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	337 print 'neg particles', sampler.positions[0].value.min(), sampler.positions[0].value.max()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	338 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	339 image_tiling.tile_raster_images(sampler.positions[0].value, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	340 "sample_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	341 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	342 image_tiling.tile_raster_images(rbm.U.value.T, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	343 "U_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	344 image_tiling.save_tiled_raster_images(
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	345 image_tiling.tile_raster_images(rbm.W.value.T, (R,C)),
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	346 "W_%06i.png"%jj)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	347
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	348
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	349
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	350 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	351 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	352 # Marc'Aurelio Ranzato's code
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	353 #
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	354 ######################################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	355 # compute the value of the free energy at a given input
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	356 # F = - sum log(1+exp(- .5 FH (VF data/norm(data))^2 + bias_cov)) +...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	357 # - sum log(1+exp(w_mean data + bias_mean)) + ...
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	358 # - bias_vis data + 0.5 data^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	359 # NOTE: FH is constrained to be positive
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	360 # (in the paper the sign is negative but the sign in front of it is also flipped)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	361 def compute_energy_mcRBM(data,normdata,vel,energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,t1,t2,t6,feat,featsq,feat_mean,length,lengthsq,normcoeff,small,num_vis):
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	362 # normalize input data vectors
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	363 data.mult(data, target = t6) # DxP (nr input dims x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	364 t6.sum(axis = 0, target = lengthsq) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	365 lengthsq.mult(0.5, target = energy) # energy of quadratic regularization term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	366 lengthsq.mult(1./num_vis) # normalize by number of components (like std)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	367
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	368 lengthsq.add(small) # small prevents division by 0
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	369 # energy_j = \sum_i 0.5 data_ij ^2
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	370 # lengthsq_j = 1/ (\sum_i data_ij ^2 + small)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	371 cmt.sqrt(lengthsq, target = length)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	372 # length_j = sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	373 length.reciprocal(target = normcoeff) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	374 # normcoef_j = 1/sqrt(lengthsq_j)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	375 data.mult_by_row(normcoeff, target = normdata) # normalized data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	376 # normdata is like data, but cols have unit L2 norm
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	377
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	378 ## potential
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	379 # covariance contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	380 cmt.dot(VF.T, normdata, target = feat) # HxP (nr factors x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	381 feat.mult(feat, target = featsq) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	382
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	383 # featsq is the squared cosines (VF with data)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	384 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	385 t1.mult(-0.5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	386 t1.add_col_vec(bias_cov) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	387 cmt.exp(t1) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	388 t1.add(1, target = t2) # OxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	389 cmt.log(t2)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	390 t2.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	391 energy.add_sums(t2, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	392 # mean contribution
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	393 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	394 feat_mean.add_col_vec(bias_mean) # HxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	395 cmt.exp(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	396 feat_mean.add(1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	397 cmt.log(feat_mean)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	398 feat_mean.mult(-1)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	399 energy.add_sums(feat_mean, axis=0)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	400 # visible bias term
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	401 data.mult_by_col(bias_vis, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	402 t6.mult(-1) # DxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	403 energy.add_sums(t6, axis=0) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	404 # kinetic
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	405 vel.mult(vel, target = t6)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	406 energy.add_sums(t6, axis = 0, mult = .5)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	407
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	408 ######################################################
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	409 # mcRBM trainer: sweeps over the training set.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	410 # For each batch of samples compute derivatives to update the parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	411 # at the training samples and at the negative samples drawn calling HMC sampler.
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	412 def train_mcRBM():
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	413
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	414 config = ConfigParser()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	415 config.read('input_configuration')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	416
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	417 verbose = config.getint('VERBOSITY','verbose')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	418
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	419 num_epochs = config.getint('MAIN_PARAMETER_SETTING','num_epochs')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	420 batch_size = config.getint('MAIN_PARAMETER_SETTING','batch_size')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	421 startFH = config.getint('MAIN_PARAMETER_SETTING','startFH')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	422 startwd = config.getint('MAIN_PARAMETER_SETTING','startwd')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	423 doPCD = config.getint('MAIN_PARAMETER_SETTING','doPCD')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	424
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	425 # model parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	426 num_fac = config.getint('MODEL_PARAMETER_SETTING','num_fac')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	427 num_hid_cov = config.getint('MODEL_PARAMETER_SETTING','num_hid_cov')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	428 num_hid_mean = config.getint('MODEL_PARAMETER_SETTING','num_hid_mean')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	429 apply_mask = config.getint('MODEL_PARAMETER_SETTING','apply_mask')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	430
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	431 # load data
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	432 data_file_name = config.get('DATA','data_file_name')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	433 d = loadmat(data_file_name) # input in the format PxD (P vectorized samples with D dimensions)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	434 totnumcases = d["whitendata"].shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	435 d = d["whitendata"][0:floor(totnumcases/batch_size)*batch_size,:].copy()
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	436 totnumcases = d.shape[0]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	437 num_vis = d.shape[1]
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	438 num_batches = int(totnumcases/batch_size)
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	439 dev_dat = cmt.CUDAMatrix(d.T) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	440
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	441 # training parameters
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	442 epsilon = config.getfloat('OPTIMIZER_PARAMETERS','epsilon')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	443 epsilonVF = 2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	444 epsilonFH = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	445 epsilonb = 0.02*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	446 epsilonw_mean = 0.2*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	447 epsilonb_mean = 0.1*epsilon
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	448 weightcost_final = config.getfloat('OPTIMIZER_PARAMETERS','weightcost_final')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	449
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	450 # HMC setting
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	451 hmc_step_nr = config.getint('HMC_PARAMETERS','hmc_step_nr')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	452 hmc_step = 0.01
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	453 hmc_target_ave_rej = config.getfloat('HMC_PARAMETERS','hmc_target_ave_rej')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	454 hmc_ave_rej = hmc_target_ave_rej
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	455
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	456 # initialize weights
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	457 VF = cmt.CUDAMatrix(np.array(0.02 * np.random.randn(num_vis, num_fac), dtype=np.float32, order='F')) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	458 if apply_mask == 0:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	459 FH = cmt.CUDAMatrix( np.array( np.eye(num_fac,num_hid_cov), dtype=np.float32, order='F') ) # HxO
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	460 else:
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	461 dd = loadmat('your_FHinit_mask_file.mat') # see CVPR2010paper_material/topo2D_3x3_stride2_576filt.mat for an example
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	462 FH = cmt.CUDAMatrix( np.array( dd["FH"], dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	463 bias_cov = cmt.CUDAMatrix( np.array(2.0*np.ones((num_hid_cov, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	464 bias_vis = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	465 w_mean = cmt.CUDAMatrix( np.array( 0.05 * np.random.randn(num_vis, num_hid_mean), dtype=np.float32, order='F') ) # VxH
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	466 bias_mean = cmt.CUDAMatrix( np.array( -2.0*np.ones((num_hid_mean,1)), dtype=np.float32, order='F') )
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	467
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	468 # initialize variables to store derivatives
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	469 VFinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_fac)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	470 FHinc = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	471 bias_covinc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_cov, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	472 bias_visinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	473 w_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_hid_mean)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	474 bias_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_mean, 1)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	475
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	476 # initialize temporary storage
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	477 data = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	478 normdata = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	479 negdataini = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	480 feat = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	481 featsq = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	482 negdata = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	483 old_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	484 new_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	485 gradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	486 normgradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	487 thresh = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	488 feat_mean = cmt.CUDAMatrix( np.array(np.empty((num_hid_mean, batch_size)), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	489 vel = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	490 length = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	491 lengthsq = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	492 normcoeff = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	493 if apply_mask==1: # this used to constrain very large FH matrices only allowing to change values in a neighborhood
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	494 dd = loadmat('your_FHinit_mask_file.mat')
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	495 mask = cmt.CUDAMatrix( np.array(dd["mask"], dtype=np.float32, order='F'))
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	496 normVF = 1
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	497 small = 0.5
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	498
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	499 # other temporary vars
90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	500 t1 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))

967

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

1 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

2 This file implements the Mean & Covariance RBM discussed in

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

3

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

4 Ranzato, M. and Hinton, G. E. (2010)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

5 Modeling pixel means and covariances using factored third-order Boltzmann machines.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

6 IEEE Conference on Computer Vision and Pattern Recognition.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

7

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

8 and performs one of the experiments on CIFAR-10 discussed in that paper.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

9

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

10

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

11 Math

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

12 ====

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

13

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

14 Energy of "covariance RBM"

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

15

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

16 E = -0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i C_{if} v_i )^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

17 = -0.5 \sum_f (\sum_k P_{fk} h_k) ( \sum_i C_{if} v_i )^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

18 "vector element f" "vector element f"

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

19

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

20 In some parts of the paper, the P matrix is chosen to be a diagonal matrix with non-positive

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

21 diagonal entries, so it is helpful to see this as a simpler equation:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

22

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

23 E = \sum_f h_f ( \sum_i C_{if} v_i )^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

24

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

25

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

26

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

27 Full Energy of mean and Covariance RBM, with

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

28 :math:`h_k = h_k^{(c)}`,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

29 :math:`g_j = h_j^{(m)}`,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

30 :math:`b_k = b_k^{(c)}`,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

31 :math:`c_j = b_j^{(m)}`,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

32 :math:`U_{if} = C_{if}`,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

33

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

34 :

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

35

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

36 E (v, h, g) =

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

37 - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / |U_{*f}|^2 |v|^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

38 - \sum_k b_k h_k

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

39 + 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

40 - \sum_j \sum_i W_{ij} g_j v_i

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

41 - \sum_j c_j g_j

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

42

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

43

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

44 Conventions in this file

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

45 ========================

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

46

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

47 This file contains some global functions, as well as a class (MeanCovRBM) that makes using them a little

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

48 more convenient.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

49

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

50

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

51 Global functions like `free_energy` work on an mcRBM as parametrized in a particular way.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

52 Suppose we have

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

53 I input dimensions,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

54 F squared filters,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

55 J mean variables, and

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

56 K covariance variables.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

57 The mcRBM is parametrized by 5 variables:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

58

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

59 - `P`, a matrix (probably sparse) of pooling (F x K)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

60 - `U`, a matrix whose rows are visible covariance directions (I x F)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

61 - `W`, a matrix whose rows are visible mean directions (I x J)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

62 - `b`, a vector of hidden covariance biases (K)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

63 - `c`, a vector of hidden mean biases (J)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

64

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

65 Matrices are generally layed out according to a C-order convention.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

66

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

67 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

68

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

69 # Free energy is the marginal energy of visible units

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

70 # Recall:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

71 # Q(x) = exp(-E(x))/Z ==> -log(Q(x)) - log(Z) = E(x)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

72 #

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

73 # Derivation, in which partition functions are ignored.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

74 #

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

75 # E(v) = -\log(Q(v))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

76 # = -\log( \sum_{h,g} Q(v,h,g))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

77 # = -\log( \sum_{h,g} exp(-E(v,h,g)))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

78 # = -\log( \sum_{h,g} exp(-

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

79 # - 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

80 # - \sum_k b_k h_k

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

81 # + 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

82 # - \sum_j \sum_i W_{ij} g_j v_i

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

83 # - \sum_j c_j g_j ))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

84 # = -\log( \sum_{h} exp(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

85 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}| * |v|)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

86 # + \sum_k b_k h_k

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

87 # - 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

88 # ) * \sum_{g} exp(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

89 # + \sum_j \sum_i W_{ij} g_j v_i

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

90 # + \sum_j c_j g_j )))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

91 # = -\log( \sum_{h} exp(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

92 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|*|v|)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

93 # + \sum_k b_k h_k

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

94 # ))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

95 # -\log( \sum_{g} exp(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

96 # + \sum_j \sum_i W_{ij} g_j v_i

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

97 # + \sum_j c_j g_j )))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

98 # + 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

99 # = -\log(\sum_{h} exp(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

100 # + 0.5 \sum_f \sum_k P_{fk} h_k ( \sum_i U_{if} v_i )^2 / (|U_{*f}|* |v|)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

101 # + \sum_k b_k h_k

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

102 # ))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

103 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

104 # + 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

105 # = - \sum_{k} \log(1 + exp(b_k + 0.5 \sum_f P_{fk}( \sum_i U_{if} v_i )^2 / (|U_{*f}|* # |v|)))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

106 # - \sum_{j} \log(1 + exp(\sum_i W_{ij} v_i + c_j ))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

107 # + 0.5 \sum_i v_i^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

108

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

109 import sys

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

110 import logging

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

111 import numpy as np

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

112 from theano import function, shared, dot

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

113 from theano import tensor as TT

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

114 import theano.sparse #installs the sparse shared var handler

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

115 floatX = theano.config.floatX

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

116

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

117 from pylearn.sampling.hmc import HMC_sampler

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

118 from pylearn.io import image_tiling

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

119

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

120 from sparse_coding import numpy_project_onto_ball

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

121

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

122 #TODO: This should be in the nnet part of the library

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

123 def sgd_updates(params, grads, lr):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

124 try:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

125 float(lr)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

126 lr = [lr for p in params]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

127 except TypeError:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

128 pass

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

129 updates = [(p, p + plr * gp) for (plr, p, gp) in zip(lr, params, grads)]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

130 return updates

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

131

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

132 def as_shared(x, name=None, dtype=floatX):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

133 if hasattr(x, 'type'):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

134 return x

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

135 else:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

136 if 'float' in str(x.dtype):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

137 return shared(x.astype(floatX), name=name)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

138 else:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

139 return shared(x, name=name)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

140

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

141 def hidden_cov_units_preactivation_given_v(rbm, v, small=1e-8):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

142 (U,W,a,b,c) = rbm

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

143 unit_v = v / (TT.sqrt(TT.sum(v**2, axis=1))+small).dimshuffle(0,'x') # unit rows

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

144 unit_U = U # assuming unit cols!

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

145 #unit_U = U / (TT.sqrt(TT.sum(U**2, axis=0))+small) #unit cols

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

146 return b - 0.5 * dot(unit_v, unit_U)**2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

147

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

148 def free_energy_given_v(rbm, v):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

149 """Returns theano expression for free energy of visible vector `v` in an mcRBM

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

150

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

151 An mcRBM is parametrized

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

152 by `U`, `W`, `b`, `c`.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

153 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

154

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

155

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

156 The free energy of v is what we need for learning and hybrid Monte-carlo negative-phase

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

157 sampling.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

158

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

159 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

160 U, W, a, b, c = rbm

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

161

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

162 t0 = -TT.sum(TT.log1p(TT.exp(hidden_cov_units_preactivation_given_v(rbm, v))),axis=1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

163 t1 = -TT.sum(TT.log1p(TT.exp(c + dot(v,W))), axis=1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

164 t2 = 0.5 * TT.sum(v**2, axis=1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

165 t3 = -TT.dot(v, a)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

166 return t0 + t1 + t2 + t3

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

167

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

168 def expected_h_g_given_v(P, U, W, b, c, v):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

169 """Returns theano expression conditional expectations (`h`, `g`) in an mcRBM.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

170

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

171 An mcRBM is parametrized

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

172 by `U`, `W`, `b`, `c`.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

173 See module - level documentation for explanations of the `U`, `W`, `b` and `c` parameters.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

174

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

175

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

176 The conditional E[h, g | v] is what we need to classify images.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

177 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

178 raise NotImplementedError()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

179

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

180 #TODO: check to see if these args should be negated?

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

181

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

182 if P is None:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

183 h = nnet.sigmoid(b + 0.5 * cosines(v,U))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

184 else:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

185 h = nnet.sigmoid(b + 0.5 * dot(cosines(v,U), P))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

186 g = nnet.sigmoid(c + dot(v,W))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

187 return (h, g)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

188

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

189 class MeanCovRBM(object):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

190 """Container for mcRBM parameters that gives more convenient access to mcRBM methods.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

191 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

192

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

193 params = property(lambda s: [s.U, s.W, s.a, s.b, s.c])

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

194

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

195 n_visible = property(lambda s: s.W.value.shape[0])

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

196

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

197 def __init__(self, U, W, a, b, c):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

198 self.U = as_shared(U, 'U')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

199 self.W = as_shared(W, 'W')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

200 self.a = as_shared(a, 'a')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

201 self.b = as_shared(b, 'b')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

202 self.c = as_shared(c, 'c')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

203

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

204 assert self.b.type.dtype == 'float32'

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

205

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

206 @classmethod

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

207 def new_from_dims(cls,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

208 n_I, # input dimensionality

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

209 n_K, # number of covariance hidden units

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

210 n_F, # number of covariance filters (squared)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

211 n_J, # number of mean filters (linear)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

212 seed = 8923402190,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

213 ):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

214 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

215 Return a MeanCovRBM instance with randomly-initialized parameters.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

216 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

217

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

218

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

219 if 0:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

220 if P_init == 'diag':

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

221 if n_K != n_F:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

222 raise ValueError('cannot use diagonal initialization of non-square P matrix')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

223 import scipy.sparse

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

224 P = -scipy.sparse.identity(n_K).tocsr()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

225 else:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

226 raise NotImplementedError()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

227

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

228 rng = np.random.RandomState(seed)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

229

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

230 # initialization taken from Marc'Aurelio

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

231

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

232 return cls(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

233 U = numpy_project_onto_ball(rng.randn(n_I, n_F).T).T,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

234 W = rng.randn(n_I, n_J)/np.sqrt((n_I+n_J)/2),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

235 a = np.ones(n_I)*(-2),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

236 b = np.ones(n_K)*2,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

237 c = np.zeros(n_J),)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

238

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

239 def __getstate__(self):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

240 # unpack shared containers, which may have references to Theano stuff

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

241 # and are not a long-term stable data type.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

242 return dict(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

243 U = self.U.value,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

244 W = self.W.value,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

245 b = self.b.value,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

246 c = self.c.value)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

247

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

248 def __setstate__(self, dct):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

249 self.__init__(**dct) # calls as_shared on pickled arrays

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

250

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

251 def hmc_sampler(self, n_particles=100, seed=7823748):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

252 return HMC_sampler(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

253 positions = [as_shared(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

254 np.random.RandomState(seed^20893).rand(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

255 n_particles,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

256 self.n_visible ))],

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

257 energy_fn = lambda p : self.free_energy_given_v(p[0]),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

258 seed=seed)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

259

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

260 def free_energy_given_v(self, v):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

261 return free_energy_given_v(self.params, v)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

262

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

263 def contrastive_gradient(self, pos_v, neg_v):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

264 """Return a list of gradient expressions for self.params

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

265

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

266 :param pos_v: positive-phase sample of visible units

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

267 :param neg_v: negative-phase sample of visible units

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

268 """

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

269 pos_FE = self.free_energy_given_v(pos_v)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

270 neg_FE = self.free_energy_given_v(neg_v)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

271

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

272 gpos_FE = theano.tensor.grad(pos_FE.sum(), self.params)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

273 gneg_FE = theano.tensor.grad(neg_FE.sum(), self.params)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

274 return [ gn - gp for (gp,gn) in zip(gpos_FE, gneg_FE)]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

275

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

276 if __name__ == '__main__':

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

277

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

278 print >> sys.stderr, "TODO: use P matrix (aka FH matrix)"

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

279

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

280 R,C= 8,8 # the size of image patches

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

281 l1_penalty=1e-3

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

282 no_l1_epochs = 10

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

283

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

284 epoch_size=50000

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

285 batchsize = 128

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

286 lr = 0.075 / batchsize

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

287 s_lr = TT.scalar()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

288 n_K=256

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

289 n_F=256

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

290 n_J=100

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

291

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

292 rbm = MeanCovRBM.new_from_dims(n_I=R*C,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

293 n_K=n_K,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

294 n_J=n_J,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

295 n_F=n_F,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

296 )

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

297

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

298 sampler = rbm.hmc_sampler(n_particles=100)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

299

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

300 from pylearn.dataset_ops import image_patches

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

301

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

302 batch_idx = TT.iscalar()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

303 train_batch = image_patches.image_patches(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

304 s_idx = (batch_idx * batchsize + np.arange(batchsize)),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

305 dims = (1000,R,C),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

306 dtype=floatX,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

307 rasterized=True)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

308

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

309 grads = rbm.contrastive_gradient(pos_v=train_batch, neg_v=sampler.positions[0])

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

310

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

311 learn_fn = function([batch_idx, s_lr],

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

312 outputs=[

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

313 grads[0].norm(2),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

314 rbm.U.norm(2)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

315 ],

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

316 updates = sgd_updates(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

317 rbm.params,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

318 grads,

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

319 lr=[2*s_lr, .2*s_lr, .02*s_lr, .1*s_lr, .02*s_lr ]))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

320

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

321 for jj in xrange(10000):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

322 sampler.simulate()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

323 l2_of_Ugrad = learn_fn(jj, lr/max(1, jj/(20*epoch_size/batchsize)))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

324

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

325 if jj > no_l1_epochs * epoch_size/batchsize:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

326 rbm.U.value -= l1_penalty * np.sign(rbm.U.value)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

327 rbm.W.value -= l1_penalty * np.sign(rbm.W.value)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

328

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

329 if jj % 5 == 0:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

330 rbm.U.value = numpy_project_onto_ball(rbm.U.value.T).T

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

331

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

332 if ((jj < 10)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

333 or (jj < 100 and 0==jj%10)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

334 or (jj < 1000 and 0==jj%100)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

335 or (jj < 10000 and 0==jj%1000)):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

336 print 'saving samples', jj, 'epoch', jj/(epoch_size/batchsize), l2_of_Ugrad

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

337 print 'neg particles', sampler.positions[0].value.min(), sampler.positions[0].value.max()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

338 image_tiling.save_tiled_raster_images(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

339 image_tiling.tile_raster_images(sampler.positions[0].value, (R,C)),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

340 "sample_%06i.png"%jj)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

341 image_tiling.save_tiled_raster_images(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

342 image_tiling.tile_raster_images(rbm.U.value.T, (R,C)),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

343 "U_%06i.png"%jj)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

344 image_tiling.save_tiled_raster_images(

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

345 image_tiling.tile_raster_images(rbm.W.value.T, (R,C)),

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

346 "W_%06i.png"%jj)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

347

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

348

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

349

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

350 #

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

351 #

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

352 # Marc'Aurelio Ranzato's code

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

353 #

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

354 ######################################################################

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

355 # compute the value of the free energy at a given input

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

356 # F = - sum log(1+exp(- .5 FH (VF data/norm(data))^2 + bias_cov)) +...

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

357 # - sum log(1+exp(w_mean data + bias_mean)) + ...

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

358 # - bias_vis data + 0.5 data^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

359 # NOTE: FH is constrained to be positive

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

360 # (in the paper the sign is negative but the sign in front of it is also flipped)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

361 def compute_energy_mcRBM(data,normdata,vel,energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,t1,t2,t6,feat,featsq,feat_mean,length,lengthsq,normcoeff,small,num_vis):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

362 # normalize input data vectors

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

363 data.mult(data, target = t6) # DxP (nr input dims x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

364 t6.sum(axis = 0, target = lengthsq) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

365 lengthsq.mult(0.5, target = energy) # energy of quadratic regularization term

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

366 lengthsq.mult(1./num_vis) # normalize by number of components (like std)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

367

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

368 lengthsq.add(small) # small prevents division by 0

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

369 # energy_j = \sum_i 0.5 data_ij ^2

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

370 # lengthsq_j = 1/ (\sum_i data_ij ^2 + small)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

371 cmt.sqrt(lengthsq, target = length)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

372 # length_j = sqrt(lengthsq_j)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

373 length.reciprocal(target = normcoeff) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

374 # normcoef_j = 1/sqrt(lengthsq_j)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

375 data.mult_by_row(normcoeff, target = normdata) # normalized data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

376 # normdata is like data, but cols have unit L2 norm

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

377

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

378 ## potential

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

379 # covariance contribution

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

380 cmt.dot(VF.T, normdata, target = feat) # HxP (nr factors x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

381 feat.mult(feat, target = featsq) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

382

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

383 # featsq is the squared cosines (VF with data)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

384 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

385 t1.mult(-0.5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

386 t1.add_col_vec(bias_cov) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

387 cmt.exp(t1) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

388 t1.add(1, target = t2) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

389 cmt.log(t2)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

390 t2.mult(-1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

391 energy.add_sums(t2, axis=0)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

392 # mean contribution

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

393 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

394 feat_mean.add_col_vec(bias_mean) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

395 cmt.exp(feat_mean)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

396 feat_mean.add(1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

397 cmt.log(feat_mean)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

398 feat_mean.mult(-1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

399 energy.add_sums(feat_mean, axis=0)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

400 # visible bias term

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

401 data.mult_by_col(bias_vis, target = t6)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

402 t6.mult(-1) # DxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

403 energy.add_sums(t6, axis=0) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

404 # kinetic

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

405 vel.mult(vel, target = t6)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

406 energy.add_sums(t6, axis = 0, mult = .5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

407

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

408 ######################################################

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

409 # mcRBM trainer: sweeps over the training set.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

410 # For each batch of samples compute derivatives to update the parameters

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

411 # at the training samples and at the negative samples drawn calling HMC sampler.

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

412 def train_mcRBM():

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

413

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

414 config = ConfigParser()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

415 config.read('input_configuration')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

416

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

417 verbose = config.getint('VERBOSITY','verbose')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

418

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

419 num_epochs = config.getint('MAIN_PARAMETER_SETTING','num_epochs')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

420 batch_size = config.getint('MAIN_PARAMETER_SETTING','batch_size')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

421 startFH = config.getint('MAIN_PARAMETER_SETTING','startFH')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

422 startwd = config.getint('MAIN_PARAMETER_SETTING','startwd')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

423 doPCD = config.getint('MAIN_PARAMETER_SETTING','doPCD')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

424

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

425 # model parameters

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

426 num_fac = config.getint('MODEL_PARAMETER_SETTING','num_fac')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

427 num_hid_cov = config.getint('MODEL_PARAMETER_SETTING','num_hid_cov')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

428 num_hid_mean = config.getint('MODEL_PARAMETER_SETTING','num_hid_mean')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

429 apply_mask = config.getint('MODEL_PARAMETER_SETTING','apply_mask')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

430

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

431 # load data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

432 data_file_name = config.get('DATA','data_file_name')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

433 d = loadmat(data_file_name) # input in the format PxD (P vectorized samples with D dimensions)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

434 totnumcases = d["whitendata"].shape[0]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

435 d = d["whitendata"][0:floor(totnumcases/batch_size)*batch_size,:].copy()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

436 totnumcases = d.shape[0]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

437 num_vis = d.shape[1]

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

438 num_batches = int(totnumcases/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

439 dev_dat = cmt.CUDAMatrix(d.T) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

440

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

441 # training parameters

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

442 epsilon = config.getfloat('OPTIMIZER_PARAMETERS','epsilon')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

443 epsilonVF = 2*epsilon

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

444 epsilonFH = 0.02*epsilon

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

445 epsilonb = 0.02*epsilon

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

446 epsilonw_mean = 0.2*epsilon

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

447 epsilonb_mean = 0.1*epsilon

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

448 weightcost_final = config.getfloat('OPTIMIZER_PARAMETERS','weightcost_final')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

449

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

450 # HMC setting

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

451 hmc_step_nr = config.getint('HMC_PARAMETERS','hmc_step_nr')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

452 hmc_step = 0.01

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

453 hmc_target_ave_rej = config.getfloat('HMC_PARAMETERS','hmc_target_ave_rej')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

454 hmc_ave_rej = hmc_target_ave_rej

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

455

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

456 # initialize weights

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

457 VF = cmt.CUDAMatrix(np.array(0.02 * np.random.randn(num_vis, num_fac), dtype=np.float32, order='F')) # VxH

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

458 if apply_mask == 0:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

459 FH = cmt.CUDAMatrix( np.array( np.eye(num_fac,num_hid_cov), dtype=np.float32, order='F') ) # HxO

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

460 else:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

461 dd = loadmat('your_FHinit_mask_file.mat') # see CVPR2010paper_material/topo2D_3x3_stride2_576filt.mat for an example

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

462 FH = cmt.CUDAMatrix( np.array( dd["FH"], dtype=np.float32, order='F') )

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

463 bias_cov = cmt.CUDAMatrix( np.array(2.0*np.ones((num_hid_cov, 1)), dtype=np.float32, order='F') )

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

464 bias_vis = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F') )

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

465 w_mean = cmt.CUDAMatrix( np.array( 0.05 * np.random.randn(num_vis, num_hid_mean), dtype=np.float32, order='F') ) # VxH

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

466 bias_mean = cmt.CUDAMatrix( np.array( -2.0*np.ones((num_hid_mean,1)), dtype=np.float32, order='F') )

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

467

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

468 # initialize variables to store derivatives

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

469 VFinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_fac)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

470 FHinc = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

471 bias_covinc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_cov, 1)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

472 bias_visinc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, 1)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

473 w_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_vis, num_hid_mean)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

474 bias_meaninc = cmt.CUDAMatrix( np.array(np.zeros((num_hid_mean, 1)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

475

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

476 # initialize temporary storage

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

477 data = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

478 normdata = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

479 negdataini = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

480 feat = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

481 featsq = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

482 negdata = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

483 old_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

484 new_energy = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

485 gradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

486 normgradient = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F')) # VxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

487 thresh = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

488 feat_mean = cmt.CUDAMatrix( np.array(np.empty((num_hid_mean, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

489 vel = cmt.CUDAMatrix( np.array(np.random.randn(num_vis, batch_size), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

490 length = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

491 lengthsq = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

492 normcoeff = cmt.CUDAMatrix( np.array(np.zeros((1, batch_size)), dtype=np.float32, order='F')) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

493 if apply_mask==1: # this used to constrain very large FH matrices only allowing to change values in a neighborhood

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

494 dd = loadmat('your_FHinit_mask_file.mat')

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

495 mask = cmt.CUDAMatrix( np.array(dd["mask"], dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

496 normVF = 1

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

497 small = 0.5

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

498

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

499 # other temporary vars

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

500 t1 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

501 t2 = cmt.CUDAMatrix( np.array(np.empty((num_hid_cov, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

502 t3 = cmt.CUDAMatrix( np.array(np.empty((num_fac, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

503 t4 = cmt.CUDAMatrix( np.array(np.empty((1,batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

504 t5 = cmt.CUDAMatrix( np.array(np.empty((1,1)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

505 t6 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

506 t7 = cmt.CUDAMatrix( np.array(np.empty((num_vis, batch_size)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

507 t8 = cmt.CUDAMatrix( np.array(np.empty((num_vis, num_fac)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

508 t9 = cmt.CUDAMatrix( np.array(np.zeros((num_fac, num_hid_cov)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

509 t10 = cmt.CUDAMatrix( np.array(np.empty((1,num_fac)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

510 t11 = cmt.CUDAMatrix( np.array(np.empty((1,num_hid_cov)), dtype=np.float32, order='F'))

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

511

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

512 # start training

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

513 for epoch in range(num_epochs):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

514

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

515 print "Epoch " + str(epoch + 1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

516

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

517 # anneal learning rates

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

518 epsilonVFc = epsilonVF/max(1,epoch/20)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

519 epsilonFHc = epsilonFH/max(1,epoch/20)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

520 epsilonbc = epsilonb/max(1,epoch/20)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

521 epsilonw_meanc = epsilonw_mean/max(1,epoch/20)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

522 epsilonb_meanc = epsilonb_mean/max(1,epoch/20)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

523 weightcost = weightcost_final

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

524

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

525 if epoch <= startFH:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

526 epsilonFHc = 0

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

527 if epoch <= startwd:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

528 weightcost = 0

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

529

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

530 for batch in range(num_batches):

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

531

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

532 # get current minibatch

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

533 data = dev_dat.slice(batch*batch_size,(batch + 1)*batch_size) # DxP (nr dims x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

534

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

535 # normalize input data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

536 data.mult(data, target = t6) # DxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

537 t6.sum(axis = 0, target = lengthsq) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

538 lengthsq.mult(1./num_vis) # normalize by number of components (like std)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

539 lengthsq.add(small) # small avoids division by 0

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

540 cmt.sqrt(lengthsq, target = length)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

541 length.reciprocal(target = normcoeff) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

542 data.mult_by_row(normcoeff, target = normdata) # normalized data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

543 ## compute positive sample derivatives

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

544 # covariance part

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

545 cmt.dot(VF.T, normdata, target = feat) # HxP (nr facs x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

546 feat.mult(feat, target = featsq) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

547 cmt.dot(FH.T,featsq, target = t1) # OxP (nr cov hiddens x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

548 t1.mult(-0.5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

549 t1.add_col_vec(bias_cov) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

550 t1.apply_sigmoid(target = t2) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

551 cmt.dot(featsq, t2.T, target = FHinc) # HxO

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

552 cmt.dot(FH,t2, target = t3) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

553 t3.mult(feat)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

554 cmt.dot(normdata, t3.T, target = VFinc) # VxH

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

555 t2.sum(axis = 1, target = bias_covinc)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

556 bias_covinc.mult(-1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

557 # visible bias

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

558 data.sum(axis = 1, target = bias_visinc)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

559 bias_visinc.mult(-1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

560 # mean part

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

561 cmt.dot(w_mean.T, data, target = feat_mean) # HxP (nr mean hiddens x nr samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

562 feat_mean.add_col_vec(bias_mean) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

563 feat_mean.apply_sigmoid() # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

564 feat_mean.mult(-1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

565 cmt.dot(data, feat_mean.T, target = w_meaninc)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

566 feat_mean.sum(axis = 1, target = bias_meaninc)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

567

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

568 # HMC sampling: draw an approximate sample from the model

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

569 if doPCD == 0: # CD-1 (set negative data to current training samples)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

570 hmc_step, hmc_ave_rej = draw_HMC_samples(data,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

571 else: # PCD-1 (use previous negative data as starting point for chain)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

572 negdataini.assign(negdata)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

573 hmc_step, hmc_ave_rej = draw_HMC_samples(negdataini,negdata,normdata,vel,gradient,normgradient,new_energy,old_energy,VF,FH,bias_cov,bias_vis,w_mean,bias_mean,hmc_step,hmc_step_nr,hmc_ave_rej,hmc_target_ave_rej,t1,t2,t3,t4,t5,t6,t7,thresh,feat,featsq,batch_size,feat_mean,length,lengthsq,normcoeff,small,num_vis)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

574

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

575 # compute derivatives at the negative samples

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

576 # normalize input data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

577 negdata.mult(negdata, target = t6) # DxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

578 t6.sum(axis = 0, target = lengthsq) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

579 lengthsq.mult(1./num_vis) # normalize by number of components (like std)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

580 lengthsq.add(small)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

581 cmt.sqrt(lengthsq, target = length)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

582 length.reciprocal(target = normcoeff) # 1xP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

583 negdata.mult_by_row(normcoeff, target = normdata) # normalized data

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

584 # covariance part

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

585 cmt.dot(VF.T, normdata, target = feat) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

586 feat.mult(feat, target = featsq) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

587 cmt.dot(FH.T,featsq, target = t1) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

588 t1.mult(-0.5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

589 t1.add_col_vec(bias_cov) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

590 t1.apply_sigmoid(target = t2) # OxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

591 FHinc.subtract_dot(featsq, t2.T) # HxO

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

592 FHinc.mult(0.5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

593 cmt.dot(FH,t2, target = t3) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

594 t3.mult(feat)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

595 VFinc.subtract_dot(normdata, t3.T) # VxH

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

596 bias_covinc.add_sums(t2, axis = 1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

597 # visible bias

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

598 bias_visinc.add_sums(negdata, axis = 1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

599 # mean part

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

600 cmt.dot(w_mean.T, negdata, target = feat_mean) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

601 feat_mean.add_col_vec(bias_mean) # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

602 feat_mean.apply_sigmoid() # HxP

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

603 w_meaninc.add_dot(negdata, feat_mean.T)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

604 bias_meaninc.add_sums(feat_mean, axis = 1)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

605

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

606 # update parameters

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

607 VFinc.add_mult(VF.sign(), weightcost) # L1 regularization

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

608 VF.add_mult(VFinc, -epsilonVFc/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

609 # normalize columns of VF: normalize by running average of their norm

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

610 VF.mult(VF, target = t8)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

611 t8.sum(axis = 0, target = t10)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

612 cmt.sqrt(t10)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

613 t10.sum(axis=1,target = t5)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

614 t5.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

615 normVF = .95*normVF + (.05/num_fac) * t5.numpy_array[0,0] # estimate norm

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

616 t10.reciprocal()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

617 VF.mult_by_row(t10)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

618 VF.mult(normVF)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

619 bias_cov.add_mult(bias_covinc, -epsilonbc/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

620 bias_vis.add_mult(bias_visinc, -epsilonbc/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

621

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

622 if epoch > startFH:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

623 FHinc.add_mult(FH.sign(), weightcost) # L1 regularization

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

624 FH.add_mult(FHinc, -epsilonFHc/batch_size) # update

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

625 # set to 0 negative entries in FH

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

626 FH.greater_than(0, target = t9)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

627 FH.mult(t9)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

628 if apply_mask==1:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

629 FH.mult(mask)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

630 # normalize columns of FH: L1 norm set to 1 in each column

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

631 FH.sum(axis = 0, target = t11)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

632 t11.reciprocal()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

633 FH.mult_by_row(t11)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

634 w_meaninc.add_mult(w_mean.sign(),weightcost)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

635 w_mean.add_mult(w_meaninc, -epsilonw_meanc/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

636 bias_mean.add_mult(bias_meaninc, -epsilonb_meanc/batch_size)

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

637

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

638 if verbose == 1:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

639 print "VF: " + '%3.2e' % VF.euclid_norm() + ", DVF: " + '%3.2e' % (VFinc.euclid_norm()*(epsilonVFc/batch_size)) + ", FH: " + '%3.2e' % FH.euclid_norm() + ", DFH: " + '%3.2e' % (FHinc.euclid_norm()*(epsilonFHc/batch_size)) + ", bias_cov: " + '%3.2e' % bias_cov.euclid_norm() + ", Dbias_cov: " + '%3.2e' % (bias_covinc.euclid_norm()*(epsilonbc/batch_size)) + ", bias_vis: " + '%3.2e' % bias_vis.euclid_norm() + ", Dbias_vis: " + '%3.2e' % (bias_visinc.euclid_norm()*(epsilonbc/batch_size)) + ", wm: " + '%3.2e' % w_mean.euclid_norm() + ", Dwm: " + '%3.2e' % (w_meaninc.euclid_norm()*(epsilonw_meanc/batch_size)) + ", bm: " + '%3.2e' % bias_mean.euclid_norm() + ", Dbm: " + '%3.2e' % (bias_meaninc.euclid_norm()*(epsilonb_meanc/batch_size)) + ", step: " + '%3.2e' % hmc_step + ", rej: " + '%3.2e' % hmc_ave_rej

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

640 sys.stdout.flush()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

641 # back-up every once in a while

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

642 if np.mod(epoch,10) == 0:

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

643 VF.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

644 FH.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

645 bias_cov.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

646 w_mean.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

647 bias_mean.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

648 bias_vis.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

649 savemat("ws_temp", {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array,'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

650 # final back-up

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

651 VF.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

652 FH.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

653 bias_cov.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

654 bias_vis.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

655 w_mean.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

656 bias_mean.copy_to_host()

90e11d5d0a41 adding algorithms/mcRBM, but it is not done yet

James Bergstra <bergstrj@iro.umontreal.ca>

parents:

diff changeset

657 savemat("ws_fac" + str(num_fac) + "_cov" + str(num_hid_cov) + "_mean" + str(num_hid_mean), {'VF':VF.numpy_array,'FH':FH.numpy_array,'bias_cov': bias_cov.numpy_array, 'bias_vis': bias_vis.numpy_array, 'w_mean': w_mean.numpy_array, 'bias_mean': bias_mean.numpy_array, 'epoch':epoch})

Mercurial > pylearn

annotate pylearn/algorithms/mcRBM.py @ 967:90e11d5d0a41