pylearn: kernel_regression.py annotate

annotate kernel_regression.py @ 426:d7611a3811f2

Moved incomplete stuff to sandbox

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Tue, 22 Jul 2008 15:20:25 -0400
parents	e2b46a8f2b7b
children	0f8c81b0776d

rev	line source
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	4
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	5 from pylearn.learner import OfflineLearningAlgorithm
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 from theano import tensor as T
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 from nnet_ops import prepend_1_to_each_row
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 from theano.scalar import as_scalar
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	9 from common.autoname import AutoName
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10 import theano
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	11 import numpy
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	12
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	13 # map a N-vector to a 1xN matrix
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	14 row_vector = theano.elemwise.DimShuffle((False,),['x',0])
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	15 # map a N-vector to a Nx1 matrix
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	16 col_vector = theano.elemwise.DimShuffle((False,),[0,'x'])
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	17
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 class KernelRegression(OfflineLearningAlgorithm):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	21 * the data are n (x_t,y_t) pairs and we want to estimate E[y\|x]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 * the predictor computes
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	23 f(x) = b + \sum_{t=1}^n \alpha_t K(x,x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 with free parameters b and alpha, training inputs x_t,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	25 and kernel function K (gaussian by default).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26 Clearly, each prediction involves O(n) computations.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 * the learner chooses b and alpha to minimize
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	28 lambda alpha' G' G alpha + \sum_{t=1}^n (f(x_t)-y_t)^2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29 where G is the matrix with entries G_ij = K(x_i,x_j).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 The first (L2 regularization) term is the squared L2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	31 norm of the primal weights w = \sum_t \alpha_t phi(x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 where phi is the function s.t. K(u,v)=phi(u).phi(v).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 * this involves solving a linear system with (n+1,n+1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 matrix, which is an O(n^3) computation. In addition,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 that linear system matrix requires O(n^2) memory.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	36 So this learning algorithm should be used only for
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 small datasets.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 * the linear system is
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	39 (M + lambda I_n) theta = (1, y)'
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	40 where theta = (b, alpha), I_n is the (n+1)x(n+1) matrix that is the identity
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	41 except with a 0 at (0,0), M is the matrix with G in the sub-matrix starting
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 at (1,1), 1's in column 0, except for a value of n at (0,0), and sum_i G_{i,j}
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 in the rest of row 0.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 Note that this is gives an estimate of E[y\|x,training_set] that is the
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46 same as obtained with a Gaussian process regression. The GP
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47 regression would also provide a Bayesian Var[y\|x,training_set].
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 It corresponds to an assumption that f is a random variable
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49 with Gaussian (process) prior distribution with covariance
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 function K. Because we assume Gaussian noise we obtain a Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 posterior for f (whose mean is computed here).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 Usage:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	56 kernel_regressor=KernelRegression(L2_regularizer=0.1,gamma=0.5) (kernel=GaussianKernel(gamma=0.5))
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 kernel_predictor=kernel_regressor(training_set)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	58 all_results_dataset=kernel_predictor(test_set) # creates a dataset with "output" and "squared_error" field
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	59 outputs = kernel_predictor.compute_outputs(inputs) # inputs and outputs are numpy arrays
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60 outputs, errors = kernel_predictor.compute_outputs_and_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 errors = kernel_predictor.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 mse = kernel_predictor.compute_mse(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 The training_set must have fields "input" and "target".
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 The test_set must have field "input", and needs "target" if
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 we want to compute the squared errors.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 The predictor parameters are obtained analytically from the training set.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 Training is only done on a whole training set rather than on minibatches
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 (no online implementation).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	74 The dataset fields expected and produced by the learning algorithm and the trained model
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 are the following:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	76
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 - Input and output dataset fields (example-wise quantities):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 - 'input' (always expected as an input_dataset field)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 - 'target' (always expected by the learning algorithm, optional for learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	81 - 'output' (always produced by learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 - 'squared_error' (optionally produced by learned model if 'target' is provided)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	83 = example-wise squared error
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 """
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	85 def __init__(self, kernel=None, L2_regularizer=0, gamma=1, use_bias=False):
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	86 # THE VERSION WITH BIAS DOES NOT SEEM RIGHT
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	87 self.kernel = kernel
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	88 self.L2_regularizer=L2_regularizer
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	89 self.use_bias=use_bias
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	90 self.gamma = gamma # until we fix things, the kernel type is fixed, Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	91 self.equations = KernelRegressionEquations()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	92
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	93 def __call__(self,trainset):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	94 n_examples = len(trainset)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	95 first_example = trainset[0]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	96 n_inputs = first_example['input'].size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	97 n_outputs = first_example['target'].size
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	98 b1=1 if self.use_bias else 0
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	99 M = numpy.zeros((n_examples+b1,n_examples+b1))
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	100 Y = numpy.zeros((n_examples+b1,n_outputs))
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	101 for i in xrange(n_examples):
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	102 M[i+b1,i+b1]=self.L2_regularizer
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	103 data = trainset.fields()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	104 train_inputs = numpy.array(data['input'])
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	105 if self.use_bias:
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	106 Y[0]=1
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	107 Y[b1:,:] = numpy.array(data['target'])
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	108 train_inputs_square,sumG,G=self.equations.compute_system_matrix(train_inputs,self.gamma)
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	109 M[b1:,b1:] += G
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	110 if self.use_bias:
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	111 M[0,1:] = sumG
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	112 M[1:,0] = 1
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	113 M[0,0] = M.shape[0]
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	114 self.M=M
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	115 self.Y=Y
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116 theta=numpy.linalg.solve(M,Y)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	117 return KernelPredictor(theta,self.gamma, train_inputs, train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	118
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119 class KernelPredictorEquations(AutoName):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	120 train_inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	121 train_inputs_square = T.vector() # n_examples
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	122 inputs = T.matrix() # minibatchsize x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	123 targets = T.matrix() # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	124 theta = T.matrix() # (n_examples+1) x n_outputs
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	125 b1 = T.shape(train_inputs_square)[0]<T.shape(theta)[0]
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	126 gamma = T.scalar()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127 inv_gamma2 = 1./(gamma*gamma)
426 d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	128 b = b1*theta[0]
d7611a3811f2 Moved incomplete stuff to sandbox Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 425 diff changeset	129 alpha = theta[b1:,:]
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	130 inputs_square = T.sum(inputs*inputs,axis=1)
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	131 Kx = T.exp(-(row_vector(train_inputs_square)-2T.dot(inputs,train_inputs.T)+col_vector(inputs_square))inv_gamma2)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132 outputs = T.dot(Kx,alpha) + b # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	133 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	134
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	135 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	138 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	139 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	141 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	142
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	143 cls.compute_outputs = fn([cls.inputs,cls.theta,cls.gamma,cls.train_inputs,cls.train_inputs_square],[cls.outputs])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144 cls.compute_errors = fn([cls.outputs,cls.targets],[cls.squared_errors])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	145
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	146 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	147
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	148 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	149 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	151 class KernelRegressionEquations(KernelPredictorEquations):
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	152 #M = T.matrix() # (n_examples+1) x (n_examples+1)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	153 inputs = T.matrix() # n_examples x n_inputs
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	154 gamma = T.scalar()
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	155 inv_gamma2 = 1./(gamma*gamma)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156 inputs_square = T.sum(inputs*inputs,axis=1)
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	157 #new_G = G+T.dot(inputs,inputs.T)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	158 #new_G = T.gemm(G,1.,inputs,inputs.T,1.)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	159 G = T.exp(-(row_vector(inputs_square)-2T.dot(inputs,inputs.T)+col_vector(inputs_square))inv_gamma2)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	160 sumG = T.sum(G,axis=0)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	161
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	162 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	163
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	165 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	166 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	167 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	169 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	170
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	171 cls.compute_system_matrix = fn([cls.inputs,cls.gamma],[cls.inputs_square,cls.sumG,cls.G])
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	172
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	173 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	174
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	175 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	176 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	177
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	178 class KernelPredictor(object):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	179 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 A kernel predictor has parameters theta (a bias vector and a weight matrix alpha)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181 it can use to make a non-linear prediction (according to the KernelPredictorEquations).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	182 It can compute its output (bias + alpha * kernel(train_inputs,input) and a squared error (\|\|output - target\|\|^2).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	183 """
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	184 def __init__(self, theta, gamma, train_inputs, train_inputs_square=None):
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	185 self.theta=theta
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	186 self.gamma=gamma
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	187 self.train_inputs=train_inputs
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	188 if train_inputs_square==None:
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	189 train_inputs_square = numpy.sum(train_inputs*train_inputs,axis=1)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	190 self.train_inputs_square=train_inputs_square
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	191 self.equations = KernelPredictorEquations()
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	192
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	193 def compute_outputs(self,inputs):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	194 return self.equations.compute_outputs(inputs,self.theta,self.gamma,self.train_inputs,self.train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	195 def compute_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	196 return self.equations.compute_errors(self.compute_outputs(inputs),targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	197 def compute_outputs_and_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	198 outputs = self.compute_outputs(inputs)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	199 return [outputs,self.equations.compute_errors(outputs,targets)]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	200 def compute_mse(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	201 errors = self.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	202 return numpy.sum(errors)/errors.size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	203
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	204 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	205 assert dataset.hasFields(["input"])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	206 if output_fieldnames is None:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207 if dataset.hasFields(["target"]):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	208 output_fieldnames = ["output","squared_error"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	210 output_fieldnames = ["output"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	211 output_fieldnames.sort()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	212 if output_fieldnames == ["squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	213 f = self.compute_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	214 elif output_fieldnames == ["output"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	215 f = self.compute_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	216 elif output_fieldnames == ["output","squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	217 f = self.compute_outputs_and_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	218 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	219 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	220
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	221 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	222 if cached_output_dataset:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	223 return CachedDataSet(ds)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	224 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	225 return ds
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	226
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	227

Mercurial > pylearn

annotate kernel_regression.py @ 426:d7611a3811f2