pylearn: kernel_regression.py annotate

annotate kernel_regression.py @ 425:e2b46a8f2b7b

Debugging kernel regression

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Sat, 19 Jul 2008 17:57:46 -0400
parents	32c5f87bc54e
children	d7611a3811f2

rev	line source
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	4
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	5 from pylearn.learner import OfflineLearningAlgorithm
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 from theano import tensor as T
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 from nnet_ops import prepend_1_to_each_row
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 from theano.scalar import as_scalar
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	9 from common.autoname import AutoName
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10 import theano
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	11 import numpy
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	12
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	13 # map a N-vector to a 1xN matrix
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	14 row_vector = theano.elemwise.DimShuffle((False,),['x',0])
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	15 # map a N-vector to a Nx1 matrix
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	16 col_vector = theano.elemwise.DimShuffle((False,),[0,'x'])
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	17
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 class KernelRegression(OfflineLearningAlgorithm):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	21 * the data are n (x_t,y_t) pairs and we want to estimate E[y\|x]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 * the predictor computes
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	23 f(x) = b + \sum_{t=1}^n \alpha_t K(x,x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 with free parameters b and alpha, training inputs x_t,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	25 and kernel function K (gaussian by default).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26 Clearly, each prediction involves O(n) computations.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 * the learner chooses b and alpha to minimize
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	28 lambda alpha' G' G alpha + \sum_{t=1}^n (f(x_t)-y_t)^2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29 where G is the matrix with entries G_ij = K(x_i,x_j).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 The first (L2 regularization) term is the squared L2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	31 norm of the primal weights w = \sum_t \alpha_t phi(x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 where phi is the function s.t. K(u,v)=phi(u).phi(v).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 * this involves solving a linear system with (n+1,n+1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 matrix, which is an O(n^3) computation. In addition,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 that linear system matrix requires O(n^2) memory.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	36 So this learning algorithm should be used only for
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 small datasets.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 * the linear system is
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	39 (M + lambda I_n) theta = (1, y)'
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	40 where theta = (b, alpha), I_n is the (n+1)x(n+1) matrix that is the identity
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	41 except with a 0 at (0,0), M is the matrix with G in the sub-matrix starting
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 at (1,1), 1's in column 0, except for a value of n at (0,0), and sum_i G_{i,j}
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 in the rest of row 0.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 Note that this is gives an estimate of E[y\|x,training_set] that is the
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46 same as obtained with a Gaussian process regression. The GP
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47 regression would also provide a Bayesian Var[y\|x,training_set].
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48 It corresponds to an assumption that f is a random variable
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49 with Gaussian (process) prior distribution with covariance
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50 function K. Because we assume Gaussian noise we obtain a Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	51 posterior for f (whose mean is computed here).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 Usage:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	56 kernel_regressor=KernelRegression(L2_regularizer=0.1,gamma=0.5) (kernel=GaussianKernel(gamma=0.5))
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 kernel_predictor=kernel_regressor(training_set)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	58 all_results_dataset=kernel_predictor(test_set) # creates a dataset with "output" and "squared_error" field
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	59 outputs = kernel_predictor.compute_outputs(inputs) # inputs and outputs are numpy arrays
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60 outputs, errors = kernel_predictor.compute_outputs_and_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 errors = kernel_predictor.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 mse = kernel_predictor.compute_mse(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 The training_set must have fields "input" and "target".
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 The test_set must have field "input", and needs "target" if
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68 we want to compute the squared errors.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 The predictor parameters are obtained analytically from the training set.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71 Training is only done on a whole training set rather than on minibatches
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 (no online implementation).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	74 The dataset fields expected and produced by the learning algorithm and the trained model
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 are the following:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	76
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 - Input and output dataset fields (example-wise quantities):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 - 'input' (always expected as an input_dataset field)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 - 'target' (always expected by the learning algorithm, optional for learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	81 - 'output' (always produced by learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 - 'squared_error' (optionally produced by learned model if 'target' is provided)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	83 = example-wise squared error
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85 def __init__(self, kernel=None, L2_regularizer=0, gamma=1):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	86 self.kernel = kernel
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	87 self.L2_regularizer=L2_regularizer
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	88 self.gamma = gamma # until we fix things, the kernel type is fixed, Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	89 self.equations = KernelRegressionEquations()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	90
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	91 def __call__(self,trainset):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	92 n_examples = len(trainset)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	93 first_example = trainset[0]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	94 n_inputs = first_example['input'].size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	95 n_outputs = first_example['target'].size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	96 M = numpy.zeros((n_examples+1,n_examples+1))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	97 Y = numpy.zeros((n_examples+1,n_outputs))
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	98 for i in xrange(n_examples):
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	99 M[i+1,i+1]=self.L2_regularizer
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	100 data = trainset.fields()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	101 train_inputs = numpy.array(data['input'])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	102 Y[0]=1
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	103 Y[1:,:] = numpy.array(data['target'])
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	104 train_inputs_square,sumG,G=self.equations.compute_system_matrix(train_inputs,self.gamma)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	105 M[1:,1:] += G
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	106 M[0,1:] = sumG
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	107 M[1:,0] = 1
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	108 M[0,0] = M.shape[0]
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	109 self.M=M
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	110 self.Y=Y
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	111 theta=numpy.linalg.solve(M,Y)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	112 return KernelPredictor(theta,self.gamma, train_inputs, train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	113
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	114 class KernelPredictorEquations(AutoName):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	115 train_inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116 train_inputs_square = T.vector() # n_examples
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	117 inputs = T.matrix() # minibatchsize x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	118 targets = T.matrix() # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119 theta = T.matrix() # (n_examples+1) x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	120 gamma = T.scalar()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	121 inv_gamma2 = 1./(gamma*gamma)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	122 b = theta[0]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	123 alpha = theta[1:,:]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	124 inputs_square = T.sum(inputs*inputs,axis=1)
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	125 Kx = T.exp(-(row_vector(train_inputs_square)-2T.dot(inputs,train_inputs.T)+col_vector(inputs_square))inv_gamma2)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	126 outputs = T.dot(Kx,alpha) + b # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	128
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	129 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	130 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	131 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	133 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	134 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	135 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137 cls.compute_outputs = fn([cls.inputs,cls.theta,cls.gamma,cls.train_inputs,cls.train_inputs_square],[cls.outputs])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	138 cls.compute_errors = fn([cls.outputs,cls.targets],[cls.squared_errors])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	139
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	141
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	142 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	143 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	145 class KernelRegressionEquations(KernelPredictorEquations):
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	146 #M = T.matrix() # (n_examples+1) x (n_examples+1)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	147 inputs = T.matrix() # n_examples x n_inputs
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	148 gamma = T.scalar()
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	149 inv_gamma2 = 1./(gamma*gamma)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150 inputs_square = T.sum(inputs*inputs,axis=1)
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	151 #new_G = G+T.dot(inputs,inputs.T)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	152 #new_G = T.gemm(G,1.,inputs,inputs.T,1.)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	153 G = T.exp(-(row_vector(inputs_square)-2T.dot(inputs,inputs.T)+col_vector(inputs_square))inv_gamma2)
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	154 sumG = T.sum(G,axis=0)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	155
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	157
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	158 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	159 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	160 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	161 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	162 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	163 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	165 cls.compute_system_matrix = fn([cls.inputs,cls.gamma],[cls.inputs_square,cls.sumG,cls.G])
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	166
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	167 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	169 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	170 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	171
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	172 class KernelPredictor(object):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	173 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	174 A kernel predictor has parameters theta (a bias vector and a weight matrix alpha)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	175 it can use to make a non-linear prediction (according to the KernelPredictorEquations).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	176 It can compute its output (bias + alpha * kernel(train_inputs,input) and a squared error (\|\|output - target\|\|^2).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	177 """
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	178 def __init__(self, theta, gamma, train_inputs, train_inputs_square=None):
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	179 self.theta=theta
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 self.gamma=gamma
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181 self.train_inputs=train_inputs
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	182 if train_inputs_square==None:
e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	183 train_inputs_square = numpy.sum(train_inputs*train_inputs,axis=1)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	184 self.train_inputs_square=train_inputs_square
425 e2b46a8f2b7b Debugging kernel regression Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 422 diff changeset	185 self.equations = KernelPredictorEquations()
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	186
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	187 def compute_outputs(self,inputs):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	188 return self.equations.compute_outputs(inputs,self.theta,self.gamma,self.train_inputs,self.train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	189 def compute_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	190 return self.equations.compute_errors(self.compute_outputs(inputs),targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	191 def compute_outputs_and_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	192 outputs = self.compute_outputs(inputs)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	193 return [outputs,self.equations.compute_errors(outputs,targets)]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	194 def compute_mse(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	195 errors = self.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	196 return numpy.sum(errors)/errors.size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	197
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	198 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	199 assert dataset.hasFields(["input"])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	200 if output_fieldnames is None:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	201 if dataset.hasFields(["target"]):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	202 output_fieldnames = ["output","squared_error"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	203 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	204 output_fieldnames = ["output"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	205 output_fieldnames.sort()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	206 if output_fieldnames == ["squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207 f = self.compute_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	208 elif output_fieldnames == ["output"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209 f = self.compute_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	210 elif output_fieldnames == ["output","squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	211 f = self.compute_outputs_and_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	212 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	213 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	214
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	215 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	216 if cached_output_dataset:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	217 return CachedDataSet(ds)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	218 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	219 return ds
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	220
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	221

Mercurial > pylearn

annotate kernel_regression.py @ 425:e2b46a8f2b7b