pylearn: kernel_regression.py annotate

annotate kernel_regression.py @ 422:32c5f87bc54e

Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None

author	Yoshua Bengio <bengioy@iro.umontreal.ca>
date	Sat, 19 Jul 2008 14:12:41 -0400
parents	e01f17be270a
children	e2b46a8f2b7b

rev	line source
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	1 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	2 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	3 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	4
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	5 from pylearn.learner import OfflineLearningAlgorithm
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	6 from theano import tensor as T
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	7 from nnet_ops import prepend_1_to_each_row
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	8 from theano.scalar import as_scalar
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	9 from common.autoname import AutoName
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	10 import theano
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	11 import numpy
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	12
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	13 class KernelRegression(OfflineLearningAlgorithm):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	14 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	15 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	16 * the data are n (x_t,y_t) pairs and we want to estimate E[y\|x]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	17 * the predictor computes
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	18 f(x) = b + \sum_{t=1}^n \alpha_t K(x,x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	19 with free parameters b and alpha, training inputs x_t,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	20 and kernel function K (gaussian by default).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	21 Clearly, each prediction involves O(n) computations.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	22 * the learner chooses b and alpha to minimize
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	23 lambda alpha' G' G alpha + \sum_{t=1}^n (f(x_t)-y_t)^2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	24 where G is the matrix with entries G_ij = K(x_i,x_j).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	25 The first (L2 regularization) term is the squared L2
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	26 norm of the primal weights w = \sum_t \alpha_t phi(x_t)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	27 where phi is the function s.t. K(u,v)=phi(u).phi(v).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	28 * this involves solving a linear system with (n+1,n+1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	29 matrix, which is an O(n^3) computation. In addition,
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	30 that linear system matrix requires O(n^2) memory.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	31 So this learning algorithm should be used only for
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	32 small datasets.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	33 * the linear system is
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	34 (M + lambda I_n) theta = (1, y)'
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	35 where theta = (b, alpha), I_n is the (n+1)x(n+1) matrix that is the identity
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	36 except with a 0 at (0,0), M is the matrix with G in the sub-matrix starting
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	37 at (1,1), 1's in column 0, except for a value of n at (0,0), and sum_i G_{i,j}
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	38 in the rest of row 0.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	39
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	40 Note that this is gives an estimate of E[y\|x,training_set] that is the
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	41 same as obtained with a Gaussian process regression. The GP
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	42 regression would also provide a Bayesian Var[y\|x,training_set].
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	43 It corresponds to an assumption that f is a random variable
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	44 with Gaussian (process) prior distribution with covariance
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	45 function K. Because we assume Gaussian noise we obtain a Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	46 posterior for f (whose mean is computed here).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	47
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	48
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	49 Usage:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	50
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	51 kernel_regressor=KernelRegression(L2_regularizer=0.1,gamma=0.5) (kernel=GaussianKernel(gamma=0.5))
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	52 kernel_predictor=kernel_regressor(training_set)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	53 all_results_dataset=kernel_predictor(test_set) # creates a dataset with "output" and "squared_error" field
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	54 outputs = kernel_predictor.compute_outputs(inputs) # inputs and outputs are numpy arrays
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	55 outputs, errors = kernel_predictor.compute_outputs_and_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	56 errors = kernel_predictor.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	57 mse = kernel_predictor.compute_mse(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	58
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	59
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	60
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	61 The training_set must have fields "input" and "target".
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	62 The test_set must have field "input", and needs "target" if
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	63 we want to compute the squared errors.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	64
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	65 The predictor parameters are obtained analytically from the training set.
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	66 Training is only done on a whole training set rather than on minibatches
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	67 (no online implementation).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	68
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	69 The dataset fields expected and produced by the learning algorithm and the trained model
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	70 are the following:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	71
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	72 - Input and output dataset fields (example-wise quantities):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	73
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	74 - 'input' (always expected as an input_dataset field)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	75 - 'target' (always expected by the learning algorithm, optional for learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	76 - 'output' (always produced by learned model)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	77 - 'squared_error' (optionally produced by learned model if 'target' is provided)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	78 = example-wise squared error
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	79 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	80 def __init__(self, kernel=None, L2_regularizer=0, gamma=1):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	81 self.kernel = kernel
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	82 self.L2_regularizer=L2_regularizer
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	83 self.gamma = gamma # until we fix things, the kernel type is fixed, Gaussian
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	84 self.equations = KernelRegressionEquations()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	85
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	86 def __call__(self,trainset):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	87 n_examples = len(trainset)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	88 first_example = trainset[0]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	89 n_inputs = first_example['input'].size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	90 n_outputs = first_example['target'].size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	91 M = numpy.zeros((n_examples+1,n_examples+1))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	92 Y = numpy.zeros((n_examples+1,n_outputs))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	93 for i in xrange(n_inputs):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	94 M[i+1,i+1]=self.L2_regularizer
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	95 data = trainset.fields()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	96 train_inputs = numpy.array(data['input'])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	97 Y[0]=1
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	98 Y[1:,:] = numpy.array(data['target'])
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	99 train_inputs_square,sumG=self.equations.compute_system_matrix(train_inputs,M)
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	100 M[0,1:] = sumG
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	101 M[1:,0] = 1
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	102 M[0,0] = M.shape[0]
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	103 print M
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	104 theta=numpy.linalg.solve(M,Y)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	105 return KernelPredictor(theta,self.gamma, train_inputs, train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	106
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	107 class KernelPredictorEquations(AutoName):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	108 train_inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	109 train_inputs_square = T.vector() # n_examples
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	110 inputs = T.matrix() # minibatchsize x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	111 targets = T.matrix() # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	112 theta = T.matrix() # (n_examples+1) x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	113 gamma = T.scalar()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	114 inv_gamma2 = 1./(gamma*gamma)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	115 b = theta[0]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	116 alpha = theta[1:,:]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	117 inputs_square = T.sum(inputs*inputs,axis=1)
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	118 Kx = T.exp(-(train_inputs_square-2T.dot(inputs,train_inputs.T)+inputs_square)inv_gamma2)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	119 outputs = T.dot(Kx,alpha) + b # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	120 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	121
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	122 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	123 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	124 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	125 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	126 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	127 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	128 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	129
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	130 cls.compute_outputs = fn([cls.inputs,cls.theta,cls.gamma,cls.train_inputs,cls.train_inputs_square],[cls.outputs])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	131 cls.compute_errors = fn([cls.outputs,cls.targets],[cls.squared_errors])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	132
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	133 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	134
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	135 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	136 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	137
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	138 class KernelRegressionEquations(KernelPredictorEquations):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	139 M = T.matrix() # (n_examples+1) x (n_examples+1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	140 inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	141 G = M[1:,1:]
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	142 new_G = T.gemm(G,1.,inputs,inputs.T,1.)
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	143 sumG = T.sum(new_G,axis=0)
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	144 inputs_square = T.sum(inputs*inputs,axis=1)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	145
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	146 __compiled = False
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	147
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	148 @classmethod
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	149 def compile(cls,linker='c\|py'):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	150 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	151 return
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	152 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	153 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	154
422 32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 421 diff changeset	155 cls.compute_system_matrix = fn([cls.inputs,cls.M],[cls.inputs_square,cls.sumG])
421 e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	156
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	157 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	158
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	159 def __init__(self):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	160 self.compile()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	161
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	162 class KernelPredictor(object):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	163 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	164 A kernel predictor has parameters theta (a bias vector and a weight matrix alpha)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	165 it can use to make a non-linear prediction (according to the KernelPredictorEquations).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	166 It can compute its output (bias + alpha * kernel(train_inputs,input) and a squared error (\|\|output - target\|\|^2).
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	167 """
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	168 def __init__(self, theta, gamma, train_inputs, train_inputs_square):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	169 self.theta=theta
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	170 self.gamma=gamma
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	171 self.train_inputs=train_inputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	172 self.train_inputs_square=train_inputs_square
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	173 self.equations = LinearPredictorEquations()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	174
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	175 def compute_outputs(self,inputs):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	176 return self.equations.compute_outputs(inputs,self.theta,self.gamma,self.train_inputs,self.train_inputs_square)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	177 def compute_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	178 return self.equations.compute_errors(self.compute_outputs(inputs),targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	179 def compute_outputs_and_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	180 outputs = self.compute_outputs(inputs)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	181 return [outputs,self.equations.compute_errors(outputs,targets)]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	182 def compute_mse(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	183 errors = self.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	184 return numpy.sum(errors)/errors.size
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	185
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	186 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	187 assert dataset.hasFields(["input"])
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	188 if output_fieldnames is None:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	189 if dataset.hasFields(["target"]):
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	190 output_fieldnames = ["output","squared_error"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	191 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	192 output_fieldnames = ["output"]
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	193 output_fieldnames.sort()
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	194 if output_fieldnames == ["squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	195 f = self.compute_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	196 elif output_fieldnames == ["output"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	197 f = self.compute_outputs
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	198 elif output_fieldnames == ["output","squared_error"]:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	199 f = self.compute_outputs_and_errors
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	200 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	201 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	202
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	203 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	204 if cached_output_dataset:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	205 return CachedDataSet(ds)
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	206 else:
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	207 return ds
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	208
e01f17be270a Kernel regression learning algorithm Yoshua Bengio <bengioy@iro.umontreal.ca> parents: diff changeset	209

Mercurial > pylearn

annotate kernel_regression.py @ 422:32c5f87bc54e