Mercurial > pylearn

annotate kernel_regression.py @ 421:e01f17be270a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Kernel regression learning algorithm
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Sat, 19 Jul 2008 10:11:22 -0400
parents
children 32c5f87bc54e
rev   line source
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 from pylearn.learner import OfflineLearningAlgorithm
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 from theano import tensor as T
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 from nnet_ops import prepend_1_to_each_row
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 from theano.scalar import as_scalar
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9 from common.autoname import AutoName
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10 import theano
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 import numpy
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
13 class KernelRegression(OfflineLearningAlgorithm):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
14 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
15 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
16 * the data are n (x_t,y_t) pairs and we want to estimate E[y|x]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
17 * the predictor computes
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18 f(x) = b + \sum_{t=1}^n \alpha_t K(x,x_t)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 with free parameters b and alpha, training inputs x_t,
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 and kernel function K (gaussian by default).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21 Clearly, each prediction involves O(n) computations.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 * the learner chooses b and alpha to minimize
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 lambda alpha' G' G alpha + \sum_{t=1}^n (f(x_t)-y_t)^2
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 where G is the matrix with entries G_ij = K(x_i,x_j).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 The first (L2 regularization) term is the squared L2
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 norm of the primal weights w = \sum_t \alpha_t phi(x_t)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 where phi is the function s.t. K(u,v)=phi(u).phi(v).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 * this involves solving a linear system with (n+1,n+1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29 matrix, which is an O(n^3) computation. In addition,
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 that linear system matrix requires O(n^2) memory.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31 So this learning algorithm should be used only for
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 small datasets.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 * the linear system is
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 (M + lambda I_n) theta = (1, y)'
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 where theta = (b, alpha), I_n is the (n+1)x(n+1) matrix that is the identity
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 except with a 0 at (0,0), M is the matrix with G in the sub-matrix starting
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 at (1,1), 1's in column 0, except for a value of n at (0,0), and sum_i G_{i,j}
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 in the rest of row 0.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 Note that this is gives an estimate of E[y|x,training_set] that is the
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 same as obtained with a Gaussian process regression. The GP
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 regression would also provide a Bayesian Var[y|x,training_set].
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 It corresponds to an assumption that f is a random variable
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44 with Gaussian (process) prior distribution with covariance
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 function K. Because we assume Gaussian noise we obtain a Gaussian
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46 posterior for f (whose mean is computed here).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 Usage:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 kernel_regressor=KernelRegression(L2_regularizer=0.1,kernel=GaussianKernel(gamma=0.5))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52 kernel_predictor=kernel_regressor(training_set)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53 all_results_dataset=kernel_predictor(test_set) # creates a dataset with "output" and "squared_error" field
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 outputs = kernel_predictor.compute_outputs(inputs) # inputs and outputs are numpy arrays
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55 outputs, errors = kernel_predictor.compute_outputs_and_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
56 errors = kernel_predictor.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
57 mse = kernel_predictor.compute_mse(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 The training_set must have fields "input" and "target".
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 The test_set must have field "input", and needs "target" if
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63 we want to compute the squared errors.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65 The predictor parameters are obtained analytically from the training set.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 Training is only done on a whole training set rather than on minibatches
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 (no online implementation).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69 The dataset fields expected and produced by the learning algorithm and the trained model
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 are the following:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 - Input and output dataset fields (example-wise quantities):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 - 'input' (always expected as an input_dataset field)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 - 'target' (always expected by the learning algorithm, optional for learned model)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76 - 'output' (always produced by learned model)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 - 'squared_error' (optionally produced by learned model if 'target' is provided)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78 = example-wise squared error
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 def __init__(self, kernel=None, L2_regularizer=0, gamma=1):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 self.kernel = kernel
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 self.L2_regularizer=L2_regularizer
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 self.gamma = gamma # until we fix things, the kernel type is fixed, Gaussian
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 self.equations = KernelRegressionEquations()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
85
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
86 def __call__(self,trainset):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 n_examples = len(trainset)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88 first_example = trainset[0]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
89 n_inputs = first_example['input'].size
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 n_outputs = first_example['target'].size
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91 M = numpy.zeros((n_examples+1,n_examples+1))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
92 Y = numpy.zeros((n_examples+1,n_outputs))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
93 for i in xrange(n_inputs):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94 M[i+1,i+1]=self.L2_regularizer
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
95 data = trainset.fields()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 train_inputs = numpy.array(data['input'])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 Y[0]=1
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
98 Y[1:,:] = numpy.array(data['target'])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
99 M,train_inputs_square=self.equations.compute_system_matrix(train_inputs,M)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
100 theta=numpy.linalg.solve(M,Y)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
101 return KernelPredictor(theta,self.gamma, train_inputs, train_inputs_square)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
102
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 class KernelPredictorEquations(AutoName):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 train_inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
105 train_inputs_square = T.vector() # n_examples
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
106 inputs = T.matrix() # minibatchsize x n_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
107 targets = T.matrix() # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
108 theta = T.matrix() # (n_examples+1) x n_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
109 gamma = T.scalar()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
110 inv_gamma2 = 1./(gamma*gamma)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
111 b = theta[0]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
112 alpha = theta[1:,:]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
113 inputs_square = T.sum(inputs*inputs,axis=1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
114 Kx = exp(-(train_inputs_square-2*dot(inputs,train_inputs.T)+inputs_square)*inv_gamma2)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
115 outputs = T.dot(Kx,alpha) + b # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
117
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118 __compiled = False
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
119 @classmethod
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
120 def compile(cls,linker='c|py'):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
121 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122 return
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
123 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
125
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126 cls.compute_outputs = fn([cls.inputs,cls.theta,cls.gamma,cls.train_inputs,cls.train_inputs_square],[cls.outputs])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 cls.compute_errors = fn([cls.outputs,cls.targets],[cls.squared_errors])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
128
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
129 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
131 def __init__(self):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
132 self.compile()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
134 class KernelRegressionEquations(KernelPredictorEquations):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 # P = KernelPredictorEquations
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 M = T.matrix() # (n_examples+1) x (n_examples+1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
138 G = M[1:,1:]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
139 new_G = gemm(G,1.,inputs,inputs.T,1.)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 M2 = T.add_inplace(M,new_G)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 M2[0,0] = M.shape[0]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
142 M2[1:,0] = 1
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143 M2[0,1:] = T.sum(G,axis=0)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 inputs_square = T.sum(inputs*inputs,axis=1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
145
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
146 __compiled = False
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
147
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
148 @classmethod
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
149 def compile(cls,linker='c|py'):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
150 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
151 return
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
152 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
154
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
155 cls.compute_system_matrix = fn([cls.inputs,cls.M],[cls.M2,cls.inputs_square])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
156
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
157 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
158
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
159 def __init__(self):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
160 self.compile()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
161
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162 class KernelPredictor(object):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 A kernel predictor has parameters theta (a bias vector and a weight matrix alpha)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 it can use to make a non-linear prediction (according to the KernelPredictorEquations).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
166 It can compute its output (bias + alpha * kernel(train_inputs,input) and a squared error (||output - target||^2).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
168 def __init__(self, theta, gamma, train_inputs, train_inputs_square):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169 self.theta=theta
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
170 self.gamma=gamma
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
171 self.train_inputs=train_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
172 self.train_inputs_square=train_inputs_square
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 self.equations = LinearPredictorEquations()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175 def compute_outputs(self,inputs):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 return self.equations.compute_outputs(inputs,self.theta,self.gamma,self.train_inputs,self.train_inputs_square)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177 def compute_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178 return self.equations.compute_errors(self.compute_outputs(inputs),targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 def compute_outputs_and_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 outputs = self.compute_outputs(inputs)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181 return [outputs,self.equations.compute_errors(outputs,targets)]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 def compute_mse(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 errors = self.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
184 return numpy.sum(errors)/errors.size
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187 assert dataset.hasFields(["input"])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
188 if output_fieldnames is None:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
189 if dataset.hasFields(["target"]):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190 output_fieldnames = ["output","squared_error"]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
191 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
192 output_fieldnames = ["output"]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193 output_fieldnames.sort()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 if output_fieldnames == ["squared_error"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
195 f = self.compute_errors
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
196 elif output_fieldnames == ["output"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 f = self.compute_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 elif output_fieldnames == ["output","squared_error"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199 f = self.compute_outputs_and_errors
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
200 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
201 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
202
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
203 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204 if cached_output_dataset:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 return CachedDataSet(ds)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 return ds
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
209