annotate kernel_regression.py @ 490:4f3c66146f17

Moved weights.py out of sandbox
author Joseph Turian <turian@gmail.com>
date Tue, 28 Oct 2008 10:54:26 -0400
parents 0f8c81b0776d
children 05f802184606
rev   line source
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
1 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
2 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
3 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
4
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
5 from pylearn.learner import OfflineLearningAlgorithm
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
6 from theano import tensor as T
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
7 from nnet_ops import prepend_1_to_each_row
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
8 from theano.scalar import as_scalar
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
9 from common.autoname import AutoName
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
10 import theano
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
11 import numpy
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
12
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
13 # map a N-vector to a 1xN matrix
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
14 row_vector = theano.elemwise.DimShuffle((False,),['x',0])
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
15 # map a N-vector to a Nx1 matrix
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
16 col_vector = theano.elemwise.DimShuffle((False,),[0,'x'])
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
17
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
18 class KernelRegression(OfflineLearningAlgorithm):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
19 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
20 Implementation of kernel regression:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
21 * the data are n (x_t,y_t) pairs and we want to estimate E[y|x]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
22 * the predictor computes
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
23 f(x) = b + \sum_{t=1}^n \alpha_t K(x,x_t)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
24 with free parameters b and alpha, training inputs x_t,
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
25 and kernel function K (gaussian by default).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
26 Clearly, each prediction involves O(n) computations.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
27 * the learner chooses b and alpha to minimize
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
28 lambda alpha' G' G alpha + \sum_{t=1}^n (f(x_t)-y_t)^2
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
29 where G is the matrix with entries G_ij = K(x_i,x_j).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
30 The first (L2 regularization) term is the squared L2
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
31 norm of the primal weights w = \sum_t \alpha_t phi(x_t)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
32 where phi is the function s.t. K(u,v)=phi(u).phi(v).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
33 * this involves solving a linear system with (n+1,n+1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
34 matrix, which is an O(n^3) computation. In addition,
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
35 that linear system matrix requires O(n^2) memory.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
36 So this learning algorithm should be used only for
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
37 small datasets.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
38 * the linear system is
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
39 (M + lambda I_n) theta = (1, y)'
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
40 where theta = (b, alpha), I_n is the (n+1)x(n+1) matrix that is the identity
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
41 except with a 0 at (0,0), M is the matrix with G in the sub-matrix starting
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
42 at (1,1), 1's in column 0, except for a value of n at (0,0), and sum_i G_{i,j}
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
43 in the rest of row 0.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
44
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
45 Note that this is gives an estimate of E[y|x,training_set] that is the
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
46 same as obtained with a Gaussian process regression. The GP
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
47 regression would also provide a Bayesian Var[y|x,training_set].
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
48 It corresponds to an assumption that f is a random variable
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
49 with Gaussian (process) prior distribution with covariance
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
50 function K. Because we assume Gaussian noise we obtain a Gaussian
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
51 posterior for f (whose mean is computed here).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
52
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
53
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
54 Usage:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
55
422
32c5f87bc54e Added __len__ to HStackedDataSet and replaced default len() by sys.maxint instead of None
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 421
diff changeset
56 kernel_regressor=KernelRegression(L2_regularizer=0.1,gamma=0.5) (kernel=GaussianKernel(gamma=0.5))
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
57 kernel_predictor=kernel_regressor(training_set)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
58 all_results_dataset=kernel_predictor(test_set) # creates a dataset with "output" and "squared_error" field
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
59 outputs = kernel_predictor.compute_outputs(inputs) # inputs and outputs are numpy arrays
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
60 outputs, errors = kernel_predictor.compute_outputs_and_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
61 errors = kernel_predictor.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
62 mse = kernel_predictor.compute_mse(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
63
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
64
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
65
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
66 The training_set must have fields "input" and "target".
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
67 The test_set must have field "input", and needs "target" if
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
68 we want to compute the squared errors.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
69
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
70 The predictor parameters are obtained analytically from the training set.
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
71 Training is only done on a whole training set rather than on minibatches
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
72 (no online implementation).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
73
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
74 The dataset fields expected and produced by the learning algorithm and the trained model
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
75 are the following:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
76
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
77 - Input and output dataset fields (example-wise quantities):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
78
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
79 - 'input' (always expected as an input_dataset field)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
80 - 'target' (always expected by the learning algorithm, optional for learned model)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
81 - 'output' (always produced by learned model)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
82 - 'squared_error' (optionally produced by learned model if 'target' is provided)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
83 = example-wise squared error
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
84 """
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
85 def __init__(self, kernel=None, L2_regularizer=0, gamma=1, use_bias=False):
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
86 # THE VERSION WITH BIAS DOES NOT SEEM RIGHT
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
87 self.kernel = kernel
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
88 self.L2_regularizer=L2_regularizer
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
89 self.use_bias=use_bias
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
90 self.gamma = gamma # until we fix things, the kernel type is fixed, Gaussian
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
91 self.equations = KernelRegressionEquations()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
92
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
93 def __call__(self,trainset):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
94 n_examples = len(trainset)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
95 first_example = trainset[0]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
96 n_inputs = first_example['input'].size
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
97 n_outputs = first_example['target'].size
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
98 b1=1 if self.use_bias else 0
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
99 M = numpy.zeros((n_examples+b1,n_examples+b1))
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
100 Y = numpy.zeros((n_examples+b1,n_outputs))
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
101 for i in xrange(n_examples):
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
102 M[i+b1,i+b1]=self.L2_regularizer
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
103 data = trainset.fields()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
104 train_inputs = numpy.array(data['input'])
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
105 if self.use_bias:
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
106 Y[0]=1
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
107 Y[b1:,:] = numpy.array(data['target'])
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
108 train_inputs_square,sumG,G=self.equations.compute_system_matrix(train_inputs,self.gamma)
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
109 M[b1:,b1:] += G
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
110 if self.use_bias:
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
111 M[0,1:] = sumG
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
112 M[1:,0] = 1
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
113 M[0,0] = M.shape[0]
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
114 self.M=M
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
115 self.Y=Y
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
116 theta=numpy.linalg.solve(M,Y)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
117 return KernelPredictor(theta,self.gamma, train_inputs, train_inputs_square)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
118
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
119 class KernelPredictorEquations(AutoName):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
120 train_inputs = T.matrix() # n_examples x n_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
121 train_inputs_square = T.vector() # n_examples
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
122 inputs = T.matrix() # minibatchsize x n_inputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
123 targets = T.matrix() # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
124 theta = T.matrix() # (n_examples+1) x n_outputs
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
125 b1 = T.shape(train_inputs_square)[0]<T.shape(theta)[0]
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
126 gamma = T.scalar()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
127 inv_gamma2 = 1./(gamma*gamma)
426
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
128 b = b1*theta[0]
d7611a3811f2 Moved incomplete stuff to sandbox
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 425
diff changeset
129 alpha = theta[b1:,:]
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
130 inputs_square = T.sum(inputs*inputs,axis=1)
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
131 Kx = T.exp(-(row_vector(train_inputs_square)-2*T.dot(inputs,train_inputs.T)+col_vector(inputs_square))*inv_gamma2)
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
132 outputs = T.dot(Kx,alpha) + b # minibatchsize x n_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
133 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
134
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
135 __compiled = False
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
136 @classmethod
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
137 def compile(cls,linker='c|py'):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
138 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
139 return
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
140 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
141 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
142
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
143 cls.compute_outputs = fn([cls.inputs,cls.theta,cls.gamma,cls.train_inputs,cls.train_inputs_square],[cls.outputs])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
144 cls.compute_errors = fn([cls.outputs,cls.targets],[cls.squared_errors])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
145
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
146 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
147
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
148 def __init__(self):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
149 self.compile()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
150
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
151 class KernelRegressionEquations(KernelPredictorEquations):
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
152 #M = T.matrix() # (n_examples+1) x (n_examples+1)
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
153 inputs = T.matrix() # n_examples x n_inputs
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
154 gamma = T.scalar()
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
155 inv_gamma2 = 1./(gamma*gamma)
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
156 inputs_square = T.sum(inputs*inputs,axis=1)
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
157 #new_G = G+T.dot(inputs,inputs.T)
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
158 #new_G = T.gemm(G,1.,inputs,inputs.T,1.)
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
159 G = T.exp(-(row_vector(inputs_square)-2*T.dot(inputs,inputs.T)+col_vector(inputs_square))*inv_gamma2)
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
160 sumG = T.sum(G,axis=0)
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
161
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
162 __compiled = False
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
163
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
164 @classmethod
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
165 def compile(cls,linker='c|py'):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
166 if cls.__compiled:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
167 return
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
168 def fn(input_vars,output_vars):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
169 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
170
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
171 cls.compute_system_matrix = fn([cls.inputs,cls.gamma],[cls.inputs_square,cls.sumG,cls.G])
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
172
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
173 cls.__compiled = True
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
174
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
175 def __init__(self):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
176 self.compile()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
177
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
178 class KernelPredictor(object):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
179 """
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
180 A kernel predictor has parameters theta (a bias vector and a weight matrix alpha)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
181 it can use to make a non-linear prediction (according to the KernelPredictorEquations).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
182 It can compute its output (bias + alpha * kernel(train_inputs,input) and a squared error (||output - target||^2).
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
183 """
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
184 def __init__(self, theta, gamma, train_inputs, train_inputs_square=None):
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
185 self.theta=theta
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
186 self.gamma=gamma
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
187 self.train_inputs=train_inputs
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
188 if train_inputs_square==None:
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
189 train_inputs_square = numpy.sum(train_inputs*train_inputs,axis=1)
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
190 self.train_inputs_square=train_inputs_square
425
e2b46a8f2b7b Debugging kernel regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 422
diff changeset
191 self.equations = KernelPredictorEquations()
421
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
192
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
193 def compute_outputs(self,inputs):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
194 return self.equations.compute_outputs(inputs,self.theta,self.gamma,self.train_inputs,self.train_inputs_square)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
195 def compute_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
196 return self.equations.compute_errors(self.compute_outputs(inputs),targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
197 def compute_outputs_and_errors(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
198 outputs = self.compute_outputs(inputs)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
199 return [outputs,self.equations.compute_errors(outputs,targets)]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
200 def compute_mse(self,inputs,targets):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
201 errors = self.compute_errors(inputs,targets)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
202 return numpy.sum(errors)/errors.size
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
203
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
204 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
205 assert dataset.hasFields(["input"])
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
206 if output_fieldnames is None:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
207 if dataset.hasFields(["target"]):
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
208 output_fieldnames = ["output","squared_error"]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
209 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
210 output_fieldnames = ["output"]
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
211 output_fieldnames.sort()
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
212 if output_fieldnames == ["squared_error"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
213 f = self.compute_errors
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
214 elif output_fieldnames == ["output"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
215 f = self.compute_outputs
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
216 elif output_fieldnames == ["output","squared_error"]:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
217 f = self.compute_outputs_and_errors
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
218 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
219 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
220
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
221 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
222 if cached_output_dataset:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
223 return CachedDataSet(ds)
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
224 else:
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
225 return ds
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
226
e01f17be270a Kernel regression learning algorithm
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents:
diff changeset
227
431
0f8c81b0776d Adding file make_test_datasets to host simple data-generating processes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 426
diff changeset
228 def kernel_predictor(inputs,params,*otherargs):
0f8c81b0776d Adding file make_test_datasets to host simple data-generating processes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 426
diff changeset
229 p = KernelPredictor(params,*otherargs[0])
0f8c81b0776d Adding file make_test_datasets to host simple data-generating processes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 426
diff changeset
230 return p.compute_outputs(inputs)
0f8c81b0776d Adding file make_test_datasets to host simple data-generating processes
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 426
diff changeset
231