Mercurial > pylearn

annotate linear_regression.py @ 376:c9a89be5cb0a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Redesigning linear_regression
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Mon, 07 Jul 2008 10:08:35 -0400
parents f6505ec32dc3
children 74b402b5a81b
rev   line source
132
f6505ec32dc3 Updated documentation slightly
Joseph Turian <turian@gmail.com>
parents: 118
diff changeset
1 """
f6505ec32dc3 Updated documentation slightly
Joseph Turian <turian@gmail.com>
parents: 118
diff changeset
2 Implementation of linear regression, with or without L2 regularization.
f6505ec32dc3 Updated documentation slightly
Joseph Turian <turian@gmail.com>
parents: 118
diff changeset
3 This is one of the simplest example of L{learner}, and illustrates
f6505ec32dc3 Updated documentation slightly
Joseph Turian <turian@gmail.com>
parents: 118
diff changeset
4 the use of theano.
f6505ec32dc3 Updated documentation slightly
Joseph Turian <turian@gmail.com>
parents: 118
diff changeset
5 """
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
6
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
7 from pylearn import OfflineLearningAlgorithm
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
8 from theano import tensor as T
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
9 from theano.scalar import as_scalar
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
10 from common.autoname import AutoName
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
11
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
12 class LinearRegression(OfflineLearningAlgorithm):
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
13 """
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
14 Implement linear regression, with or without L2 regularization
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
15 (the former is called Ridge Regression and the latter Ordinary Least Squares).
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
16
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
17 The predictor parameters are obtained analytically from the training set.
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
18 Training can proceed sequentially (with multiple calls to update with
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
19 different disjoint subsets of the training sets). After each call to
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
20 update the predictor is ready to be used (and optimized for the union
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
21 of all the training sets passed to update since construction or since
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
22 the last call to forget).
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
23
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
24 For each (input[t],output[t]) pair in a minibatch,::
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
25
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
26 output_t = b + W * input_t
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
27
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
28 where b and W are obtained by minimizing::
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
29
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
30 L2_regularizer sum_{ij} W_{ij}^2 + sum_t ||output_t - target_t||^2
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
31
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
32 Let X be the whole training set inputs matrix (one input example per row),
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
33 with the first column full of 1's, and Let Y the whole training set
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
34 targets matrix (one example's target vector per row).
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
35 Let theta = the matrix with b in its first column and W in the others,
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
36 then each theta[:,i] is the solution of the linear system::
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
37
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
38 XtX * theta[:,i] = XtY[:,i]
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
39
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
40 where XtX is a (n_inputs+1)x(n_inputs+1) matrix containing X'*X
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
41 plus L2_regularizer on the diagonal except at (0,0),
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
42 and XtY is a (n_inputs+1)*n_outputs matrix containing X'*Y.
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
43
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
44 The dataset fields expected and produced by the learning algorithm and the trained model
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
45 are the following:
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
46
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
47 - Input and output dataset fields (example-wise quantities):
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
48
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
49 - 'input' (always expected as an input_dataset field)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
50 - 'target' (always expected by the learning algorithm, optional for learned model)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
51 - 'output' (always produced by learned model)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
52 - 'squared_error' (optionally produced by learned model if 'target' is provided)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
53 = example-wise squared error
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
54 """
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
55 def __init__(self, L2_regularizer=0):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
56 self.predictor = LinearPredictor(None,None
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
57 self.L2_regularizer=L2_regularizer
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
58 self._XtX = T.matrix('XtX')
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
59 self._XtY = T.matrix('XtY')
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
60 self._extended_input = T.prepend_one_to_each_row(self._input)
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
61
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
62 class LinearPredictorEquations(AutoName):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
63 inputs = T.matrix() # minibatchsize x n_inputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
64 targets = T.matrix() # minibatchsize x n_outputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
65 theta = T.matrix() # (n_inputs+1) x n_outputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
66 b = theta[0]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
67 Wt = theta[1:,:]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
68 outputs = T.dot(inputs,Wt) + b # minibatchsize x n_outputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
69 squared_errors = T.sum(T.sqr(targets-outputs),axis=1)
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
70
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
71 __compiled = False
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
72 @classmethod
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
73 def compile(cls,linker='c|py'):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
74 if cls.__compiled:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
75 return
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
76 def fn(input_vars,output_vars):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
77 return staticmethod(theano.function(input_vars,output_vars, linker=linker))
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
78
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
79 cls.compute_outputs = fn([inputs,theta],[outputs])
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
80 cls.compute_errors = fn([outputs,targets],[squared_errors])
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
81
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
82 cls.__compiled = True
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
83
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
84 def __init__(self)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
85 self.compile()
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
86
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
87 class LinearRegressionEquations(LinearPredictorEquations):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
88 P = LinearPredictorEquations
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
89 XtX = T.matrix() # (n_inputs+1) x (n_inputs+1)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
90 XtY = T.matrix() # (n_inputs+1) x n_outputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
91 extended_input = T.prepend_scalar_to_each_row(1,P.inputs)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
92 new_XtX = add_inplace(XtX,T.dot(extended_input.T,extended_input))
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
93 new_XtY = add_inplace(XtY,T.dot(extended_input.T,P.targets))
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
94
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
95 class LinearPredictor(object):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
96 """
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
97 A linear predictor has parameters theta (a bias vector and a weight matrix)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
98 it can use to make a linear prediction (according to the LinearPredictorEquations).
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
99 It can compute its output (bias + weight * input) and a squared error (||output - target||^2).
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
100 """
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
101 def __init__(self, theta):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
102 self.theta=theta
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
103 self.n_inputs=theta.shape[0]-1
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
104 self.n_outputs=theta.shape[1]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
105 self.predict_equations = LinearPredictorEquations()
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
106
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
107 def compute_outputs(self,inputs):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
108 return self.predict_equations.compute_outputs(inputs,self.theta)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
109 def compute_errors(self,inputs,targets):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
110 return self.predict_equations.compute_errors(self.compute_outputs(inputs),targets)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
111 def compute_outputs_and_errors(self,inputs,targets):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
112 outputs = self.compute_outputs(inputs)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
113 return [outputs,self.predict_equations.compute_errors(outputs,targets)]
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
114
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
115 def __call__(self,dataset,output_fieldnames=None,cached_output_dataset=False):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
116 assert dataset.hasFields(["input"])
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
117 if output_fieldnames is None:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
118 if dataset.hasFields(["target"]):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
119 output_fieldnames = ["output","squared_error"]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
120 else:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
121 output_fieldnames = ["output"]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
122 output_fieldnames.sort()
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
123 if output_fieldnames == ["squared_error"]:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
124 f = self.compute_errors
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
125 elif output_fieldnames == ["output"]:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
126 f = self.compute_outputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
127 elif output_fieldnames == ["output","squared_error"]:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
128 f = self.compute_outputs_and_errors
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
129 else:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
130 raise ValueError("unknown field(s) in output_fieldnames: "+str(output_fieldnames))
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
131
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
132 ds=ApplyFunctionDataSet(dataset,f,output_fieldnames)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
133 if cached_output_dataset:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
134 return CachedDataSet(ds)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
135 else:
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
136 return ds
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
137
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
138
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
139 self._XtX = T.matrix('XtX')
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
140 self._XtY = T.matrix('XtY')
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
141 self._extended_input = T.prepend_one_to_each_row(self._input)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
142 self._output = T.dot(self._input,self._W.T) + self._b # (n_examples , n_outputs) matrix
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
143 self._squared_error = T.sum_within_rows(T.sqr(self._output-self._target)) # (n_examples ) vector
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
144 self._regularizer = self._L2_regularizer * T.dot(self._W,self._W)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
145 self._new_XtX = add_inplace(self._XtX,T.dot(self._extended_input.T,self._extended_input))
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
146 self._new_XtY = add_inplace(self._XtY,T.dot(self._extended_input.T,self._target))
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
147 self._new_theta = T.solve_inplace(self._theta,self._XtX,self._XtY)
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
148
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
149 def allocate(self,dataset):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
150 dataset_n_inputs = dataset["input"].shape[1]
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
151 dataset_n_outputs = dataset["target"].shape[1]
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
152 if not self._n_inputs:
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
153 self._n_inputs = dataset_n_inputs
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
154 self._n_outputs = dataset_n_outputs
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
155 self.XtX = numpy.zeros((1+self._n_inputs,1+self._n_inputs))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
156 self.XtY = numpy.zeros((1+self._n_inputs,self._n_outputs))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
157 self.theta = numpy.zeros((self._n_outputs,1+self._n_inputs))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
158 self.forget()
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
159 elif self._n_inputs!=dataset_n_inputs or self._n_outputs!=dataset_n_outputs:
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
160 # if the input or target changes dimension on the fly, we resize and forget everything
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
161 self.forget()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
162
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
163 def forget(self):
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
164 if self._n_inputs and self._n_outputs:
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
165 self.XtX.resize((1+self.n_inputs,1+self.n_inputs))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
166 self.XtY.resize((1+self.n_inputs,self.n_outputs))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
167 self.XtX.data[:,:]=0
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 107
diff changeset
168 self.XtY.data[:,:]=0
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
169 numpy.diag(self.XtX.data)[1:]=self.L2_regularizer
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents:
diff changeset
170
376
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
171 def __call__(self,dataset):
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
172
c9a89be5cb0a Redesigning linear_regression
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 132
diff changeset
173