Mercurial > pylearn

annotate mlp.py @ 113:b6bc1e769b36

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Automated merge with ssh://p-omega1@lgcm.iro.umontreal.ca/tlearn
author Frederic Bastien <bastienf@iro.umontreal.ca>
date Wed, 07 May 2008 12:12:48 -0400
parents 88257dfedf8c
children d0a1bd0378c6
rev   line source
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
1
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
2 from learner import *
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
3 from theano import tensor as t
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
4 from theano.scalar import as_scalar
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
5
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
6 # this is one of the simplest example of learner, and illustrates
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
7 # the use of theano
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
8
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
9
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
10 class OneHiddenLayerNNetClassifier(MinibatchUpdatesTLearner):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
11 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
12 Implement a straightforward classicial feedforward
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
13 one-hidden-layer neural net, with L2 regularization.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
14
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
15 The predictor parameters are obtained by minibatch/online gradient descent.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
16 Training can proceed sequentially (with multiple calls to update with
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
17 different disjoint subsets of the training sets).
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
18
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
19 Hyper-parameters:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
20 - L2_regularizer
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
21 - learning_rate
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
22 - n_hidden
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
23
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
24 For each (input_t,output_t) pair in a minibatch,::
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
25
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
26 output_activations_t = b2+W2*tanh(b1+W1*input_t)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
27 output_t = softmax(output_activations_t)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
28 output_class_t = argmax(output_activations_t)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
29 class_error_t = 1_{output_class_t != target_t}
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
30 nll_t = -log(output_t[target_t])
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
31
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
32 and the training criterion is::
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
33
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
34 loss = L2_regularizer*(||W1||^2 + ||W2||^2) + sum_t nll_t
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
35
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
36 The parameters are [b1,W1,b2,W2] and are obtained by minimizing the loss by
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
37 stochastic minibatch gradient descent::
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
38
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
39 parameters[i] -= learning_rate * dloss/dparameters[i]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
40
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
41 The fields and attributes expected and produced by use and update are the following:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
42
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
43 - Input and output fields (example-wise quantities):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
44
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
45 - 'input' (always expected by use and update)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
46 - 'target' (optionally expected by use and always by update)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
47 - 'output' (optionally produced by use)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
48 - 'output_class' (optionally produced by use)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
49 - 'class_error' (optionally produced by use)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
50 - 'nll' (optionally produced by use)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
51
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
52 - optional attributes (optionally expected as input_dataset attributes)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
53 (warning, this may be dangerous, the 'use' method will use those provided in the
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
54 input_dataset rather than those learned during 'update'; currently no support
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
55 for providing these to update):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
56
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
57 - 'L2_regularizer'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
58 - 'b1'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
59 - 'W1'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
60 - 'b2'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
61 - 'W2'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
62 - 'parameters' = [b1, W1, b2, W2]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
63 - 'regularization_term'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
64
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
65 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
66
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
67 def attributeNames(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
68 return ["parameters","b1","W2","b2","W2", "L2_regularizer","regularization_term"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
69
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
70 def parameterAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
71 return ["b1","W1", "b2", "W2"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
72
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
73 def useInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
74 return self.parameterAttributes()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
75
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
76 def useOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
77 return []
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
78
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
79 def updateInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
80 return self.parameterAttributes() + ["L2_regularizer"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
81
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
82 def updateMinibatchInputFields(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
83 return ["input","target"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
84
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
85 def updateMinibatchInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
86 return self.parameterAttributes()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
87
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
88 def updateMinibatchOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
89 return self.parameterAttributes()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
90
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
91 def updateEndInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
92 return self.parameterAttributes()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
93
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
94 def updateEndOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
95 return ["regularization_term"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
96
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
97 def defaultOutputFields(self, input_fields):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
98 output_fields = ["output", "output_class",]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
99 if "target" in input_fields:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
100 output_fields += ["class_error", "nll"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
101 return output_fields
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
102
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
103 def __init__(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
104 self._input = t.matrix('input') # n_examples x n_inputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
105 self._target = t.matrix('target') # n_examples x n_outputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
106 self._lambda = as_scalar(0.,'lambda')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
107 self._theta = t.matrix('theta')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
108 self._W = self._theta[:,1:]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
109 self._b = self._theta[:,0]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
110 self._XtX = t.matrix('XtX')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
111 self._XtY = t.matrix('XtY')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
112 self._extended_input = t.prepend_one_to_each_row(self._input)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
113 self._output = t.dot(self._input,self._W.T) + self._b # (n_examples , n_outputs) matrix
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
114 self._squared_error = t.sum_within_rows(t.sqr(self._output-self._target)) # (n_examples ) vector
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
115 self._regularizer = self._lambda * t.dot(self._W,self._W)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
116 self._new_XtX = add_inplace(self._XtX,t.dot(self._extended_input.T,self._extended_input))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
117 self._new_XtY = add_inplace(self._XtY,t.dot(self._extended_input.T,self._target))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
118 self._new_theta = t.solve_inplace(self._theta,self._XtX,self._XtY)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
119
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
120 OneShotTLearner.__init__(self)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
121
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
122 def allocate(self,minibatch):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
123 minibatch_n_inputs = minibatch["input"].shape[1]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
124 minibatch_n_outputs = minibatch["target"].shape[1]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
125 if not self._n_inputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
126 self._n_inputs = minibatch_n_inputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
127 self._n_outputs = minibatch_n_outputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
128 self.XtX = numpy.zeros((1+self._n_inputs,1+self._n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
129 self.XtY = numpy.zeros((1+self._n_inputs,self._n_outputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
130 self.theta = numpy.zeros((self._n_outputs,1+self._n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
131 self.forget()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
132 elif self._n_inputs!=minibatch_n_inputs or self._n_outputs!=minibatch_n_outputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
133 # if the input or target changes dimension on the fly, we resize and forget everything
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
134 self.forget()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
135
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
136 def forget(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
137 if self._n_inputs and self._n_outputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
138 self.XtX.resize((1+self.n_inputs,1+self.n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
139 self.XtY.resize((1+self.n_inputs,self.n_outputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
140 self.XtX.data[:,:]=0
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
141 self.XtY.data[:,:]=0
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
142 numpy.diag(self.XtX.data)[1:]=self.lambda
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
143
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
144
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
145 class MLP(MinibatchUpdatesTLearner):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
146 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
147 Implement a feedforward multi-layer perceptron, with or without L1 and/or L2 regularization.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
148
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
149 The predictor parameters are obtained by minibatch/online gradient descent.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
150 Training can proceed sequentially (with multiple calls to update with
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
151 different disjoint subsets of the training sets).
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
152
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
153 Hyper-parameters:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
154 - L1_regularizer
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
155 - L2_regularizer
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
156 - neuron_sparsity_regularizer
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
157 - initial_learning_rate
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
158 - learning_rate_decrease_rate
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
159 - n_hidden_per_layer (a list of integers)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
160 - activation_function ("sigmoid","tanh", or "ratio")
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
161
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
162 The output/task type (classification, regression, etc.) is obtained by specializing MLP.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
163
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
164 For each (input[t],output[t]) pair in a minibatch,::
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
165
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
166 activation[0] = input_t
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
167 for k=1 to n_hidden_layers:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
168 activation[k]=activation_function(b[k]+ W[k]*activation[k-1])
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
169 output_t = output_activation_function(b[n_hidden_layers+1]+W[n_hidden_layers+1]*activation[n_hidden_layers])
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
170
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
171 and the b and W are obtained by minimizing the following by stochastic minibatch gradient descent::
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
172
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
173 L2_regularizer sum_{ijk} W_{kij}^2 + L1_regularizer sum_{kij} |W_{kij}|
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
174 + neuron_sparsity_regularizer sum_{ki} |b_{ki} + infinity|
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
175 - sum_t log P_{output_model}(target_t | output_t)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
176
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
177 The fields and attributes expected and produced by use and update are the following:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
178
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
179 - Input and output fields (example-wise quantities):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
180
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
181 - 'input' (always expected by use and update)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
182 - 'target' (optionally expected by use and always by update)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
183 - 'output' (optionally produced by use)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
184 - error fields produced by sub-class of MLP
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
185
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
186 - optional attributes (optionally expected as input_dataset attributes)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
187 (warning, this may be dangerous, the 'use' method will use those provided in the
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
188 input_dataset rather than those learned during 'update'; currently no support
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
189 for providing these to update):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
190
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
191 - 'L1_regularizer'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
192 - 'L2_regularizer'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
193 - 'b'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
194 - 'W'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
195 - 'parameters' = [b[1], W[1], b[2], W[2], ...]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
196 - 'regularization_term'
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
197
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
198 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
199
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
200 def attributeNames(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
201 return ["parameters","b","W","L1_regularizer","L2_regularizer","neuron_sparsity_regularizer","regularization_term"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
202
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
203 def useInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
204 return ["b","W"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
205
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
206 def useOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
207 return []
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
208
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
209 def updateInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
210 return ["b","W","L1_regularizer","L2_regularizer","neuron_sparsity_regularizer"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
211
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
212 def updateMinibatchInputFields(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
213 return ["input","target"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
214
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
215 def updateMinibatchInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
216 return ["b","W"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
217
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
218 def updateMinibatchOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
219 return ["new_XtX","new_XtY"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
220
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
221 def updateEndInputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
222 return ["theta","XtX","XtY"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
223
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
224 def updateEndOutputAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
225 return ["new_theta","b","W","regularization_term"] # CHECK: WILL b AND W CONTAIN OLD OR NEW THETA? @todo i.e. order of computation = ?
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
226
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
227 def parameterAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
228 return ["b","W"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
229
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
230 def defaultOutputFields(self, input_fields):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
231 output_fields = ["output"]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
232 if "target" in input_fields:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
233 output_fields.append("squared_error")
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
234 return output_fields
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
235
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
236 def __init__(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
237 self._input = t.matrix('input') # n_examples x n_inputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
238 self._target = t.matrix('target') # n_examples x n_outputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
239 self._lambda = as_scalar(0.,'lambda')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
240 self._theta = t.matrix('theta')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
241 self._W = self._theta[:,1:]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
242 self._b = self._theta[:,0]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
243 self._XtX = t.matrix('XtX')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
244 self._XtY = t.matrix('XtY')
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
245 self._extended_input = t.prepend_one_to_each_row(self._input)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
246 self._output = t.dot(self._input,self._W.T) + self._b # (n_examples , n_outputs) matrix
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
247 self._squared_error = t.sum_within_rows(t.sqr(self._output-self._target)) # (n_examples ) vector
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
248 self._regularizer = self._lambda * t.dot(self._W,self._W)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
249 self._new_XtX = add_inplace(self._XtX,t.dot(self._extended_input.T,self._extended_input))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
250 self._new_XtY = add_inplace(self._XtY,t.dot(self._extended_input.T,self._target))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
251 self._new_theta = t.solve_inplace(self._theta,self._XtX,self._XtY)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
252
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
253 OneShotTLearner.__init__(self)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
254
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
255 def allocate(self,minibatch):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
256 minibatch_n_inputs = minibatch["input"].shape[1]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
257 minibatch_n_outputs = minibatch["target"].shape[1]
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
258 if not self._n_inputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
259 self._n_inputs = minibatch_n_inputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
260 self._n_outputs = minibatch_n_outputs
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
261 self.XtX = numpy.zeros((1+self._n_inputs,1+self._n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
262 self.XtY = numpy.zeros((1+self._n_inputs,self._n_outputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
263 self.theta = numpy.zeros((self._n_outputs,1+self._n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
264 self.forget()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
265 elif self._n_inputs!=minibatch_n_inputs or self._n_outputs!=minibatch_n_outputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
266 # if the input or target changes dimension on the fly, we resize and forget everything
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
267 self.forget()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
268
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
269 def forget(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
270 if self._n_inputs and self._n_outputs:
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
271 self.XtX.resize((1+self.n_inputs,1+self.n_inputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
272 self.XtY.resize((1+self.n_inputs,self.n_outputs))
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
273 self.XtX.data[:,:]=0
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
274 self.XtY.data[:,:]=0
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
275 numpy.diag(self.XtX.data)[1:]=self.lambda
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents:
diff changeset
276