Mercurial > pylearn

annotate gradient_learner.py @ 13:633453635d51

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Starting to work on gradient_based_learner.py
author bengioy@bengiomac.local
date Wed, 26 Mar 2008 21:38:08 -0400
parents
children 5ede27026e05
rev   line source
13
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
1
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
2 from learner import *
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
3 from tensor import *
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
4 import gradient
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
5 from compile import Function
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
6 from gradient_based_optimizer import *
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
7
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
8 class GradientLearner(Learner):
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
9 """
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
10 Generic Learner for gradient-based optimization of a training criterion
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
11 that can consist in two parts, an additive part over examples, and
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
12 an example-independent part (usually called the regularizer).
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
13 The user provides a Theano formula that maps the fields of a training example
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
14 and parameters to output fields (for the use function), one of which must be a cost
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
15 that is the training criterion to be minimized. The user also provides
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
16 a GradientBasedOptimizer that implements the optimization strategy.
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
17 The inputs, parameters, outputs and lists of Theano tensors,
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
18 while the example_wise_cost and regularization_term are Theano tensors.
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
19 The user can specify a regularization coefficient that multiplies the regularization term.
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
20 The training algorithm looks for parameters that minimize
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
21 regularization_coefficienet * regularization_term(parameters) +
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
22 sum_{inputs in training_set} example_wise_cost(inputs,parameters)
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
23 i.e. the regularization_term should not depend on the inputs, only on the parameters.
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
24 The learned function can map a subset of inputs to a subset of outputs (as long as the inputs subset
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
25 includes all the inputs required in the Theano expression for the selected outputs).
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
26 """
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
27 def __init__(self, inputs, parameters, outputs, example_wise_cost, regularization_term,
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
28 gradient_based_optimizer=StochasticGradientDescent(), regularization_coefficient = astensor(1.0)):
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
29 self.inputs = inputs
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
30 self.outputs = outputs
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
31 self.parameters = parameters
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
32 self.example_wise_cost = example_wise_cost
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
33 self.regularization_term = regularization_term
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
34 self.gradient_based_optimizer = gradient_based_optimizer
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
35 self.regularization_coefficient = regularization_coefficient
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
36 self.parameters_example_wise_gradient = gradient.grad(example_wise_cost, parameters)
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
37 self.parameters_regularization_gradient = gradient.grad(self.regularization_coefficient * regularization, parameters)
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
38
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
39 # def update(self,training_set):
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents:
diff changeset
40