Mercurial > pylearn

annotate learner.py @ 118:d0a1bd0378c6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Wed, 07 May 2008 15:07:56 -0400
parents 88257dfedf8c
children 4efe6d36c061
rev   line source
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
1
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
2 from dataset import AttributesHolder
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
3 import compile
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
4
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
5 class Learner(AttributesHolder):
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
6 """Base class for learning algorithms, provides an interface
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
7 that allows various algorithms to be applicable to generic learning
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
8 algorithms.
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
9
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
10 A Learner can be seen as a learning algorithm, a function that when
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
11 applied to training data returns a learned function, an object that
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
12 can be applied to other data and return some output data.
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
13 """
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
14
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
15 def __init__(self):
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
16 pass
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
17
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
18 def forget(self):
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
19 """
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
20 Reset the state of the learner to a blank slate, before seeing
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
21 training data. The operation may be non-deterministic if the
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
22 learner has a random number generator that is set to use a
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
23 different seed each time it forget() is called.
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
24 """
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
25 raise NotImplementedError
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
26
14
5ede27026e05 Working on gradient_based_learner
bengioy@bengiomac.local
parents: 13
diff changeset
27 def update(self,training_set,train_stats_collector=None):
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
28 """
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
29 Continue training a learner, with the evidence provided by the given training set.
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
30 Hence update can be called multiple times. This is particularly useful in the
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
31 on-line setting or the sequential (Bayesian or not) settings.
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
32 The result is a function that can be applied on data, with the same
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
33 semantics of the Learner.use method.
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents: 20
diff changeset
34
14
5ede27026e05 Working on gradient_based_learner
bengioy@bengiomac.local
parents: 13
diff changeset
35 The user may optionally provide a training StatsCollector that is used to record
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents: 20
diff changeset
36 some statistics of the outputs computed during training. It is update(d) during
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents: 20
diff changeset
37 training.
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
38 """
13
633453635d51 Starting to work on gradient_based_learner.py
bengioy@bengiomac.local
parents: 10
diff changeset
39 return self.use # default behavior is 'non-adaptive', i.e. update does not do anything
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
40
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
41
20
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works.
bengioy@bengiomac.local
parents: 14
diff changeset
42 def __call__(self,training_set,train_stats_collector=None):
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
43 """
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
44 Train a learner from scratch using the provided training set,
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
45 and return the learned function.
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
46 """
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
47 self.forget()
20
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works.
bengioy@bengiomac.local
parents: 14
diff changeset
48 return self.update(learning_task,train_stats_collector)
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
49
20
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works.
bengioy@bengiomac.local
parents: 14
diff changeset
50 def use(self,input_dataset,output_fields=None,copy_inputs=True):
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
51 """Once a Learner has been trained by one or more call to 'update', it can
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
52 be used with one or more calls to 'use'. The argument is a DataSet (possibly
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
53 containing a single example) and the result is a DataSet of the same length.
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
54 If output_fields is specified, it may be use to indicate which fields should
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
55 be constructed in the output DataSet (for example ['output','classification_error']).
20
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works.
bengioy@bengiomac.local
parents: 14
diff changeset
56 Optionally, if copy_inputs, the input fields (of the input_dataset) can be made
75
90e4c0784d6e Added draft of LinearRegression learner
bengioy@bengiomac.local
parents: 20
diff changeset
57 visible in the output DataSet returned by this method.
10
80bf5492e571 Rewrote learner.py according to the specs in the wiki for learners.
bengioy@esprit.iro.umontreal.ca
parents: 1
diff changeset
58 """
1
2cd82666b9a7 Added statscollector and started writing dataset and learner.
bengioy@esprit.iro.umontreal.ca
parents: 0
diff changeset
59 raise NotImplementedError
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
60
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
61 def attributeNames(self):
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
62 """
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
63 A Learner may have attributes that it wishes to export to other objects. To automate
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
64 such export, sub-classes should define here the names (list of strings) of these attributes.
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
65
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
66 @todo By default, attributeNames looks for all dictionary entries whose name does not start with _.
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
67 """
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
68 return []
77
1e2bb5bad636 toying with different ways to implement learners
bengioy@bengiomac.local
parents: 75
diff changeset
69
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
70 def updateInputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
71 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
72 A subset of self.attributeNames() which are the names of attributes needed by update() in order
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
73 to do its work.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
74 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
75 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
76
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
77 def useInputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
78 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
79 A subset of self.attributeNames() which are the names of attributes needed by use() in order
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
80 to do its work.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
81 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
82 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
83
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
84 def updateOutputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
85 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
86 A subset of self.attributeNames() which are the names of attributes modified/created by update() in order
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
87 to do its work.
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
88
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
89 By default these are inferred from the various update output attributes:
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
90 """
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
91 return ["parameters"] + self.updateMinibatchOutputAttributes() + self.updateEndOutputAttributes()
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
92
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
93 def useOutputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
94 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
95 A subset of self.attributeNames() which are the names of attributes modified/created by use() in order
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
96 to do its work.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
97 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
98 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
99
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
100
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
101 class TLearner(Learner):
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
102 """
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
103 TLearner is a virtual class of Learners that attempts to factor out of the definition
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
104 of a learner the steps that are common to many implementations of learning algorithms,
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
105 so as to leave only 'the equations' to define in particular sub-classes, using Theano.
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
106
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
107 In the default implementations of use and update, it is assumed that the 'use' and 'update' methods
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
108 visit examples in the input dataset sequentially. In the 'use' method only one pass through the dataset is done,
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
109 whereas the sub-learner may wish to iterate over the examples multiple times. Subclasses where this
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
110 basic model is not appropriate can simply redefine update or use.
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
111
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
112 Sub-classes must provide the following functions and functionalities:
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
113 - attributeNames(): defines all the names of attributes which can be used as fields or
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
114 attributes in input/output datasets or in stats collectors.
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
115 All these attributes are expected to be theano.Result objects
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
116 (with a .data property and recognized by theano.Function for compilation).
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
117 The sub-class constructor defines the relations between
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
118 the Theano variables that may be used by 'use' and 'update'
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
119 or by a stats collector.
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
120 - defaultOutputFields(input_fields): return a list of default dataset output fields when
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
121 None are provided by the caller of use.
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
122 The following naming convention is assumed and important.
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
123 Attributes whose names are listed in attributeNames() can be of any type,
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
124 but those that can be referenced as input/output dataset fields or as
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
125 output attributes in 'use' or as input attributes in the stats collector
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
126 should be associated with a Theano Result variable. If the exported attribute
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
127 name is <name>, the corresponding Result name (an internal attribute of
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
128 the TLearner, created in the sub-class constructor) should be _<name>.
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
129 Typically <name> will be numpy ndarray and _<name> will be the corresponding
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
130 Theano Tensor (for symbolic manipulation).
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
131
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
132 @todo pousser dans Learner toute la poutine qui peut l'etre sans etre
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
133 dependant de Theano
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
134 """
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
135
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
136 def __init__(self):
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
137 Learner.__init__(self)
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
138
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
139 def defaultOutputFields(self, input_fields):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
140 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
141 Return a default list of output field names (to put in the output dataset).
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
142 This will be used when None are provided (as output_fields) by the caller of the 'use' method.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
143 This may involve looking at the input_fields (names) available in the
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
144 input_dataset.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
145 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
146 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
147
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
148 def allocate(self, minibatch):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
149 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
150 This function is called at the beginning of each updateMinibatch
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
151 and should be used to check that all required attributes have been
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
152 allocated and initialized (usually this function calls forget()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
153 when it has to do an initialization).
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
154 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
155 raise AbstractFunction()
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
156
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
157 def minibatchwise_use_functions(self, input_fields, output_fields, stats_collector):
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
158 """
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
159 Private helper function called by the generic TLearner.use. It returns a function
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
160 that can map the given input fields to the given output fields (along with the
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
161 attributes that the stats collector needs for its computation. The function
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
162 called also automatically makes use of the self.useInputAttributes() and
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
163 sets the self.useOutputAttributes().
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
164 """
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
165 if not output_fields:
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
166 output_fields = self.defaultOutputFields(input_fields)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
167 if stats_collector:
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
168 stats_collector_inputs = stats_collector.input2UpdateAttributes()
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
169 for attribute in stats_collector_inputs:
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
170 if attribute not in input_fields:
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
171 output_fields.append(attribute)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
172 key = (input_fields,output_fields)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
173 if key not in self.use_functions_dictionary:
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
174 use_input_attributes = self.useInputAttributes()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
175 use_output_attributes = self.useOutputAttributes()
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
176 complete_f = compile.function(self.names2OpResults(input_fields+use_input_attributes),
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
177 self.names2OpResults(output_fields+use_output_attributes))
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
178 def f(*input_field_values):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
179 input_attribute_values = self.names2attributes(use_input_attributes)
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
180 results = complete_f(*(input_field_values + input_attribute_values))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
181 output_field_values = results[0:len(output_fields)]
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
182 output_attribute_values = results[len(output_fields):len(results)]
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
183 if use_output_attributes:
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
184 self.setAttributes(use_output_attributes,output_attribute_values)
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
185 return output_field_values
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
186 self.use_functions_dictionary[key]=f
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
187 return self.use_functions_dictionary[key]
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
188
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
189 def attributes(self,return_copy=False):
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
190 """
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
191 Return a list with the values of the learner's attributes (or optionally, a deep copy).
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
192 """
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
193 return self.names2attributes(self.attributeNames(),return_copy)
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
194
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
195 def names2attributes(self,names,return_copy=False):
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
196 """
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
197 Private helper function that maps a list of attribute names to a list
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
198 of (optionally copies) values of attributes.
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
199 """
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
200 if return_copy:
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
201 return [copy.deepcopy(self.__getattr__(name).data) for name in names]
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
202 else:
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
203 return [self.__getattr__(name).data for name in names]
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
204
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
205 def names2OpResults(self,names):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
206 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
207 Private helper function that maps a list of attribute names to a list
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
208 of corresponding Op Results (with the same name but with a '_' prefix).
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
209 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
210 return [self.__getattr__('_'+name).data for name in names]
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
211
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
212 def use(self,input_dataset,output_fieldnames=None,output_attributes=[],
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
213 test_stats_collector=None,copy_inputs=True, put_stats_in_output_dataset=True):
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
214 """
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
215 The learner tries to compute in the output dataset the output fields specified
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
216
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
217 @todo check if some of the learner attributes are actually SPECIFIED
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
218 as attributes of the input_dataset, and if so use their values instead
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
219 of the ones in the learner.
109
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
220
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
221 The learner tries to compute in the output dataset the output fields specified.
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
222 If None is specified then self.defaultOutputFields(input_dataset.fieldNames())
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
223 is called to determine the output fields.
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
224
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
225 Attributes of the learner can also optionally be copied into the output dataset.
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
226 If output_attributes is None then all of the attributes in self.AttributeNames()
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
227 are copied in the output dataset, but if it is [] (the default), then none are copied.
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
228 If a test_stats_collector is provided, then its attributes (test_stats_collector.AttributeNames())
d97f6fe6bdf9 Merged with this morning unsaved edits
bengioy@bengiomac.local
parents: 107
diff changeset
229 are also copied into the output dataset attributes.
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
230 """
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
231 minibatchwise_use_function = minibatchwise_use_functions(input_dataset.fieldNames(),
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
232 output_fieldnames,
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
233 test_stats_collector)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
234 virtual_output_dataset = ApplyFunctionDataSet(input_dataset,
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
235 minibatchwise_use_function,
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
236 True,DataSet.numpy_vstack,
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
237 DataSet.numpy_hstack)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
238 # actually force the computation
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
239 output_dataset = CachedDataSet(virtual_output_dataset,True)
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
240 if copy_inputs:
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
241 output_dataset = input_dataset | output_dataset
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
242 # copy the wanted attributes in the dataset
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
243 if output_attributes is None:
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
244 output_attributes = self.attributeNames()
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
245 if output_attributes:
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
246 assert set(attribute_names) <= set(self.attributeNames())
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
247 output_dataset.setAttributes(output_attributes,
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
248 self.names2attributes(output_attributes,return_copy=True))
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
249 if test_stats_collector:
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
250 test_stats_collector.update(output_dataset)
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
251 if put_stats_in_output_dataset:
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
252 output_dataset.setAttributes(test_stats_collector.attributeNames(),
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
253 test_stats_collector.attributes())
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
254 return output_dataset
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
255
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
256
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
257 class MinibatchUpdatesTLearner(TLearner):
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
258 """
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
259 This adds to TLearner a
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
260 - updateStart(), updateEnd(), updateMinibatch(minibatch), isLastEpoch():
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
261 functions executed at the beginning, the end, in the middle
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
262 (for each minibatch) of the update method, and at the end
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
263 of each epoch. This model only
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
264 works for 'online' or one-shot learning that requires
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
265 going only once through the training data. For more complicated
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
266 models, more specialized subclasses of TLearner should be used
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
267 or a learning-algorithm specific update method should be defined.
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
268
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
269 - a 'parameters' attribute which is a list of parameters (whose names are
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
270 specified by the user's subclass with the parameterAttributes() method)
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
271
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
272 """
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
273
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
274 def __init__(self):
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
275 TLearner.__init__(self)
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
276 self.update_minibatch_function = compile.function
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
277 (self.names2OpResults(self.updateMinibatchOutputAttributes()+
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
278 self.updateMinibatchInputFields()),
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
279 self.names2OpResults(self.updateMinibatchOutputAttributes()))
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
280 self.update_end_function = compile.function
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
281 (self.names2OpResults(self.updateEndInputAttributes()),
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
282 self.names2OpResults(self.updateEndOutputAttributes()))
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
283
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
284 def updateMinibatchInputFields(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
285 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
286
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
287 def updateMinibatchInputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
288 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
289
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
290 def updateMinibatchOutputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
291 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
292
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
293 def updateEndInputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
294 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
295
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
296 def updateEndOutputAttributes(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
297 raise AbstractFunction()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
298
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
299 def parameterAttributes(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
300 raise AbstractFunction()
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
301
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
302 def updateStart(self): pass
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
303
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
304 def updateEnd(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
305 self.setAttributes(self.updateEndOutputAttributes(),
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
306 self.update_end_function
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
307 (self.names2attributes(self.updateEndInputAttributes())))
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
308 self.parameters = self.names2attributes(self.parameterAttributes())
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
309
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
310 def updateMinibatch(self,minibatch):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
311 # make sure all required fields are allocated and initialized
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
312 self.allocate(minibatch)
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
313 self.setAttributes(self.updateMinibatchOutputAttributes(),
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
314 # concatenate the attribute values and field values and then apply update fn
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
315 self.update_minibatch_function(*(self.names2attributes
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
316 (self.updateMinibatchInputAttributes()))
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
317 + minibatch(self.updateMinibatchInputFields())))
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
318
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
319 def isLastEpoch(self):
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
320 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
321 This method is called at the end of each epoch (cycling over the training set).
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
322 It returns a boolean to indicate if this is the last epoch.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
323 By default just do one epoch.
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
324 """
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
325 return True
78
3499918faa9d In the middle of designing TLearner
bengioy@bengiomac.local
parents: 77
diff changeset
326
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
327 def update(self,training_set,train_stats_collector=None):
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
328 """
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
329 @todo check if some of the learner attributes are actually SPECIFIED
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
330 in as attributes of the training_set.
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
331 """
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
332 self.updateStart(training_set)
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
333 stop=False
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
334 while not stop:
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
335 if train_stats_collector:
107
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
336 train_stats_collector.forget() # restart stats collectin at the beginning of each epoch
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
337 for minibatch in training_set.minibatches(self.training_set_input_fields,
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
338 minibatch_size=self.minibatch_size):
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
339 self.update_minibatch(minibatch)
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
340 if train_stats_collector:
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
341 minibatch_set = minibatch.examples()
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
342 minibatch_set.setAttributes(self.attributeNames(),self.attributes())
c4916445e025 Comments from Pascal V.
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 92
diff changeset
343 train_stats_collector.update(minibatch_set)
110
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
344 stop = self.isLastEpoch()
8fa1ef2411a0 Worked on OneShotTLearner and implementation of LinearRegression
bengioy@bengiomac.local
parents: 109
diff changeset
345 self.updateEnd()
92
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
346 return self.use
c4726e19b8ec Finished first draft of TLearner
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 78
diff changeset
347
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
348 class OnlineGradientBasedTLearner(MinibatchUpdatesTLearner):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
349 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
350 Specialization of MinibatchUpdatesTLearner in which the minibatch updates
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
351 are obtained by performing an online (minibatch-based) gradient step.
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
352
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
353 Sub-classes must define the following:
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
354
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
355 self._learning_rate (may be changed by the sub-class between epochs or minibatches)
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
356
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
357 self.lossAttribute() = name of the loss field
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
358
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
359 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
360 def __init__(self,truly_online=False):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
361 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
362 If truly_online then only one pass is made through the training set passed to update().
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
363
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
364 SUBCLASSES SHOULD CALL THIS CONSTRUCTOR ONLY AFTER HAVING DEFINED ALL THEIR THEANO FORMULAS
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
365 """
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
366 self.truly_online=truly_online
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
367
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
368 # create the formulas for the gradient update
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
369 old_params = [self.__getattr__("_"+name) for name in self.parameterAttributes()]
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
370 new_params_names = ["_new_"+name for name in self.parameterAttributes()]
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
371 loss = self.__getattr__(self.lossAttribute())
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
372 self.setAttributes(new_params_names,
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
373 [t.add_inplace(self.param,
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
374 self._learning_rate*t.grad(loss,param))
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
375 for param in old_params])
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
376
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
377 def isLastEpoch(self):
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
378 return self.truly_online
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
379
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
380 def updateMinibatchInputAttributes(self):
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
381 return self.parameterAttributes()
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
382
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
383 def updateMinibatchOutputAttributes(self):
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
384 return ["_new"+name for name in self.parameterAttributes()]
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
385
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
386 def updateEndInputAttributes(self):
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
387 return self.parameterAttributes()
111
88257dfedf8c Added another work in progress, for mlp's
bengioy@bengiomac.local
parents: 110
diff changeset
388
118
d0a1bd0378c6 Finished draft of OneHiddenLayerNNetClassifier to debut learner.py
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 111
diff changeset
389