Mercurial > pylearn

annotate doc/v2_planning/datalearn.txt @ 1361:7548dc1b163c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Some question/suggestions to datalearn
author Razvan Pascanu <r.pascanu@gmail.com>
date Thu, 11 Nov 2010 22:40:01 -0500
parents 5db730bb0e8e
children 6b9673d72a41
rev   line source
1357
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
1 DataLearn: How to plug Datasets & Learner together?
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
2 ===================================================
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
3
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
4 Participants
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
5 ------------
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
6 - Yoshua
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
7 - Razvan
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
8 - Olivier D [leader?]
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
9
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
10 High-Level Objectives
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
11 ---------------------
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
12
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
13 * Simple ML experiments should be simple to write
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
14 * More complex / advanced scenarios should be possible without being forced
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
15 to work "outside" of this framework
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
16 * Computations should be optimized whenever possible
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
17 * Existing code (in any language) should be "wrappable" within this
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
18 framework
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
19 * It should be possible to replace [parts of] this framework with C++ code
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
20
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
21 Theano-Like Data Flow
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
22 ---------------------
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
23
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
24 We want to rely on Theano to be able to take advantage of its efficient
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
25 computations. The general idea is that if we chain multiple processing
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
26 elements (think e.g. of a feature selection step followed by a PCA projection,
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
27 then a rescaling within a fixed bounded interval), the overall transformation
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
28 from input to output data can be represented by a Theano symbolic graph. When
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
29 one wants to access the actual numeric data, a function is compiled so as to
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
30 do these computations efficiently.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
31
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
32 We discussed some specific API options for datasets and learners, which will
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
33 be added to this file in the future, but a core question that we feel should
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
34 be addressed first is how this Theano-based implementation could be achieved
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
35 exactly. For this purpose, in the following, let us assume that a dataset is
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
36 simply a matrix whose rows represent individual samples, and columns
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
37 individual features. How to handle field names, non-tensor-like data, etc. is
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
38 a very important topic that is not yet discussed in this file.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
39
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
40 A question we did not really discuss is whether datasets should be Theano
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
41 Variables. The advantage would be that they would fit directly within the
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
42 Theano framework, which may allow high level optimizations on data
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
43 transformations. However, we would lose the ability to combine Theano
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
44 expressions coded in individual datasets into a single graph. Currently, we
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
45 instead consider that a dataset has a member that is a Theano variable, and
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
46 this variable represents the data stored in the dataset. The same is done for
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
47 individual data samples.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
48
1359
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
49 James asks: Why would a Theano graph in which some nodes represent datasets give
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
50 up the ability to combine Theano expressions coded in individual datasets?
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
51 Firstly, if you want to use Theano expressions and compiled functions to
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
52 implement the perform() method of an Op, you can do that. Secondly, you can
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
53 just include those 'expressions coded in individual datasets' into the overall
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
54 graph.
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
55
1361
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
56 Razvan comments: 1) Having Theano expressions inside the perform of a Theano
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
57 Op can lead to issues. I know I had to deal with a few when implementing
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
58 Scan which does exactly this. Well to be fair these issues mostly come into
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
59 play when the inner graph has to interact with the outer graph and most of
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
60 the time they can be solved. I guess all that I'm saying is going that way
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
61 might lead to some head-ache to developers, though I guess some head-ache
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
62 will be involved no matter what
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
63 2) In my view (I'm not sure this is what Olivier was saying) the idea of
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
64 not putting the Dataset into a Variable is to not put the logic related to
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
65 loading data, dividing it into slices when running it on the GPU and so on
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
66 into a theano variable. In my view this logic goes into a DataSet class
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
67 that gives you shared variables, symbolic indices into that shared
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
68 variables, and also numeric indices. When looping through those numeric
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
69 indices, the dataset class can reload parts of the data into the
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
70 shared variable and so on.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
71
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
72
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
73
1357
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
74 One issue with this approach is illustrated by the following example. Imagine
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
75 we want to iterate on samples in a dataset and do something with their
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
76 numeric value. We would want the code to be as close as possible to:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
77
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
78 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
79
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
80 for sample in dataset:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
81 do_something_with(sample.numeric_value())
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
82
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
83 A naive implementation of the sample API could be (assuming each sample
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
84 contains a ``variable`` member which is the variable representing this
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
85 sample's data):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
86
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
87 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
88
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
89 def numeric_value(self):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
90 if self.function is None:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
91 # Compile function to output the numeric value stored in this
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
92 # sample's variable.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
93 self.function = theano.function([], self.variable)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
94 return self.function()
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
95
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
96 However, this is not a good idea, because it would trigger a new function
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
97 compilation for each sample. Instead, we would want something like this:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
98
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
99 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
100
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
101 def numeric_value(self):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
102 if self.function_storage[0] is None:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
103 # Compile function to output the numeric value stored in this
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
104 # sample's variable. This function takes as input the index of
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
105 # the sample in the dataset, and is shared among all samples.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
106 self.function_storage[0] = theano.function(
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
107 [self.symbolic_index], self.variable)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
108 return self.function(self.numeric_index)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
109
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
110 In the code above, we assume that all samples created by the action of
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
111 iterating over the dataset share the same ``function_storage``,
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
112 ``symbolic_index`` and ``variable``: the first time we try to access the numeric
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
113 value of some sample, a function is compiled, that takes as input the index,
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
114 and outputs the variable. The only difference between samples is thus that
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
115 they are given a different numeric value for the index (``numeric_index``).
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
116
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
117 Another way to obtain the same result is to actually let the user take care of
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
118 compiling the function. It would allow the user to really control what is
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
119 being compiled, at the cost of having to write more code:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
120
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
121 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
122
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
123 symbolic_index = dataset.get_index() # Or just theano.tensor.iscalar()
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
124 get_sample = theano.function([symbolic_index],
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
125 dataset[symbolic_index].variable)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
126 for numeric_index in xrange(len(dataset))
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
127 do_something_with(get_sample(numeric_index))
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
128
1359
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
129 James comments: this is how I have written the last couple of projects, it's
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
130 slightly verbose but it's clear and efficient.
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
131
1361
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
132 <Razvan comments>: I assume that ``do_something_with`` is suppose to be some
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
133 numeric function, and dataset in this case is the result of some
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
134 computations on a initial dataset.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
135 I would differentiate the two approaches (1) and (2) as :
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
136 - first of all whatever you can do with (1) you can do with (2)
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
137 - approach (1) hides the fact that you are working with symbolic graphs.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
138 You apply functions to datasets, and when you want to see values a
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
139 function is compiled under the hood and those values are computed for
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
140 you. In approach (2) the fact that you deal with a symbolic graph is
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
141 explicit because you have to manually compile your functions.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
142 - approach (1) needs to use this function_storage trick shared between
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
143 certain nodes of the graph to reduce the number of compilation while in
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
144 approach (2) we don't need to deal with the complexity of lazy
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
145 compilation
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
146 - approach (1) needs a replace function if you want to change the dataset.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
147 What you would do, is once you have a "computational graph" or pipeline
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
148 or whatever you call it, say ``graph``, to change the input you would do
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
149 graph.replace({ init_data_X: new_data_X}), In approach (2) the init_data_X
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
150 and new_data_X is the ``dataset`` so you would compile two different
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
151 functions. Well I would re-write (2) -- to make the above more clear --
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
152 as :
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
153
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
154 .. code-block:: python
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
155
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
156 symbolic_index = theano.tensor.iscalar()
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
157 get_sample1 = theano.function( [symbolic_index],
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
158 graph( dataset[symbolic_index] ).variable)
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
159 for numeric_index in xrange(len(dataset)):
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
160 do_something_with(get_sample(numeric_index))
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
161
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
162 get_sample2 = theano.function( [symbolic_index],
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
163 graph( new_dataset[symbolic_index] ).variable)
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
164 ## Note: the dataset was replaced with new_dataset
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
165 for numeric_index in xrange(len(new_dataset)):
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
166 do_something_with(get_sample2(numeric_index))
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
167
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
168 ######### FOR (1) you write:
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
169
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
170 for datapoint in graph:
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
171 do_something_with( datapoint() )
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
172
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
173 new_graph = graph.replace({dataset:dataset2})
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
174
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
175 for datapoint in new_graph:
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
176 do_something_with(datapoint())
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
177
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
178 - in approach (1) the initial dataset object (the one that loads the data)
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
179 decides if you will use shared variables and indices to deal with the
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
180 dataset or if you will use ``theano.tensor.matrix`` and not the user( at
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
181 least not without hacking the code). Of course whoever writes that class
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
182 can add a flag to it to switch between behaviours that make sense.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
183 In approach (2) one is not forced to do this
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
184 inside that class by construction, though by convention I would do it.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
185 So if you consider the one who writes that class as a developer than
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
186 in (2) the user can decide/deal with this and not the developer.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
187 Though this is a fine-line -- I would say the user would actually
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
188 write that class as well using some template.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
189 That is to say (2) looks and feels more like working with Theano
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
190 directly,
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
191
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
192 Bottom line, I think (1) puts more stress on the development of the library,
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
193 and hides Theano and some of the complexity for day to day usage.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
194 In (2) everything is a bit more explicit, leaving the impression that you
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
195 have more control over the code, though I strongly feel that whatever can
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
196 be done in (2) can be done in (1). Traditionally I was more inclined
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
197 towards (1) but now I'm not that sure, I think both are equally interesting
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
198 and valid options.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
199 </Razvan comments>
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
200
1357
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
201 Note that although the above example focused on how to iterate over a dataset,
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
202 it can be cast into a more generic problem, where some data (either dataset or
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
203 sample) is the result of some transformation applied to other data, which is
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
204 parameterized by parameters p1, p2, ..., pN (in the above example, we were
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
205 considering a sample that was obtained by taking the p1-th element in a
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
206 dataset). If we use different values for a subset Q of the parameters but keep
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
207 other parameters fixed, we would probably want to compile a single function
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
208 that takes as input all parameters in Q, while other parameters are fixed.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
209 Ideally it would be nice to let the user take control on what is being
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
210 compiled, while leaving the option of using a default sensible behavior for
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
211 those who do not want to worry about it. How to achieve this is still to be
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
212 determined.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
213
1361
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
214 Razvan Comment: I thought about this a bit at the Pylearn level. In my
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
215 original train of thought you would have the distinction between ``hand
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
216 picked parameters`` which I would call hyper-parameter and learned
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
217 parameters. A transformation in this framework (an op if you wish) could
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
218 take as inputs DataSet(s), DataField(s), Parameter(s) (which are the things
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
219 that the learner should adapt) and HyperParameter(s). All hyper-parameters
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
220 will turn into arguments of the compiled function (like the indices of each
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
221 of the dataset objects ) and therefore they can be changed without
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
222 re-compilation. Or in other words this can be easily done by having new
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
223 types of Variables that would represent Parameters and Hyper-parameters.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
224 And as an ending note I would say that there are
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
225 hyper-parameters for which you need to recompile the thenao function and
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
226 can not be just parameters ( so we would have yet another category ?).
1359
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
227
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
228 Another syntactic option for iterating over datasets is
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
229
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
230 .. code-block:: python
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
231
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
232 for sample in dataset.numeric_iterator(batchsize=10):
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
233 do_something_with(sample)
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
234
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
235 The numeric_iterator would create a symbolic batch index, and compile a single function
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
236 that extracts the corresponding minibatch. The arguments to the
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
237 numeric_iterator function can also specify what compile mode to use, any givens
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
238 you might want to apply, etc.
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
239
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
240
1357
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
241 What About Learners?
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
242 --------------------
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
243
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
244 The discussion above only mentioned datasets, but not learners. The learning
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
245 part of a learner is not a main concern (currently). What matters most w.r.t.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
246 what was discussed above is how a learner takes as input a dataset and outputs
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
247 another dataset that can be used with the dataset API.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
248
1359
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
249 James asks:
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
250 What's wrong with simply passing the variables corresponding to the dataset to
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
251 the constructor of the learner?
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
252 That seems much more flexible, compact, and clear than the decorator.
5db730bb0e8e comments on datalearn
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1357
diff changeset
253
1357
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
254 A Learner may be able to compute various things. For instance, a Neural
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
255 Network may output a ``prediction`` vector (whose elements correspond to
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
256 estimated probabilities of each class in a classification task), as well as a
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
257 ``cost`` vector (whose elements correspond to the penalized NLL, the NLL alone
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
258 and the classification error). We would want to be able to build a dataset
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
259 that contains some of these quantities computed on each sample in the input
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
260 dataset.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
261
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
262 The Neural Network code would then look something like this:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
263
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
264 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
265
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
266 class NeuralNetwork(Learner):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
267
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
268 @datalearn(..)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
269 def compute_prediction(self, sample):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
270 return softmax(theano.tensor.dot(self.weights, sample.input))
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
271
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
272 @datalearn(..)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
273 def compute_nll(self, sample):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
274 return - log(self.compute_prediction(sample)[sample.target])
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
275
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
276 @datalearn(..)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
277 def compute_penalized_nll(self, sample):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
278 return (self.compute_nll(self, sample) +
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
279 theano.tensor.sum(self.weights**2))
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
280
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
281 @datalearn(..)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
282 def compute_class_error(self, sample):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
283 probabilities = self.compute_prediction(sample)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
284 predicted_class = theano.tensor.argmax(probabilities)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
285 return predicted_class != sample.target
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
286
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
287 @datalearn(..)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
288 def compute_cost(self, sample):
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
289 return theano.tensor.concatenate([
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
290 self.compute_penalized_nll(sample),
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
291 self.compute_nll(sample),
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
292 self.compute_class_error(sample),
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
293 ])
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
294
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
295 The ``@datalearn`` decorator would be responsible for allowing such a Learner
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
296 to be used e.g. like this:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
297
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
298 .. code-block:: python
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
299
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
300 nnet = NeuralNetwork()
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
301 predict_dataset = nnet.compute_prediction(dataset)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
302 for sample in dataset:
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
303 predict_sample = nnet.compute_prediction(sample)
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
304 predict_numeric = nnet.compute_prediction({'input': numpy.zeros(10)})
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
305 multiple_fields_dataset = ConcatDataSet([
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
306 nnet.compute_prediction(dataset),
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
307 nnet.compute_cost(dataset),
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
308 ])
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
309
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
310 In the code above, if one wants to obtain the numeric value of an element of
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
311 ``multiple_fields_dataset``, the Theano function being compiled would be able
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
312 to optimize computations so that the simultaneous computation of
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
313 ``prediction`` and ``cost`` is done efficiently.
ffa2932a8cba Added datalearn committee discussion file
Olivier Delalleau <delallea@iro>
parents:
diff changeset
314
1361
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
315 Razvan asks: What is predict_sample for ? What is predict_dataset? What I
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
316 guess you mean is that the decorator is used to convert a function that
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
317 takes a theano variable and outputs a theano variable into a class/function
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
318 that takes a DataField/DataSet and outputs a DataField/DataSet. It could
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
319 also register all those different functions, so that the Dataset that
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
320 you get out of (not one of the function) the entire Learner (this Dataset
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
321 is returned by __call__) would contain all those as fields.
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
322 I would use it like this:
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
323
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
324 .. code-block:: python
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
325
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
326 nnet = NeuralNetwork()
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
327 results = nnet(dataset)
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
328 for datapoint in results:
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
329 print datapoint.prediction, datapoint.nll, ...
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
330
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
331 Is this close to what you are suggesting?
7548dc1b163c Some question/suggestions to datalearn
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1359
diff changeset
332