Mercurial > pylearn

annotate doc/v2_planning/architecture.txt @ 1369:f3a549bd8688

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
datalearn: Added another comment on James' numeric iterator function
author Olivier Delalleau <delallea@iro>
date Mon, 15 Nov 2010 15:20:49 -0500
parents b9d0a326e3e7
children
rev   line source
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 Pylearn Architecture
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
6 SE + VM Approach
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
7 =================
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
9 One avenue for the basic design of the library is to follow the Symbolic
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
10 Expression (SE) structure + virtual machine (VM) pattern that worked for Theano.
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
12 The main things for the library to provide would be:
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 - a few VMs, some of which can run programs in parallel across processors,
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 hosts, and networks [R6,R8];
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 - MLA components as either individual Expressions (similar to Ops) or as
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 subgraphs of SEs [R5,R7,R10,R11]
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 - machine learning algorithms including their training and testing in the form
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 of python functions that build SE graphs.[R1,R8].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 This design addresses R2 (modularity) because swapping components is literally implemented by
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 swapping subgraphs.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 The design addresses R9 (algorithmic efficiency) because we can write
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 Theano-style graph transformations to recognize special cases of component
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 combinations.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 The design addresses R3 if we make the additional decision that the VMs (at
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 least sometimes) cache the return value of program function calls. This cache
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 serves as a database of experimental results, indexed by the functions that
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 originally computed them. I think this is a very natural scheme for organizing
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 experiment results, and ensuring experiment reproducibility [R1].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 At the same time, this is a clean and simple API behind which experiments can be
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 saved using a number of database technologies.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 APIs vs. lambda
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 ----------------
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 Modularity in general is achieved when pieces can be substituted one for the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 other.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 In an object-oriented design, modularity is achieved by agreeing on interface
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 APIs, but in a functional design there is another possibility: the lambda.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 In an SE these pieces are expression [applications] and the subgraphs they form.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 A subgraph is characterized syntactically within the program by its arguments
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 and its return values. A lambda function allows the User to create new
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 Expression types from arbitrary subgraphs with very few keystrokes. When a
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 lambda is available and easy to use, there is much less pressure on the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 expression library to follow calling and return conventions strictly.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 Of course, the closer are two subgraphs in terms of their inputs, outputs, and
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 semantics, the easier it is to substitute one for the other. As library
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 designers, we should still aim for compatibility of similar algorithms. It's
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 just not essential to choose an API that will guarantee a match, or indeed to
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 choose any explicit API at all.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59
1125
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
60 YB: I agree that lambdas are more flexible, but from the user's point of
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
61 view it is really important to know what can swap with what, so that they
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
62 can easily plug-and-play. So even if informal, something in the spirit
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
63 of an API must be described somewhere, and components should declare
1126
7ba1854f03e7 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1125
diff changeset
64 either formally or through comments what functionality 'type'
7ba1854f03e7 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1125
diff changeset
65 they can take on.
1125
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
66
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
67 Encapsulation vs. linearity
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
68 ---------------------------
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
70 A while ago, the Apstat crew went to fight "encapsulation" to propose instead
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
71 a more "linearized" approach to experiment design. I must admit I didn't
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
72 really understand the deep motivations behind this, and after practicing both
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
73 styles (encapsulation for PLearn / Theano, linearity @ ARL / Ubisoft), I still
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
74 don't. I do find, however, some not-so-deep-but-still-significant advantages
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
75 to the linear version, which hopefully can be made clear (along with a
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
76 clarification of what the h*** am I talking about) in the following example:
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
77
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
78 * Linear version:
1190
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
79
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
80 .. code-block:: python
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
81
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
82 my_experiment = pipeline([
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
83 data,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
84 filter_samples,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
85 PCA,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
86 k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
87 neural_net,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
88 evaluation,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
89 ])
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
90
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
91 * Encapsulated version:
1190
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
92
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
93 .. code-block:: python
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
94
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
95 my_experiment = evaluation(
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
96 data=PCA(filter_samples(data)),
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
97 split=k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
98 model=neural_net)
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
99
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
100 What I like in the linear version is it is much more easily human-readable
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
101 (once you know what it means): you just follow the flow of the experiment by
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
102 reading through a single list.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
103 On the other hand, the encapsulated version requires some deeper analysis to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
104 understand what is going on and in which order.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
105 Also, commenting out parts of the processing is simpler in the first case (it
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
106 takes a single # in front of an element).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
107 However, linearity tends to break when the experiment is actually not linear,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
108 i.e. the graph of object dependencies is more complex (*).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
109
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
110 I'm just bringing this up because it may be nice to be able to provide the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
111 user with the most intuitive way to design experiments. I actually don't think
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
112 those approaches are mutually exclusive, and it could be possible for the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
113 underlying system to use the more flexible / powerful encapsulated
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
114 representation, while having the option to write simple scripts in a form that
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
115 is easier to understand and manipulate.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
116
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
117 It could also be worth discussing this issue with Xavier / Christian /
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
118 Nicolas.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
119
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
120 (*) Note that I cheated a bit in my example above: the graph from the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
121 encapsulated version is not a simple chain, so it is not obvious how to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
122 convert it into the pipeline given in the linear version. It's still possible
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
123 though, but this is probably not the place to get into the details.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
124
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
125 RP comment : The way I see it, you could always have everything using the
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
126 encapsulation paradigm ( which as you pointed out is a bit more powerful) and
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
127 then have linear shortcuts ( functions that take a list of functions and some
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
128 inputs and apply them in some order). You will not be able to have a one case
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
129 cover all pipeline function, but I think it is sufficient to offer such
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
130 options (linear functions) for a few widely used cases ..
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
131
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
132
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
133 Jobman Compatibility Approach
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
134 =============================
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
135
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
136 One basic approach for the library is to provide a set of components that are
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
137 compatible with remote execution. The emphasis could be not so much on
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
138 standardizing the roles and APIs of components, so much as ensuring that they
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
139 can be glued together and supports parallel execution on one or more CPUs or
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
140 clusters.
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
141
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
142 In this approach we would provide a proxy for asynchronous execution
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
143 (e.g. "pylearn.call(fn, args, kwargs, backend=default_backend)"), which would
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
144 come with constraints on what fn, args, and kwargs can be. Specifically, they
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
145 must be picklable, and there are benefits (e.g. automatic function call caching)
1129
40c1461ce9ef architecture: Minor typo fix
Olivier Delalleau <delallea@iro>
parents: 1126
diff changeset
146 associated with them being hashable as well.
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
147
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
148
1186
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
149 Benchmark
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
150 =========
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
151
1186
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
152 During the general meeting on sept. 17th, we agreed to produce at least pseudo-code (if
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
153 possible, actual code) for the following model:
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
154 A Deep Belief Net (with greedy layerwise pre-training, and supervised
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
155 fine-tuning), with preprocessing of the data, double cross-validation, and
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
156 save/load of the model.
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
157
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
158 The different approach to be tested are:
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
159 - Plugins with a global scheduler driving the experiment (Razvan's team)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
160 - Objects, with basic hooks at predefined places (Pascal L.'s team)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
161 - Existing objects and code (including dbi and Jobman), with some more
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
162 pieces to tie things together (Fred B.)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
163
1201
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
164 OD comments: We were in a hurry to close the meeting and I did not have time
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
165 to really explain what I meant when I suggested we should add the requirement
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
166 of saving the final "best" model. What I had in mind is a typical "applied ML"
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
167 experiment, i.e. the following approach that hopefully can be understood just
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
168 by writing it down in the form of a processing pipeline. The double cross
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
169 validation step, whose goal is to obtain an estimate of the generalization
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
170 error of our final model, is:
1248
b9d0a326e3e7 architecture: Removed duplicated 'evaluate' statement in my pipeline sketch
Olivier Delalleau <delallea@iro>
parents: 1201
diff changeset
171 data -> k_fold_outer(preprocessing -> k_fold_inner(dbn -> evaluate) -> select_best -> retrain_on_all_data -> evaluate)
1201
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
172 Once this is done, the model we want to save is obtained by doing
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
173 data -> preprocessing -> k_fold(dbn -> evaluate) -> select_best -> retrain_on_all_data
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
174 and we save
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
175 preprocessing -> best_model_selected