Mercurial > pylearn

annotate doc/v2_planning/architecture.txt @ 1390:746ebceeb46f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added comments to hmc code (old outstanding changes)
author gdesjardins
date Mon, 20 Dec 2010 18:08:04 -0500
parents b9d0a326e3e7
children
rev   line source
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 Pylearn Architecture
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
6 SE + VM Approach
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
7 =================
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
9 One avenue for the basic design of the library is to follow the Symbolic
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
10 Expression (SE) structure + virtual machine (VM) pattern that worked for Theano.
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
12 The main things for the library to provide would be:
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 - a few VMs, some of which can run programs in parallel across processors,
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 hosts, and networks [R6,R8];
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 - MLA components as either individual Expressions (similar to Ops) or as
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 subgraphs of SEs [R5,R7,R10,R11]
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 - machine learning algorithms including their training and testing in the form
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 of python functions that build SE graphs.[R1,R8].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 This design addresses R2 (modularity) because swapping components is literally implemented by
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 swapping subgraphs.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 The design addresses R9 (algorithmic efficiency) because we can write
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 Theano-style graph transformations to recognize special cases of component
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 combinations.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 The design addresses R3 if we make the additional decision that the VMs (at
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 least sometimes) cache the return value of program function calls. This cache
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 serves as a database of experimental results, indexed by the functions that
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 originally computed them. I think this is a very natural scheme for organizing
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 experiment results, and ensuring experiment reproducibility [R1].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 At the same time, this is a clean and simple API behind which experiments can be
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 saved using a number of database technologies.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 APIs vs. lambda
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 ----------------
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 Modularity in general is achieved when pieces can be substituted one for the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 other.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 In an object-oriented design, modularity is achieved by agreeing on interface
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 APIs, but in a functional design there is another possibility: the lambda.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 In an SE these pieces are expression [applications] and the subgraphs they form.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 A subgraph is characterized syntactically within the program by its arguments
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 and its return values. A lambda function allows the User to create new
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 Expression types from arbitrary subgraphs with very few keystrokes. When a
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 lambda is available and easy to use, there is much less pressure on the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 expression library to follow calling and return conventions strictly.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 Of course, the closer are two subgraphs in terms of their inputs, outputs, and
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 semantics, the easier it is to substitute one for the other. As library
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 designers, we should still aim for compatibility of similar algorithms. It's
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 just not essential to choose an API that will guarantee a match, or indeed to
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 choose any explicit API at all.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59
1125
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
60 YB: I agree that lambdas are more flexible, but from the user's point of
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
61 view it is really important to know what can swap with what, so that they
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
62 can easily plug-and-play. So even if informal, something in the spirit
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
63 of an API must be described somewhere, and components should declare
1126
7ba1854f03e7 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1125
diff changeset
64 either formally or through comments what functionality 'type'
7ba1854f03e7 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1125
diff changeset
65 they can take on.
1125
5387666d49b4 YB: comment on lambdas vs API for architecture.txt
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 1115
diff changeset
66
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
67 Encapsulation vs. linearity
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
68 ---------------------------
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
70 A while ago, the Apstat crew went to fight "encapsulation" to propose instead
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
71 a more "linearized" approach to experiment design. I must admit I didn't
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
72 really understand the deep motivations behind this, and after practicing both
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
73 styles (encapsulation for PLearn / Theano, linearity @ ARL / Ubisoft), I still
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
74 don't. I do find, however, some not-so-deep-but-still-significant advantages
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
75 to the linear version, which hopefully can be made clear (along with a
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
76 clarification of what the h*** am I talking about) in the following example:
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
77
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
78 * Linear version:
1190
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
79
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
80 .. code-block:: python
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
81
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
82 my_experiment = pipeline([
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
83 data,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
84 filter_samples,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
85 PCA,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
86 k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
87 neural_net,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
88 evaluation,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
89 ])
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
90
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
91 * Encapsulated version:
1190
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
92
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
93 .. code-block:: python
9ff2242a817b fix rst syntax errors/warnings
Frederic Bastien <nouiz@nouiz.org>
parents: 1186
diff changeset
94
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
95 my_experiment = evaluation(
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
96 data=PCA(filter_samples(data)),
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
97 split=k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
98 model=neural_net)
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
99
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
100 What I like in the linear version is it is much more easily human-readable
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
101 (once you know what it means): you just follow the flow of the experiment by
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
102 reading through a single list.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
103 On the other hand, the encapsulated version requires some deeper analysis to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
104 understand what is going on and in which order.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
105 Also, commenting out parts of the processing is simpler in the first case (it
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
106 takes a single # in front of an element).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
107 However, linearity tends to break when the experiment is actually not linear,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
108 i.e. the graph of object dependencies is more complex (*).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
109
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
110 I'm just bringing this up because it may be nice to be able to provide the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
111 user with the most intuitive way to design experiments. I actually don't think
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
112 those approaches are mutually exclusive, and it could be possible for the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
113 underlying system to use the more flexible / powerful encapsulated
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
114 representation, while having the option to write simple scripts in a form that
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
115 is easier to understand and manipulate.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
116
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
117 It could also be worth discussing this issue with Xavier / Christian /
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
118 Nicolas.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
119
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
120 (*) Note that I cheated a bit in my example above: the graph from the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
121 encapsulated version is not a simple chain, so it is not obvious how to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
122 convert it into the pipeline given in the linear version. It's still possible
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
123 though, but this is probably not the place to get into the details.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
124
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
125 RP comment : The way I see it, you could always have everything using the
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
126 encapsulation paradigm ( which as you pointed out is a bit more powerful) and
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
127 then have linear shortcuts ( functions that take a list of functions and some
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
128 inputs and apply them in some order). You will not be able to have a one case
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
129 cover all pipeline function, but I think it is sufficient to offer such
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1129
diff changeset
130 options (linear functions) for a few widely used cases ..
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
131
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
132
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
133 Jobman Compatibility Approach
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
134 =============================
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
135
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
136 One basic approach for the library is to provide a set of components that are
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
137 compatible with remote execution. The emphasis could be not so much on
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
138 standardizing the roles and APIs of components, so much as ensuring that they
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
139 can be glued together and supports parallel execution on one or more CPUs or
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
140 clusters.
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
141
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
142 In this approach we would provide a proxy for asynchronous execution
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
143 (e.g. "pylearn.call(fn, args, kwargs, backend=default_backend)"), which would
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
144 come with constraints on what fn, args, and kwargs can be. Specifically, they
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
145 must be picklable, and there are benefits (e.g. automatic function call caching)
1129
40c1461ce9ef architecture: Minor typo fix
Olivier Delalleau <delallea@iro>
parents: 1126
diff changeset
146 associated with them being hashable as well.
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
147
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
148
1186
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
149 Benchmark
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
150 =========
1115
967975f9c574 added Jobman compatibility to architecture.txt
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1107
diff changeset
151
1186
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
152 During the general meeting on sept. 17th, we agreed to produce at least pseudo-code (if
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
153 possible, actual code) for the following model:
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
154 A Deep Belief Net (with greedy layerwise pre-training, and supervised
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
155 fine-tuning), with preprocessing of the data, double cross-validation, and
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
156 save/load of the model.
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
157
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
158 The different approach to be tested are:
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
159 - Plugins with a global scheduler driving the experiment (Razvan's team)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
160 - Objects, with basic hooks at predefined places (Pascal L.'s team)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
161 - Existing objects and code (including dbi and Jobman), with some more
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
162 pieces to tie things together (Fred B.)
f111f8c2a280 Update after sept. 17th meeting
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1153
diff changeset
163
1201
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
164 OD comments: We were in a hurry to close the meeting and I did not have time
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
165 to really explain what I meant when I suggested we should add the requirement
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
166 of saving the final "best" model. What I had in mind is a typical "applied ML"
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
167 experiment, i.e. the following approach that hopefully can be understood just
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
168 by writing it down in the form of a processing pipeline. The double cross
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
169 validation step, whose goal is to obtain an estimate of the generalization
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
170 error of our final model, is:
1248
b9d0a326e3e7 architecture: Removed duplicated 'evaluate' statement in my pipeline sketch
Olivier Delalleau <delallea@iro>
parents: 1201
diff changeset
171 data -> k_fold_outer(preprocessing -> k_fold_inner(dbn -> evaluate) -> select_best -> retrain_on_all_data -> evaluate)
1201
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
172 Once this is done, the model we want to save is obtained by doing
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
173 data -> preprocessing -> k_fold(dbn -> evaluate) -> select_best -> retrain_on_all_data
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
174 and we save
46527ae6db53 architecture: Clarified what I meant about saving the model
Olivier Delalleau <delallea@iro>
parents: 1190
diff changeset
175 preprocessing -> best_model_selected