Mercurial > pylearn

annotate doc/v2_planning/architecture.txt @ 1107:e5306f5626d4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
architecture: Yet another rant, this time about encapsulation vs. linearization
author Olivier Delalleau <delallea@iro>
date Mon, 13 Sep 2010 22:45:10 -0400
parents 4eda3f52ebef
children 967975f9c574
rev   line source
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2 Pylearn Architecture
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 ====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 Basic Design Approach
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 =====================
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 I propose that the basic design of the library follow the Symbolic Expression
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 (SE) structure + virtual machine (VM) pattern that worked for Theano.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 So the main things for the library to provide would be:
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 - a few VMs, some of which can run programs in parallel across processors,
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 hosts, and networks [R6,R8];
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 - MLA components as either individual Expressions (similar to Ops) or as
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 subgraphs of SEs [R5,R7,R10,R11]
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 - machine learning algorithms including their training and testing in the form
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 of python functions that build SE graphs.[R1,R8].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 This design addresses R2 (modularity) because swapping components is literally implemented by
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 swapping subgraphs.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 The design addresses R9 (algorithmic efficiency) because we can write
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 Theano-style graph transformations to recognize special cases of component
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 combinations.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 The design addresses R3 if we make the additional decision that the VMs (at
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 least sometimes) cache the return value of program function calls. This cache
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 serves as a database of experimental results, indexed by the functions that
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 originally computed them. I think this is a very natural scheme for organizing
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 experiment results, and ensuring experiment reproducibility [R1].
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 At the same time, this is a clean and simple API behind which experiments can be
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36 saved using a number of database technologies.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 APIs vs. lambda
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 ----------------
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 Modularity in general is achieved when pieces can be substituted one for the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 other.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 In an object-oriented design, modularity is achieved by agreeing on interface
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 APIs, but in a functional design there is another possibility: the lambda.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 In an SE these pieces are expression [applications] and the subgraphs they form.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48 A subgraph is characterized syntactically within the program by its arguments
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 and its return values. A lambda function allows the User to create new
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 Expression types from arbitrary subgraphs with very few keystrokes. When a
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 lambda is available and easy to use, there is much less pressure on the
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 expression library to follow calling and return conventions strictly.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 Of course, the closer are two subgraphs in terms of their inputs, outputs, and
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 semantics, the easier it is to substitute one for the other. As library
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 designers, we should still aim for compatibility of similar algorithms. It's
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 just not essential to choose an API that will guarantee a match, or indeed to
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 choose any explicit API at all.
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
60 Encapsulation vs. linearity
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
61 ---------------------------
1098
4eda3f52ebef v2planning - revs to requirements, added architecture
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62
1107
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
63 A while ago, the Apstat crew went to fight "encapsulation" to propose instead
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
64 a more "linearized" approach to experiment design. I must admit I didn't
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
65 really understand the deep motivations behind this, and after practicing both
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
66 styles (encapsulation for PLearn / Theano, linearity @ ARL / Ubisoft), I still
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
67 don't. I do find, however, some not-so-deep-but-still-significant advantages
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
68 to the linear version, which hopefully can be made clear (along with a
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
69 clarification of what the h*** am I talking about) in the following example:
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
70
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
71 * Linear version:
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
72 my_experiment = pipeline([
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
73 data,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
74 filter_samples,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
75 PCA,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
76 k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
77 neural_net,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
78 evaluation,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
79 ])
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
80
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
81 * Encapsulated version:
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
82 my_experiment = evaluation(
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
83 data=PCA(filter_samples(data)),
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
84 split=k_fold_split,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
85 model=neural_net)
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
86
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
87 What I like in the linear version is it is much more easily human-readable
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
88 (once you know what it means): you just follow the flow of the experiment by
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
89 reading through a single list.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
90 On the other hand, the encapsulated version requires some deeper analysis to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
91 understand what is going on and in which order.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
92 Also, commenting out parts of the processing is simpler in the first case (it
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
93 takes a single # in front of an element).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
94 However, linearity tends to break when the experiment is actually not linear,
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
95 i.e. the graph of object dependencies is more complex (*).
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
96
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
97 I'm just bringing this up because it may be nice to be able to provide the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
98 user with the most intuitive way to design experiments. I actually don't think
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
99 those approaches are mutually exclusive, and it could be possible for the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
100 underlying system to use the more flexible / powerful encapsulated
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
101 representation, while having the option to write simple scripts in a form that
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
102 is easier to understand and manipulate.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
103
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
104 It could also be worth discussing this issue with Xavier / Christian /
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
105 Nicolas.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
106
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
107 (*) Note that I cheated a bit in my example above: the graph from the
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
108 encapsulated version is not a simple chain, so it is not obvious how to
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
109 convert it into the pipeline given in the linear version. It's still possible
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
110 though, but this is probably not the place to get into the details.
e5306f5626d4 architecture: Yet another rant, this time about encapsulation vs. linearization
Olivier Delalleau <delallea@iro>
parents: 1098
diff changeset
111