Mercurial > pylearn

annotate doc/v2_planning/architecture_discussion.txt @ 1260:a565c20a39d7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
general file to talk about the different approaches
author Razvan Pascanu <r.pascanu@gmail.com>
date Sun, 26 Sep 2010 14:10:33 -0400
parents
children 93e1c7c9172b
rev   line source
1260
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 Arnaud:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
3 From what I recall for the meeting last Friday, we saw three
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
4 propositions for a runtime architecture for the experiments in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
5 pylearn.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
6
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
7 The thing I noticed was that none of the three propositions was
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
8 addressing the same problem. So not only do we have to choose which
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
9 one(s) we want, but we also have to decide upon what do we need.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
10
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
11 The proposals and the problems they address are outlined below, please
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
12 comment if you see inaccuracies:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
13
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
14 - PL's proposal, the hooks thing, was about enabling hooks to be
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
15 registered at predefined points in functions and giving them access to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16 the local variables. This addresses nicely the problem of collecting
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
17 stats and printing progress.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
18
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
19 - OB's proposal, the checkpoints thing, was about enabling the saving
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
20 and loading of state at predefined points in the function. Other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
21 actions could also be performed at these points.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
22
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
23 - JB's proposal, the new language thing, was about expressing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24 algorithms with a control structure made of classes so that its state
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
25 and structure could be preserved. It could also define new control
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
26 structure to run things in parallel, over multiple machines or not.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
27
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
28 Razvan:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
29
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
30 I would add the following observations:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
31
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
32 #1
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
33 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
34
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
35 This might be an artificial created issue, but I will write it down anyhow.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 We can decide later if we care about it.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38 Imagine you have some function provided by the library that implements
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
39 some (complicated) pattern. Let say deeplearning ( the pretraining followed
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
40 by finetuning). You instantiate this somehow :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
41
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
42 instance = deeplearning(..)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
43
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
44 Now you want to add some function to a given hook, checkpoint or whatever
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
45 to calculate some statistics. You of course can do that ( the documentation
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
46 can tell you how those hooks are named), but what the function will get is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
47 the locals defined in deeplearning. So you need to open up the file that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
48 implements that deeplearning and understand the code to figure out what
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
49 variable does what.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
50
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
51 Secondly if you need to execute a function in a unforseen place by the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
52 deeplearning,you can only do that by hacking the file implementing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
53 deeplearning function, i.e. by hacking the library. One can make sure that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
54 does not happen by overpopulating the code with hooks, but then we need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
55 a name for each hook.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
56
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
57 I can add that probably in most cases the logic that goes into this is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
58 simple enough that the issues above are insignificant, but I might be wrong.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
59
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
60
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
61 #2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
62 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
63
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
64 I think it is much healthier to think of James proposal as a glorified
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
65 pipeline and not as a new language. You have components that you add in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
66 you pipeline. A CALL is such a component. You run the program by executing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
67 the pipeline ( which goes from one component to the other and calls it)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
68
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
69 We are dealing with a glorified pipeline because :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
70 - when running the pipeline you can loop over a certain segment of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
71 pipeline if you need to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
72 - you can, at run time, swtich between two possible terminations of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73 pipeline (the if command)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
74 - you can have two pipelines running in paralel, by running one
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
75 component from one pipeline and then going to the other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
76
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
77 You can also think of what James proposes as sort of the same as
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
78 Olivier's with the following differences:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
79 - Olivier makes this entire mechanism invisible to the eye while in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
80 James' case it is explicit
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
81 - James has inplicit checkpoints between any component, in Olivier's
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
82 case you can define pipelines at different points ( maybe even more
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
83 finely grained that what James mechanism offers)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
84 - One can imagine how, though Olivier did not exactly explained
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
85 how you could have hooks in a template such that you do not actually need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
86 to hack that code.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
87
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
88 James proposal also offers a way of expressing the distributed part in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
89 your main program. Is the same as having two pipelines between which you
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
90 switch. Just think now each pipeline runs on a different machine
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
91 independently and you just wait as the server for them to return. This
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
92 is just one possibility.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
93
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
94 In this proposal you can also see how you would solve the unforseen hooks
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
95 problem, by having a special function that could alter the pipeline in some
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
96 way (for example by introducing new components).
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
97