Mercurial > pylearn

annotate doc/v2_planning/architecture_discussion.txt @ 1328:63fe96ede21d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added function doc and made the interface more constant between fct.
author Frederic Bastien <nouiz@nouiz.org>
date Thu, 14 Oct 2010 13:33:33 -0400
parents 93e1c7c9172b
children
rev   line source
1260
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 Arnaud:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
3 From what I recall for the meeting last Friday, we saw three
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
4 propositions for a runtime architecture for the experiments in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
5 pylearn.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
6
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
7 The thing I noticed was that none of the three propositions was
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
8 addressing the same problem. So not only do we have to choose which
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
9 one(s) we want, but we also have to decide upon what do we need.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
10
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
11 The proposals and the problems they address are outlined below, please
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
12 comment if you see inaccuracies:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
13
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
14 - PL's proposal, the hooks thing, was about enabling hooks to be
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
15 registered at predefined points in functions and giving them access to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16 the local variables. This addresses nicely the problem of collecting
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
17 stats and printing progress.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
18
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
19 - OB's proposal, the checkpoints thing, was about enabling the saving
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
20 and loading of state at predefined points in the function. Other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
21 actions could also be performed at these points.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
22
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
23 - JB's proposal, the new language thing, was about expressing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24 algorithms with a control structure made of classes so that its state
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
25 and structure could be preserved. It could also define new control
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
26 structure to run things in parallel, over multiple machines or not.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
27
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
28 Razvan:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
29
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
30 I would add the following observations:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
31
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
32 #1
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
33 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
34
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
35 This might be an artificial created issue, but I will write it down anyhow.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 We can decide later if we care about it.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38 Imagine you have some function provided by the library that implements
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
39 some (complicated) pattern. Let say deeplearning ( the pretraining followed
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
40 by finetuning). You instantiate this somehow :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
41
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
42 instance = deeplearning(..)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
43
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
44 Now you want to add some function to a given hook, checkpoint or whatever
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
45 to calculate some statistics. You of course can do that ( the documentation
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
46 can tell you how those hooks are named), but what the function will get is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
47 the locals defined in deeplearning. So you need to open up the file that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
48 implements that deeplearning and understand the code to figure out what
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
49 variable does what.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
50
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
51 Secondly if you need to execute a function in a unforseen place by the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
52 deeplearning,you can only do that by hacking the file implementing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
53 deeplearning function, i.e. by hacking the library. One can make sure that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
54 does not happen by overpopulating the code with hooks, but then we need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
55 a name for each hook.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
56
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
57 I can add that probably in most cases the logic that goes into this is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
58 simple enough that the issues above are insignificant, but I might be wrong.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
59
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
60
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
61 #2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
62 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
63
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
64 I think it is much healthier to think of James proposal as a glorified
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
65 pipeline and not as a new language. You have components that you add in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
66 you pipeline. A CALL is such a component. You run the program by executing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
67 the pipeline ( which goes from one component to the other and calls it)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
68
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
69 We are dealing with a glorified pipeline because :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
70 - when running the pipeline you can loop over a certain segment of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
71 pipeline if you need to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
72 - you can, at run time, swtich between two possible terminations of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73 pipeline (the if command)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
74 - you can have two pipelines running in paralel, by running one
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
75 component from one pipeline and then going to the other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
76
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
77 You can also think of what James proposes as sort of the same as
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
78 Olivier's with the following differences:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
79 - Olivier makes this entire mechanism invisible to the eye while in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
80 James' case it is explicit
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
81 - James has inplicit checkpoints between any component, in Olivier's
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
82 case you can define pipelines at different points ( maybe even more
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
83 finely grained that what James mechanism offers)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
84 - One can imagine how, though Olivier did not exactly explained
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
85 how you could have hooks in a template such that you do not actually need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
86 to hack that code.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
87
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
88 James proposal also offers a way of expressing the distributed part in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
89 your main program. Is the same as having two pipelines between which you
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
90 switch. Just think now each pipeline runs on a different machine
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
91 independently and you just wait as the server for them to return. This
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
92 is just one possibility.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
93
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
94 In this proposal you can also see how you would solve the unforseen hooks
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
95 problem, by having a special function that could alter the pipeline in some
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
96 way (for example by introducing new components).
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
97
1261
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
98 OD comments: It seemed to me that one major issue we are trying to solve with
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
99 these approaches is that of being able to interrupt an experiment, then
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
100 restart it later without starting again from scratch. OB and JB's proposals
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
101 handle this more or less automatically (compared to PL's that would require
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
102 more manual engineering of the save/load process). However it is not obvious
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
103 to me that they would necessarily make things much easier, because:
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
104 - One needs to use the same "framework" in all pieces of code (the +
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
105 syntax for OB, or a single program for JB), otherwise some manual
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
106 engineering will also be required. Can we reasonably expect the whole
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
107 code to adhere to this? (maybe...)
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
108 - If you want to be smart about what you (or rather do not) want to save,
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
109 it may add yet another layer of complexity (I'm not sure though how hard
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
110 it would be, so it'd be nice to have an example, e.g. if you are doing
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
111 K-Fold CV with the training set stored in memory, but you don't want to
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
112 save it on disk when serializing your experiment).