Mercurial > pylearn

annotate doc/v2_planning/architecture_discussion.txt @ 1321:ebcb76b38817

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tinyimages - added main script to whiten patches
author James Bergstra <bergstrj@iro.umontreal.ca>
date Sun, 10 Oct 2010 13:43:53 -0400
parents 93e1c7c9172b
children
rev   line source
1260
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 Arnaud:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
3 From what I recall for the meeting last Friday, we saw three
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
4 propositions for a runtime architecture for the experiments in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
5 pylearn.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
6
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
7 The thing I noticed was that none of the three propositions was
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
8 addressing the same problem. So not only do we have to choose which
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
9 one(s) we want, but we also have to decide upon what do we need.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
10
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
11 The proposals and the problems they address are outlined below, please
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
12 comment if you see inaccuracies:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
13
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
14 - PL's proposal, the hooks thing, was about enabling hooks to be
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
15 registered at predefined points in functions and giving them access to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16 the local variables. This addresses nicely the problem of collecting
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
17 stats and printing progress.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
18
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
19 - OB's proposal, the checkpoints thing, was about enabling the saving
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
20 and loading of state at predefined points in the function. Other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
21 actions could also be performed at these points.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
22
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
23 - JB's proposal, the new language thing, was about expressing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24 algorithms with a control structure made of classes so that its state
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
25 and structure could be preserved. It could also define new control
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
26 structure to run things in parallel, over multiple machines or not.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
27
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
28 Razvan:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
29
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
30 I would add the following observations:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
31
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
32 #1
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
33 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
34
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
35 This might be an artificial created issue, but I will write it down anyhow.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 We can decide later if we care about it.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38 Imagine you have some function provided by the library that implements
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
39 some (complicated) pattern. Let say deeplearning ( the pretraining followed
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
40 by finetuning). You instantiate this somehow :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
41
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
42 instance = deeplearning(..)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
43
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
44 Now you want to add some function to a given hook, checkpoint or whatever
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
45 to calculate some statistics. You of course can do that ( the documentation
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
46 can tell you how those hooks are named), but what the function will get is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
47 the locals defined in deeplearning. So you need to open up the file that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
48 implements that deeplearning and understand the code to figure out what
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
49 variable does what.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
50
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
51 Secondly if you need to execute a function in a unforseen place by the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
52 deeplearning,you can only do that by hacking the file implementing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
53 deeplearning function, i.e. by hacking the library. One can make sure that
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
54 does not happen by overpopulating the code with hooks, but then we need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
55 a name for each hook.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
56
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
57 I can add that probably in most cases the logic that goes into this is
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
58 simple enough that the issues above are insignificant, but I might be wrong.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
59
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
60
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
61 #2
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
62 ---
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
63
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
64 I think it is much healthier to think of James proposal as a glorified
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
65 pipeline and not as a new language. You have components that you add in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
66 you pipeline. A CALL is such a component. You run the program by executing
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
67 the pipeline ( which goes from one component to the other and calls it)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
68
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
69 We are dealing with a glorified pipeline because :
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
70 - when running the pipeline you can loop over a certain segment of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
71 pipeline if you need to
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
72 - you can, at run time, swtich between two possible terminations of the
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73 pipeline (the if command)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
74 - you can have two pipelines running in paralel, by running one
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
75 component from one pipeline and then going to the other
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
76
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
77 You can also think of what James proposes as sort of the same as
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
78 Olivier's with the following differences:
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
79 - Olivier makes this entire mechanism invisible to the eye while in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
80 James' case it is explicit
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
81 - James has inplicit checkpoints between any component, in Olivier's
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
82 case you can define pipelines at different points ( maybe even more
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
83 finely grained that what James mechanism offers)
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
84 - One can imagine how, though Olivier did not exactly explained
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
85 how you could have hooks in a template such that you do not actually need
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
86 to hack that code.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
87
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
88 James proposal also offers a way of expressing the distributed part in
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
89 your main program. Is the same as having two pipelines between which you
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
90 switch. Just think now each pipeline runs on a different machine
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
91 independently and you just wait as the server for them to return. This
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
92 is just one possibility.
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
93
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
94 In this proposal you can also see how you would solve the unforseen hooks
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
95 problem, by having a special function that could alter the pipeline in some
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
96 way (for example by introducing new components).
a565c20a39d7 general file to talk about the different approaches
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
97
1261
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
98 OD comments: It seemed to me that one major issue we are trying to solve with
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
99 these approaches is that of being able to interrupt an experiment, then
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
100 restart it later without starting again from scratch. OB and JB's proposals
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
101 handle this more or less automatically (compared to PL's that would require
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
102 more manual engineering of the save/load process). However it is not obvious
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
103 to me that they would necessarily make things much easier, because:
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
104 - One needs to use the same "framework" in all pieces of code (the +
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
105 syntax for OB, or a single program for JB), otherwise some manual
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
106 engineering will also be required. Can we reasonably expect the whole
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
107 code to adhere to this? (maybe...)
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
108 - If you want to be smart about what you (or rather do not) want to save,
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
109 it may add yet another layer of complexity (I'm not sure though how hard
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
110 it would be, so it'd be nice to have an example, e.g. if you are doing
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
111 K-Fold CV with the training set stored in memory, but you don't want to
93e1c7c9172b Added a comment on architecture discussion
Olivier Delalleau <delallea@iro>
parents: 1260
diff changeset
112 save it on disk when serializing your experiment).