Mercurial > pylearn
annotate doc/v2_planning/plugin_RP.py @ 1183:bc1b445e22fa
API_coding_style: Added code example to explain the point about the number of spaces after a period
| author | Olivier Delalleau <delallea@iro> |
|---|---|
| date | Fri, 17 Sep 2010 16:51:09 -0400 |
| parents | fe6c25eb1e37 |
| children | 7fff3d5c7694 |
| rev | line source |
|---|---|
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
1 ''' |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
2 ================================================= |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
3 Plugin system for interative algortithm Version B |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
4 ================================================= |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
5 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
6 After the meeting (September 16) we sort of stumbled on |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
7 two possible versions of the plug-in system. This represents |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
8 the second version. It suffered a few changes after seeing |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
9 Olivier's code and talking to him. |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
10 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
11 Concept |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
12 ======= |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
13 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
14 The basic idea behind this version is not to have a list of all |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
15 possible events, but rather have plugin register to events.By |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
16 specifying what plugin listens to which event produced by what |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
17 plugin you define a sort of dependency graph. Structuring things |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
18 in such a graph might make the script more intuitive when reading. |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
19 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
20 I will first go through pseudo-code for two example and then enumerate |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
21 my insights and concepts on the matter |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
22 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
23 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
24 Example : Producer - Consumer that Guillaume described |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
25 ====================================================== |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
26 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
27 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
28 .. code-block:: |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
29 ''' |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
30 sch = Schedular() |
| 1154 | 31 |
| 32 @FnPlugin(sch) | |
| 33 def producer(self,event): | |
| 34 self.fire('stuff', value = 'some text') | |
| 35 | |
| 36 @FnPlugin(sch) | |
| 37 def consumer(self,event): | |
| 38 print event.value | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
39 |
| 1154 | 40 @FnPlugin(sch) |
| 41 def prod_consumer(self,event): | |
| 42 print event.value | |
| 43 self.fire('stuff2', value = 'stuff') | |
| 44 | |
| 45 producer.act( on = Event('begin'), when = once() ) | |
| 46 producer.act( on = Event('stuff'), when = always() ) | |
| 47 consumer.act( on = Event('stuff'), when = always() ) | |
| 48 prod_consumer.act( on = Event('stuff'), when = always() ) | |
| 49 | |
| 50 sch.run() | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
51 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
52 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
53 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
54 ''' |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
55 Example : Logistic regression |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
56 ============================= |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
57 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
58 Task description |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
59 ---------------- |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
60 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
61 Apply a logistic regression network to some dataset. Use early stopping. |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
62 Save the weights everytime a new best score is obtained. Print trainnig score |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
63 after each epoch. |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
64 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
65 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
66 Possible script |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
67 --------------- |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
68 |
| 1154 | 69 Notes : This would look the same for any other architecture that does not |
| 70 imply pre-training ( i.e. deep networks). For example the mlp. | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
71 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
72 .. code-block:: |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
73 ''' |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
74 |
| 1154 | 75 sched = Schedular() |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
76 |
| 1154 | 77 # Data / Model Building : |
| 78 # I skiped over how to design this part | |
| 79 # though I have some ideas | |
| 80 real_train_data, real_valid_data = load_mnist() | |
| 81 model = logreg() | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
82 |
| 1154 | 83 # Main Plugins ( already provided in the library ); |
| 84 # This wrappers also registers the plugin | |
| 85 train_data = create_data_plugin( sched, data = real_train_data) | |
| 86 train_model = create_train_model(sched, model = model) | |
| 87 validate_model = create_valid_model(sched, model = model, data = valid_data) | |
| 88 early_stopper = create_early_stopper(sched) | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
89 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
90 |
| 1154 | 91 # On the fly plugins ( print random stuff); the main difference from my |
| 92 # FnPlugin from Olivier's version is that it also register the plugin in sched | |
| 93 @FnPlugin(sched) | |
| 94 def print_error(self, event): | |
| 95 if event.type == Event('begin'): | |
| 96 self.value = [] | |
| 97 elif event.type == train_model.error(): | |
| 98 self.value += [event.value] | |
| 99 else event.type == train_data.eod(): | |
| 100 print 'Error :', numpy.mean(self.value) | |
| 101 | |
| 102 @FnPlugin(sched) | |
| 103 def save_model(self, event): | |
| 104 if event.type == early_stopper.new_best_error(): | |
| 105 cPickle.dump(model.parameters(), open('best_params.pkl','wb')) | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
106 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
107 |
| 1173 | 108 # Create the dependency graph describing what does what |
| 109 train_data.act( on = sched.begin(), when = once() ) | |
| 110 train_data.act( on = Event('batch'), | |
| 111 train_data.act( on = train_model.done(), when = always()) | |
| 1154 | 112 train_model.act(on = train_data.batch(), when = always()) |
| 1173 | 113 validate_model.act(on = train_model.done(), when = every(n=10000)) |
| 1154 | 114 early_stopper.act(on = validate_model.error(), when = always()) |
| 115 print_error.act( on = train_model.error(), when = always() ) | |
| 116 print_error.act( on = train_data.eod(), when = always() ) | |
| 117 save_model.act( on = eraly_stopper.new_best_errot(), when = always() ) | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
118 |
| 1154 | 119 # Run the entire thing |
| 120 sched.run() | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
121 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
122 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
123 ''' |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
124 Notes |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
125 ===== |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
126 |
| 1154 | 127 * I think we should have a FnPlugin decorator ( exactly like Olivier's) just |
| 128 that also attaches the new created plugin to the schedule. This way you | |
| 129 can create plugin on the fly ( as long as they are simple functions that | |
| 130 print stuff, or compute simple statitics ). | |
| 131 * I added a method act to a Plugin. You use that to create the dependency | |
| 132 graph ( it could also be named listen to be more plugin like interface) | |
| 133 * Plugins are obtained in 3 ways : | |
| 134 - by wrapping a dataset / model or something similar | |
| 135 - by a function that constructs it from nothing | |
| 136 - by decorating a function | |
| 137 In all cases I would suggest then when creating them you should provide | |
| 138 the schedular as well, and the constructor also registers the plugin | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
139 |
| 1154 | 140 * The plugin concept works well as long as the plugins are a bit towards |
| 141 heavy duty computation, disregarding printing plugins and such. If you have | |
| 142 many small plugins this system might only introduce an overhead. I would | |
| 143 argue that using theano is restricted to each plugin. Therefore I would | |
| 144 strongly suggest that the architecture to be done outside the schedular | |
| 145 with a different approach. | |
| 146 | |
| 147 * I would suggest that the framework to be used only for the training loop | |
| 148 (after you get the adapt function, compute error function) so is more about | |
| 149 the meta-learner, hyper-learner learner level. | |
| 150 | |
| 151 * A general remark that I guess everyone will agree on. We should make | |
| 152 sure that implementing a new plugin is as easy/simple as possible. We | |
| 153 have to hide all the complexity in the schedular ( it is the part of the | |
| 154 code we will not need or we would rarely need to work on). | |
| 155 | |
| 156 * I have not went into how to implement the different components, but | |
| 157 following Olivier's code I think that part would be more or less straight | |
| 158 forward. | |
| 159 | |
| 160 ''' | |
|
1153
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
161 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
162 |
|
ae5ba6206fd3
a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff
changeset
|
163 ''' |