Mercurial > pylearn

annotate doc/v2_planning/plugin_RP.py @ 1419:cff305ad9f60

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
TensorFnDataset - added x_ attribute that caches the dataset function return value, but does not get pickled.
author James Bergstra <bergstrj@iro.umontreal.ca>
date Fri, 04 Feb 2011 16:05:22 -0500
parents 681b5e7e3b81
children
rev   line source
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
3 !!! Incomplete file .. many of the things I've set up to do are not done
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
4 yet !!!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
6 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
7 Introduction
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
8 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
9
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
10 What this file talks about
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
11 ==========================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
12 * Proposal for the layer committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
13 * Proposal of how to deal with plug-ins ( STEP 2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
14 * Description of how to glue the two parts
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
15 * Some personal beliefs and argumentation
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
17 The file will point out how :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
18 * to use the API's other committee proposed or why and how they should
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
19 change
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
20 * it satisfies the listed requirements ( or why it doesn't)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
21 * this approach might be better then others ( or worse) to the best of
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
22 my knowledge
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
23
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
25 Motivation for writing this file
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
26 ================================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
27
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
28 I wrote this file because:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
29 * It will probably answer most of the questions regarding my view,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
30 minimizing the time wasted on talks
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
31 * When prezenting the entire interface helps see holes in the approach
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
32 * Is here for everybody to read ( easier disimination of information)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
33
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
34
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
35 =======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 Concept
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37 =======
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
39 I think any experiment that we ( or anybody else ) would want to run with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
40 our library will be composed of two steps :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
41
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
42 * Step 1. Constructing (or choosing or initializing) the model, the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
43 datasets, error measures, optimizers and so on ( everything up to the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
44 iterative loop). I think this step has been covered by different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
45 committies but possibly glued together by the layer committee.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
46
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
47 * Step 2. Compose the iterative loops and perform them ( this is what the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
48 architecture committee dealt with)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
49
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
50 I believe there is a natural way of going from *Step 1* to *Step 2*
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
51 which would be presented as Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
52
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
53 Step 2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
54 ======
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
55
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
56 I will start with step 2 ( because I think that is more of a hot subject
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
57 right now). I will assume you have the right plugins at hand.
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
58 This is a DBN with early stopping and ..
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
59
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
60 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
61 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
62 data = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
63 train_xy valid_xy test_xy = split(data, split =
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
64 [(0,40000),(40000,50000),[50000,60000]])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
65 train_x, train_y = train_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
66 valid_x, valid_y = valid_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
67 test_x, test_y = test_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
68
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
69 ################# CONSTRUCTING THE MODEL ###################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
70 ############################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
71
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
72 x0 = pca(train_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
74 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
75 h1 = sigmoid(dotW_b(x0,units = 200), constraint = L1( coeff = 0.1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
76 x1 = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
77 x1.t0 = x0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
78 x1.value = binomial_sample(sigmoid( reconstruct( binomial_sample(h1), x0)))
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
79 cost = free_energy(train_x) - free_energy(x1.t(5))
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
80 grads = [ (g.var, T.grad(cost.var, g.var)) for g in cost.params ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
81 pseudo_cost = sum([ pl.sum(pl.abs(g)) for g in cost.params])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
82 rbm1 = SGD( cost = pseudo_cost, grads = grads)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
83
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
84 # Layer 2:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
85 rbm2,h2 = rbm(h1, units = 200, k = 5, use= 'CD')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
86 # Logreg
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
87 logreg,out = logreg(h2, units = 10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
88 train_err = mean_over(missclassification(argmax(out), train_y))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
89 valid_err = train_err.replace({train_x:valid_x, train_y:valid_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
90 test_err = train_err.replace({train_x: test_x, train_y: test_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
91
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
92 ##########################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
93 ############### Constructing the training loop ###########################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
94
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
95 ca = Schedular()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
96
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
97
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
98 ### Constructing Modes ###
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
99 class pretrain_layer1 ()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
100
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
101 def register()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
102 {
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
103 }
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
104 pretrain_layer2 = ca.mode('pretrain1')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
105 early_stopping = ca.mode('early')
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
106 code_block = ca.mode('code_block')
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
107 kfolds = ca.mode('kfolds')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
108
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
109 # Construct modes dependency graph
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
110 code_block.include([ pretrian_layer1, pretrain_layer2, early_stopper])
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
111 kfolds.include( code_block )
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
112
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
113 pretrain_layer1.act( on = code_block.begin(), when = always())
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
114 pretrain_layer2.act( on = pretrain_layer1.end(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
115 early_stopping.act ( on = pretrain_layer2.end(), when = always())
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
116
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
117
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
118 # Construct counter plugin that keeps track of number of epochs
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
119 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
120 def counter(self, msg):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
121 # a bit of a hack.. it will look more classic if you would
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
122 # start with a class instead
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
123 if not hasattr(self, 'val'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
124 self.val = 0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
125
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
126 if msg = Message('eod'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
127 self.val += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
128 if self.val < 10:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
129 self.fire(Message('continue'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
130 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
131 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
132
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
133
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
134 # Construct pre-training plugins
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
135 rbm1_plugin = pretrain_layer1.include(plugin_wrapper(rbm1))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
136 rbm2_plugin = pretrain_layer2.include(plugin_wrapper(rbm2))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
137 rbm1_counter = pretrain_layer1.include(counter)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
138 rbm2_counter = pretrain_layer2.include(counter)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
139
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
140 rbm1_plugin.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
141 rbm1_plugin.listen(Message('continue'), dataset_restart)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
142 rbm2_plugin.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
143 rbm2_plugin.listen(Message('continue'), dataset_restart)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
144
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
145
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
146 # Dependency graph for pre-training layer 0
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
147 rbm1_plugin.act( on = [ pretrain_layer1.begin() ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
148 rbm1_plugin.value() ] ,
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
149 when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
150 rbm1_counter.act( on = rbm1_plugin.eod(), when = always() )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
151
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
152
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
153 # Dependency graph for pre-training layer 1
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
154 rbm2_plugin.act( on = [ pretrain_layer2.begin() ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
155 rbm2_plugin.value() ] ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
156 when = always())
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
157 pretrain_layer2.stop( on = rbm2_plugin.eod(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
158
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
159
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
160 # Constructing fine-tunning plugins
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
161 learner = early_stopper.include(plugin_wrapper(logreg))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
162 validation = early_stopper.include( plugin_wrapper(valid_err)))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
163 clock = early_stopper.include( ca.generate_clock())
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
164 early_stopper_plugin = early_stopper.include( early_stopper_plugin)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
165
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
166
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
167 def save_model(plugin):
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
168 cPickle.dump(plugin.object, 'just_the_model.pkl')
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
169
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
170 learner.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
171 validation.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
172 validation.listen(early_stopper_plugin.new_best_score(), save_model)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
173
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
174 learner.act( on = early_stopper.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
175 learner.act( on = learner.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
176 validation.act( on = clock.hour(), when = every(n = 1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
177 early_stopper.act( on = validation.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
178
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
179 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
180 def kfolds_plugin(self,event):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
181 if not hasattr(self, 'n'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
182 self.n = -1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
183 self.splits = [ [ ( 0,40000),(40000,50000),(50000,60000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
184 [ (10000,50000),(50000,60000),( 0,10000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
185 [ (20000,60000),( 0,10000),(10000,20000) ] ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
186 if self.n < 3:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
187 self.n += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
188 msg = Message('new split')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
189 msg.data = (data.get_hyperparam('split'),self.splits[self.n])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
190 self.fire(msg)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
191 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
192 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
193
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
194
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
195 kfolds.include(kfolds_plugin)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
196 kfolds_plugin.act([kfolds.begin(), Message('new split')], when = always())
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
197 kfolds_plugin.act(code_block.end(), always() )
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
198 code_block.act(Message('new split'), always() )
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
199
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
200 sched.include(kfolds)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
201
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
202 sched.run()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
203
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
204 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
205
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
206
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
207
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
208 Notes:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
209 when a mode is regstered to begin with a certain message, it will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
210 rebroadcast that message when it starts, with only switching the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
211 type from whatever it was to 'init'. It will also send all 'init' messages
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
212 of the mode in which is included ( or of the schedular).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
213
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
214 one might be able to shorten this by having Macros that creates modes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
215 and automatically register certain plugins to it; you can always
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
216 afterwards add plugins to any mode
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
217
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
218
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
219
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
220 Step 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
221 ======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
222
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
223
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
224 You start with the dataset that you construct as the dataset committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
225 proposed to. You continue constructing your model by applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
226 transformation, more or less like you would in Theano. When constructing
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
227 your model you also get a graph "behind the scene". Note though that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
228 this graph is totally different then the one Theano would create!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
229 Let start with an example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
230
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
231 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
232
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
233 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
234 data_x, data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
235 output = sigmoid(dotW_b(data_x,10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
236 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
237 learner = SGD(err)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
238 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
239
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
240 This shows how to create the learner behind the logistic regression,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
241 but not the function that will compute the validation error or the test
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
242 error ( or any other statistics). Before going into the detail of what
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
243 all those transforms ( or the results after applying one) means, here
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
244 is another partial example for a SdA :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
245
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
246 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
247
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
248 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
249 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
250
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
251 data_x,data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
252 noisy_data_x = gaussian_noise(data_x, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
253 hidden1 = tanh(dotW_b(data_x, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
254 reconstruct1 = reconstruct(hidden1.replace(data_x, noisy_data_x),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
255 noisy_data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
256 err1 = cross_entropy(reconstruct1, data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
257 learner1 = SGD(err1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
258
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
259 # Layer 2 :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
260 noisy_hidden1 = gaussian_noise(hidden1, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
261 hidden2 = tanh(dotW_b(hidden1, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
262 reconstruct2 = reconstruct(hidden2.replace(hidden1,noisy_hidden1),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
263 noisy_hidden1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
264 err2 = cross_entropy(reconstruct2, hidden)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
265 learner2 = SGD(err2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
266
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
267 # Top layer:
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
268
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
269 output = sigmoid(dotW_b(hidden2, n_units = 10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
270 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
271 learner = SGD(err)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
272
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
273 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
274
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
275 What's going on here?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
276 ---------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
277
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
278 By calling different "transforms" (we could call them ops or functions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
279 you decide what the architecture does. What you get back from applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
280 any of these transforms, are nodes. You have different types of nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
281 (which I will enumerate a bit later) but they all offer a basic interface.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
282 That interface is the dataset API + a few more methods and/or attributes.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
283 There are also a few transform that work on the graph that I think will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
284 be pretty useful :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
285
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
286 * .replace(dict) -> method; replaces the subgraphs given as keys with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
287 the ones given as values; throws an exception if it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
288 is impossible
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
289
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
290 * replace(nodes, dict) -> function; call replace on all nodes given that dictionary
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
291
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
292 * reconstruct(dict) -> transform; tries to reconstruct the nodes given as
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
293 keys starting from the nodes given as values by
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
294 going through the inverse of all transforms that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
295 are in between
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
296
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
297 * .tm, .tp -> methods; returns nodes that correspond to the value
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
298 at t-k or t+k
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
299 * recurrent_layer -> function; creates a special type of node that is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
300 recurrent; the node has two important attributes that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
301 need to be specified before calling the node iterator;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
302 those attributes are .t0 which represents the initial
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
303 value and .value which should describe the recurrent
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
304 relation
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
305 * add_constraints -> transform; adds a constraint to a given node
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
306 * data_listener -> function; creates a special node that listens for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
307 messages to get data; it should be used to decompose
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
308 the architecture in modules that can run on different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
309 machines
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
310
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
311 * switch(hyperparam, dict) -> transform; a lazy switch that allows you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
312 do construct by hyper-parameters
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
313
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
314 * get_hyperparameter(name) -> method; given a name it will return the first node
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
315 starting from top that is a hyper parameter and has
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
316 that name
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
317 * get_parameter(name) -> method; given a name it will return the first node
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
318 starting from top that is a parameter and has that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
319 name
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
320 * get_hyperparameters()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
321 * get_parameters()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
322
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
323
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
324
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
325
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
326 Because every node provides the dataset API it means you can iterate over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
327 any of the nodes. They will produce the original dataset transformed up
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
328 to that point.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
329
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
330 ** NOTES **
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
331 1. This is not like a symbolic graph. When adding a transform
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
332 you can get a warning straight forward. This is because you start from
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
333 the dataset and you always have access to some data. Though sometime
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
334 you would want to have the nodes lazy, i.e. not try to compute everything
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
335 until the graph is done.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
336
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
337 2. You can still have complex Theano expressions. Each node has a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
338 theano variable describing the graph up to that point + optionally
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
339 a compiled function over which you can iterate. We can use some
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
340 on_demand mechanism to compile when needed.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
341
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
342 What types of nodes do you have
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
343 --------------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
344
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
345 Note that this differentiation is more or less semantical and not
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
346 mandatory syntactical. Is just to help understanding the graph.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
347
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
348
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
349 * Data Nodes -- datasets are such nodes; the result of any
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
350 simple transform is also a data node ( like the result
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
351 of a sigmoid, or dotW_b)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
352 * Learner Nodes -- they are the same as data nodes, with the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
353 difference that they have side effects on the model;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
354 they update the weights
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
355 * Apply Nodes -- they are used to connect input variables to
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
356 the transformation/op node and output nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
357 * Dependency Nodes -- very similar to apply nodes just that they connect
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
358 constraints subgraphs to a model graph
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
359 * Parameter Nodes -- when iterating over them they will only output
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
360 the values of the parameters;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
361 * Hyper-parameter Nodes -- very similar to parameter nodes; this is a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
362 semantical difference ( they are not updated by the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
363 any learner nodes)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
364 * Transform Nodes -- this nodes describe the mathematical function
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
365 and if there is one the inverse of that transform; there
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
366 would usually be two types of transforms; ones that use
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
367 theano and those that do not -- this is because those that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
368 do can be composed
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
369
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
370 Each node is lazy, in the sense that unless you try to iterate on it, it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
371 will not try to compute the next value.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
372
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
373
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
374 Isn't this too low level ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
375 --------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
376
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
377 I think that way of writing and decomposing your neural network is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
378 efficient and useful when writing such networks. Of course when you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
379 just want to run a classical SdA you shouldn't need to go through the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
380 trouble of writing all that. I think we should have Macors for this.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
381
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
382 * Macro -- syntactically it looks just like a transform (i.e. a python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
383 function) only that it actually applies multiple transforms to the input
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
384 and might return several nodes (not just one).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
385 Example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
386
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
387
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
388 learner, prediction, pretraining-learners = SdA(
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
389 input = data_x,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
390 target = data_y,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
391 hiddens = [200,200],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
392 noises = [0.1,0.1])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
393
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
394
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
395 How do you deal with loops ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
396 ----------------------------
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
397
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
398 When implementing architectures you some time need to loop like for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
399 RNN or CD, PCD etc. Adding loops in such a scheme is always hard.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
400 I borrowed the idea in the code below from PyBrain. You first construct
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
401 a shell layer that you call recurrent layer. Then you define the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
402 functionality by giving the initial value and the recurrent step.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
403 For example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
404
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
405 .. code-block:: python
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
406
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
407 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
408 # sketch of writing a RNN
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
409 x = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
410 y = recurrent_layer()
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
411 y.value = tanh(dotW(x, n=50).t(0) + dotW(y.t(-1),50))
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
412 y.t0 = zeros( (50,))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
413 out = dotW(y,10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
414
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
415
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
416 # sketch of writing CDk starting from x
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
417 x = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
418 x.t0 = input_values
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
419 h = binomial_sample( sigmoid( dotW_b(x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
420 x.value = binomial_sample( sigmoid( reconstruct(h, x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
421 ## the assumption is that the inverse of sigmoid is the identity fn
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
422 pseudo_cost = free_energy(x.tp(k)) - free_energy(x.t0)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
423
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
424
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
425 '''
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
426
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
427 How do I deal with constraints ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
428 --------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
429
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
430 Use the add constrain. You are required to pass a transform with its
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
431 hyper-parameters initial values ?
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
432
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
433
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
434 How do I deal with other type of networs ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
435 ------------------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
436
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
437 (opaque transforms)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
438
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
439 new_data = PCA(data_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
440
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
441
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
442 svn_predictions = SVN(data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
443 svn_learner = SVN_learner(svn_predictions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
444 # Note that for the SVN this might be just syntactic sugar; we have the two
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
445 # steps because we expect different interfaces for this nodes
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
446
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
447
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
448
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
449 Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
450 ========
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
451
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
452 There is a wrapper function called plugin. Once you call plugin over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
453 any of the previous nodes you will get a plugin that has a certain
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
454 set of conventions
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
455
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
456 ''''