Mercurial > pylearn

annotate doc/v2_planning/plugin_RP.py @ 1423:ea5d27727869

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added pickle disk usage inspection utility 'pkldu'
author Ian Goodfellow
date Mon, 07 Feb 2011 17:04:56 -0500
parents 681b5e7e3b81
children
rev   line source
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
3 !!! Incomplete file .. many of the things I've set up to do are not done
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
4 yet !!!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
6 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
7 Introduction
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
8 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
9
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
10 What this file talks about
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
11 ==========================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
12 * Proposal for the layer committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
13 * Proposal of how to deal with plug-ins ( STEP 2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
14 * Description of how to glue the two parts
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
15 * Some personal beliefs and argumentation
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
17 The file will point out how :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
18 * to use the API's other committee proposed or why and how they should
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
19 change
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
20 * it satisfies the listed requirements ( or why it doesn't)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
21 * this approach might be better then others ( or worse) to the best of
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
22 my knowledge
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
23
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
25 Motivation for writing this file
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
26 ================================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
27
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
28 I wrote this file because:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
29 * It will probably answer most of the questions regarding my view,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
30 minimizing the time wasted on talks
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
31 * When prezenting the entire interface helps see holes in the approach
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
32 * Is here for everybody to read ( easier disimination of information)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
33
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
34
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
35 =======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 Concept
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37 =======
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
39 I think any experiment that we ( or anybody else ) would want to run with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
40 our library will be composed of two steps :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
41
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
42 * Step 1. Constructing (or choosing or initializing) the model, the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
43 datasets, error measures, optimizers and so on ( everything up to the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
44 iterative loop). I think this step has been covered by different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
45 committies but possibly glued together by the layer committee.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
46
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
47 * Step 2. Compose the iterative loops and perform them ( this is what the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
48 architecture committee dealt with)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
49
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
50 I believe there is a natural way of going from *Step 1* to *Step 2*
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
51 which would be presented as Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
52
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
53 Step 2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
54 ======
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
55
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
56 I will start with step 2 ( because I think that is more of a hot subject
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
57 right now). I will assume you have the right plugins at hand.
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
58 This is a DBN with early stopping and ..
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
59
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
60 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
61 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
62 data = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
63 train_xy valid_xy test_xy = split(data, split =
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
64 [(0,40000),(40000,50000),[50000,60000]])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
65 train_x, train_y = train_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
66 valid_x, valid_y = valid_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
67 test_x, test_y = test_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
68
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
69 ################# CONSTRUCTING THE MODEL ###################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
70 ############################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
71
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
72 x0 = pca(train_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
74 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
75 h1 = sigmoid(dotW_b(x0,units = 200), constraint = L1( coeff = 0.1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
76 x1 = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
77 x1.t0 = x0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
78 x1.value = binomial_sample(sigmoid( reconstruct( binomial_sample(h1), x0)))
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
79 cost = free_energy(train_x) - free_energy(x1.t(5))
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
80 grads = [ (g.var, T.grad(cost.var, g.var)) for g in cost.params ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
81 pseudo_cost = sum([ pl.sum(pl.abs(g)) for g in cost.params])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
82 rbm1 = SGD( cost = pseudo_cost, grads = grads)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
83
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
84 # Layer 2:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
85 rbm2,h2 = rbm(h1, units = 200, k = 5, use= 'CD')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
86 # Logreg
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
87 logreg,out = logreg(h2, units = 10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
88 train_err = mean_over(missclassification(argmax(out), train_y))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
89 valid_err = train_err.replace({train_x:valid_x, train_y:valid_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
90 test_err = train_err.replace({train_x: test_x, train_y: test_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
91
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
92 ##########################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
93 ############### Constructing the training loop ###########################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
94
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
95 ca = Schedular()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
96
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
97
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
98 ### Constructing Modes ###
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
99 class pretrain_layer1 ()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
100
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
101 def register()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
102 {
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
103 }
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
104 pretrain_layer2 = ca.mode('pretrain1')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
105 early_stopping = ca.mode('early')
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
106 code_block = ca.mode('code_block')
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
107 kfolds = ca.mode('kfolds')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
108
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
109 # Construct modes dependency graph
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
110 code_block.include([ pretrian_layer1, pretrain_layer2, early_stopper])
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
111 kfolds.include( code_block )
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
112
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
113 pretrain_layer1.act( on = code_block.begin(), when = always())
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
114 pretrain_layer2.act( on = pretrain_layer1.end(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
115 early_stopping.act ( on = pretrain_layer2.end(), when = always())
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
116
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
117
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
118 # Construct counter plugin that keeps track of number of epochs
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
119 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
120 def counter(self, msg):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
121 # a bit of a hack.. it will look more classic if you would
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
122 # start with a class instead
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
123 if not hasattr(self, 'val'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
124 self.val = 0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
125
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
126 if msg = Message('eod'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
127 self.val += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
128 if self.val < 10:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
129 self.fire(Message('continue'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
130 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
131 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
132
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
133
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
134 # Construct pre-training plugins
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
135 rbm1_plugin = pretrain_layer1.include(plugin_wrapper(rbm1))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
136 rbm2_plugin = pretrain_layer2.include(plugin_wrapper(rbm2))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
137 rbm1_counter = pretrain_layer1.include(counter)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
138 rbm2_counter = pretrain_layer2.include(counter)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
139
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
140 rbm1_plugin.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
141 rbm1_plugin.listen(Message('continue'), dataset_restart)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
142 rbm2_plugin.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
143 rbm2_plugin.listen(Message('continue'), dataset_restart)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
144
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
145
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
146 # Dependency graph for pre-training layer 0
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
147 rbm1_plugin.act( on = [ pretrain_layer1.begin() ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
148 rbm1_plugin.value() ] ,
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
149 when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
150 rbm1_counter.act( on = rbm1_plugin.eod(), when = always() )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
151
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
152
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
153 # Dependency graph for pre-training layer 1
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
154 rbm2_plugin.act( on = [ pretrain_layer2.begin() ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
155 rbm2_plugin.value() ] ,
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
156 when = always())
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
157 pretrain_layer2.stop( on = rbm2_plugin.eod(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
158
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
159
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
160 # Constructing fine-tunning plugins
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
161 learner = early_stopper.include(plugin_wrapper(logreg))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
162 validation = early_stopper.include( plugin_wrapper(valid_err)))
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
163 clock = early_stopper.include( ca.generate_clock())
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
164 early_stopper_plugin = early_stopper.include( early_stopper_plugin)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
165
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
166
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
167 def save_model(plugin):
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
168 cPickle.dump(plugin.object, 'just_the_model.pkl')
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
169
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
170 learner.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
171 validation.listen(Message('init'), update_hyperparameters)
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
172 validation.listen(early_stopper_plugin.new_best_score(), save_model)
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
173
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
174 learner.act( on = early_stopper.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
175 learner.act( on = learner.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
176 validation.act( on = clock.hour(), when = every(n = 1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
177 early_stopper.act( on = validation.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
178
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
179 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
180 def kfolds_plugin(self,event):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
181 if not hasattr(self, 'n'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
182 self.n = -1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
183 self.splits = [ [ ( 0,40000),(40000,50000),(50000,60000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
184 [ (10000,50000),(50000,60000),( 0,10000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
185 [ (20000,60000),( 0,10000),(10000,20000) ] ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
186 if self.n < 3:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
187 self.n += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
188 msg = Message('new split')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
189 msg.data = (data.get_hyperparam('split'),self.splits[self.n])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
190 self.fire(msg)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
191 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
192 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
193
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
194
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
195 kfolds.include(kfolds_plugin)
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
196 kfolds_plugin.act([kfolds.begin(), Message('new split')], when = always())
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
197 kfolds_plugin.act(code_block.end(), always() )
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
198 code_block.act(Message('new split'), always() )
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
199
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
200 sched.include(kfolds)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
201
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
202 sched.run()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
203
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
204 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
205
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
206
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
207
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
208 Notes:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
209 when a mode is regstered to begin with a certain message, it will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
210 rebroadcast that message when it starts, with only switching the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
211 type from whatever it was to 'init'. It will also send all 'init' messages
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
212 of the mode in which is included ( or of the schedular).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
213
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
214 one might be able to shorten this by having Macros that creates modes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
215 and automatically register certain plugins to it; you can always
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
216 afterwards add plugins to any mode
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
217
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
218
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
219
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
220 Step 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
221 ======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
222
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
223
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
224 You start with the dataset that you construct as the dataset committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
225 proposed to. You continue constructing your model by applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
226 transformation, more or less like you would in Theano. When constructing
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
227 your model you also get a graph "behind the scene". Note though that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
228 this graph is totally different then the one Theano would create!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
229 Let start with an example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
230
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
231 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
232
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
233 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
234 data_x, data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
235 output = sigmoid(dotW_b(data_x,10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
236 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
237 learner = SGD(err)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
238 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
239
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
240 This shows how to create the learner behind the logistic regression,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
241 but not the function that will compute the validation error or the test
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
242 error ( or any other statistics). Before going into the detail of what
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
243 all those transforms ( or the results after applying one) means, here
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
244 is another partial example for a SdA :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
245
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
246 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
247
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
248 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
249 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
250
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
251 data_x,data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
252 noisy_data_x = gaussian_noise(data_x, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
253 hidden1 = tanh(dotW_b(data_x, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
254 reconstruct1 = reconstruct(hidden1.replace(data_x, noisy_data_x),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
255 noisy_data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
256 err1 = cross_entropy(reconstruct1, data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
257 learner1 = SGD(err1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
258
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
259 # Layer 2 :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
260 noisy_hidden1 = gaussian_noise(hidden1, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
261 hidden2 = tanh(dotW_b(hidden1, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
262 reconstruct2 = reconstruct(hidden2.replace(hidden1,noisy_hidden1),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
263 noisy_hidden1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
264 err2 = cross_entropy(reconstruct2, hidden)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
265 learner2 = SGD(err2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
266
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
267 # Top layer:
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
268
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
269 output = sigmoid(dotW_b(hidden2, n_units = 10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
270 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
271 learner = SGD(err)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
272
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
273 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
274
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
275 What's going on here?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
276 ---------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
277
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
278 By calling different "transforms" (we could call them ops or functions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
279 you decide what the architecture does. What you get back from applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
280 any of these transforms, are nodes. You have different types of nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
281 (which I will enumerate a bit later) but they all offer a basic interface.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
282 That interface is the dataset API + a few more methods and/or attributes.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
283 There are also a few transform that work on the graph that I think will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
284 be pretty useful :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
285
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
286 * .replace(dict) -> method; replaces the subgraphs given as keys with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
287 the ones given as values; throws an exception if it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
288 is impossible
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
289
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
290 * replace(nodes, dict) -> function; call replace on all nodes given that dictionary
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
291
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
292 * reconstruct(dict) -> transform; tries to reconstruct the nodes given as
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
293 keys starting from the nodes given as values by
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
294 going through the inverse of all transforms that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
295 are in between
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
296
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
297 * .tm, .tp -> methods; returns nodes that correspond to the value
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
298 at t-k or t+k
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
299 * recurrent_layer -> function; creates a special type of node that is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
300 recurrent; the node has two important attributes that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
301 need to be specified before calling the node iterator;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
302 those attributes are .t0 which represents the initial
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
303 value and .value which should describe the recurrent
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
304 relation
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
305 * add_constraints -> transform; adds a constraint to a given node
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
306 * data_listener -> function; creates a special node that listens for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
307 messages to get data; it should be used to decompose
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
308 the architecture in modules that can run on different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
309 machines
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
310
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
311 * switch(hyperparam, dict) -> transform; a lazy switch that allows you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
312 do construct by hyper-parameters
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
313
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
314 * get_hyperparameter(name) -> method; given a name it will return the first node
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
315 starting from top that is a hyper parameter and has
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
316 that name
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
317 * get_parameter(name) -> method; given a name it will return the first node
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
318 starting from top that is a parameter and has that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
319 name
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
320 * get_hyperparameters()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
321 * get_parameters()
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
322
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
323
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
324
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
325
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
326 Because every node provides the dataset API it means you can iterate over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
327 any of the nodes. They will produce the original dataset transformed up
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
328 to that point.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
329
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
330 ** NOTES **
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
331 1. This is not like a symbolic graph. When adding a transform
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
332 you can get a warning straight forward. This is because you start from
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
333 the dataset and you always have access to some data. Though sometime
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
334 you would want to have the nodes lazy, i.e. not try to compute everything
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
335 until the graph is done.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
336
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
337 2. You can still have complex Theano expressions. Each node has a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
338 theano variable describing the graph up to that point + optionally
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
339 a compiled function over which you can iterate. We can use some
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
340 on_demand mechanism to compile when needed.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
341
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
342 What types of nodes do you have
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
343 --------------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
344
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
345 Note that this differentiation is more or less semantical and not
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
346 mandatory syntactical. Is just to help understanding the graph.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
347
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
348
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
349 * Data Nodes -- datasets are such nodes; the result of any
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
350 simple transform is also a data node ( like the result
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
351 of a sigmoid, or dotW_b)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
352 * Learner Nodes -- they are the same as data nodes, with the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
353 difference that they have side effects on the model;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
354 they update the weights
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
355 * Apply Nodes -- they are used to connect input variables to
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
356 the transformation/op node and output nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
357 * Dependency Nodes -- very similar to apply nodes just that they connect
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
358 constraints subgraphs to a model graph
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
359 * Parameter Nodes -- when iterating over them they will only output
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
360 the values of the parameters;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
361 * Hyper-parameter Nodes -- very similar to parameter nodes; this is a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
362 semantical difference ( they are not updated by the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
363 any learner nodes)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
364 * Transform Nodes -- this nodes describe the mathematical function
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
365 and if there is one the inverse of that transform; there
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
366 would usually be two types of transforms; ones that use
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
367 theano and those that do not -- this is because those that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
368 do can be composed
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
369
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
370 Each node is lazy, in the sense that unless you try to iterate on it, it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
371 will not try to compute the next value.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
372
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
373
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
374 Isn't this too low level ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
375 --------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
376
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
377 I think that way of writing and decomposing your neural network is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
378 efficient and useful when writing such networks. Of course when you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
379 just want to run a classical SdA you shouldn't need to go through the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
380 trouble of writing all that. I think we should have Macors for this.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
381
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
382 * Macro -- syntactically it looks just like a transform (i.e. a python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
383 function) only that it actually applies multiple transforms to the input
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
384 and might return several nodes (not just one).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
385 Example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
386
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
387
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
388 learner, prediction, pretraining-learners = SdA(
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
389 input = data_x,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
390 target = data_y,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
391 hiddens = [200,200],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
392 noises = [0.1,0.1])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
393
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
394
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
395 How do you deal with loops ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
396 ----------------------------
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
397
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
398 When implementing architectures you some time need to loop like for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
399 RNN or CD, PCD etc. Adding loops in such a scheme is always hard.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
400 I borrowed the idea in the code below from PyBrain. You first construct
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
401 a shell layer that you call recurrent layer. Then you define the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
402 functionality by giving the initial value and the recurrent step.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
403 For example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
404
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
405 .. code-block:: python
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
406
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
407 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
408 # sketch of writing a RNN
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
409 x = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
410 y = recurrent_layer()
1214
681b5e7e3b81 a few comments on James version
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1202
diff changeset
411 y.value = tanh(dotW(x, n=50).t(0) + dotW(y.t(-1),50))
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
412 y.t0 = zeros( (50,))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
413 out = dotW(y,10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
414
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
415
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
416 # sketch of writing CDk starting from x
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
417 x = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
418 x.t0 = input_values
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
419 h = binomial_sample( sigmoid( dotW_b(x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
420 x.value = binomial_sample( sigmoid( reconstruct(h, x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
421 ## the assumption is that the inverse of sigmoid is the identity fn
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
422 pseudo_cost = free_energy(x.tp(k)) - free_energy(x.t0)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
423
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
424
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
425 '''
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
426
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
427 How do I deal with constraints ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
428 --------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
429
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
430 Use the add constrain. You are required to pass a transform with its
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
431 hyper-parameters initial values ?
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
432
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
433
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
434 How do I deal with other type of networs ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
435 ------------------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
436
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
437 (opaque transforms)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
438
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
439 new_data = PCA(data_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
440
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
441
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
442 svn_predictions = SVN(data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
443 svn_learner = SVN_learner(svn_predictions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
444 # Note that for the SVN this might be just syntactic sugar; we have the two
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
445 # steps because we expect different interfaces for this nodes
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
446
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
447
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
448
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
449 Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
450 ========
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
451
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
452 There is a wrapper function called plugin. Once you call plugin over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
453 any of the previous nodes you will get a plugin that has a certain
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
454 set of conventions
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
455
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
456 ''''