Mercurial > pylearn

annotate doc/v2_planning/plugin_RP.py @ 1202:7fff3d5c7694

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
author pascanur
date Mon, 20 Sep 2010 20:35:03 -0400
parents fe6c25eb1e37
children 681b5e7e3b81
rev   line source
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
1 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
3 !!! Incomplete file .. many of the things I've set up to do are not done
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
4 yet !!!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
6 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
7 Introduction
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
8 ============
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
9
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
10 What this file talks about
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
11 ==========================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
12 * Proposal for the layer committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
13 * Proposal of how to deal with plug-ins ( STEP 2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
14 * Description of how to glue the two parts
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
15 * Some personal beliefs and argumentation
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
16
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
17 The file will point out how :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
18 * to use the API's other committee proposed or why and how they should
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
19 change
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
20 * it satisfies the listed requirements ( or why it doesn't)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
21 * this approach might be better then others ( or worse) to the best of
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
22 my knowledge
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
23
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
24
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
25 Motivation for writing this file
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
26 ================================
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
27
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
28 I wrote this file because:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
29 * It will probably answer most of the questions regarding my view,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
30 minimizing the time wasted on talks
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
31 * When prezenting the entire interface helps see holes in the approach
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
32 * Is here for everybody to read ( easier disimination of information)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
33
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
34
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
35 =======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
36 Concept
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
37 =======
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
38
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
39 I think any experiment that we ( or anybody else ) would want to run with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
40 our library will be composed of two steps :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
41
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
42 * Step 1. Constructing (or choosing or initializing) the model, the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
43 datasets, error measures, optimizers and so on ( everything up to the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
44 iterative loop). I think this step has been covered by different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
45 committies but possibly glued together by the layer committee.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
46
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
47 * Step 2. Compose the iterative loops and perform them ( this is what the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
48 architecture committee dealt with)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
49
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
50 I believe there is a natural way of going from *Step 1* to *Step 2*
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
51 which would be presented as Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
52
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
53 Step 2
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
54 ======
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
55
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
56 I will start with step 2 ( because I think that is more of a hot subject
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
57 right now). I will assume you have the write plugins at had.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
58 This is a DBN with early stopping and ..
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
59
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
60 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
61 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
62 data = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
63 train_xy valid_xy test_xy = split(data, split =
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
64 [(0,40000),(40000,50000),[50000,60000]])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
65 train_x, train_y = train_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
66 valid_x, valid_y = valid_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
67 test_x, test_y = test_xy
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
68
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
69 ################# CONSTRUCTING THE MODEL ###################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
70 ############################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
71
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
72 x0 = pca(train_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
73
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
74 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
75 h1 = sigmoid(dotW_b(x0,units = 200), constraint = L1( coeff = 0.1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
76 x1 = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
77 x1.t0 = x0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
78 x1.value = binomial_sample(sigmoid( reconstruct( binomial_sample(h1), x0)))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
79 cost = free_energy(train_x) - free_energy(x1.tp(5))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
80 grads = [ (g.var, T.grad(cost.var, g.var)) for g in cost.params ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
81 pseudo_cost = sum([ pl.sum(pl.abs(g)) for g in cost.params])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
82 rbm1 = SGD( cost = pseudo_cost, grads = grads)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
83
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
84 # Layer 2:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
85 rbm2,h2 = rbm(h1, units = 200, k = 5, use= 'CD')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
86 # Logreg
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
87 logreg,out = logreg(h2, units = 10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
88 train_err = mean_over(missclassification(argmax(out), train_y))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
89 valid_err = train_err.replace({train_x:valid_x, train_y:valid_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
90 test_err = train_err.replace({train_x: test_x, train_y: test_y})
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
91
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
92 ##########################################################################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
93 ############### Constructing the training loop ###########################
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
94
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
95 ca = Schedular()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
96
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
97
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
98 ### Constructing Modes ###
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
99 pretrain_layer1 = ca.mode('pretrain0')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
100 pretrain_layer2 = ca.mode('pretrain1')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
101 early_stopping = ca.mode('early')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
102 valid1 = ca.mode('stuff')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
103 kfolds = ca.mode('kfolds')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
104
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
105 # Construct modes dependency graph
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
106 valid0.include([ pretrian_layer1, pretrain_layer2, early_stopper])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
107 kfolds.include( valid0 )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
108
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
109 pretrain_layer1.act( on = valid1.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
110 pretrain_layer2.act( on = pretrain_layer1.end(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
111 early_stopping.act ( on = pretrain_layer2.end(), when = always())
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
112
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
113
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
114 # Construct counter plugin that keeps track of number of epochs
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
115 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
116 def counter(self, msg):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
117 # a bit of a hack.. it will look more classic if you would
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
118 # start with a class instead
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
119 if not hasattr(self, 'val'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
120 self.val = 0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
121
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
122 if msg = Message('eod'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
123 self.val += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
124 if self.val < 10:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
125 self.fire(Message('continue'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
126 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
127 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
128
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
129
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
130 # Construct pre-training plugins
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
131 rbm1_plugin = plugin_wrapper(rbm1, sched = pretrain_layer1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
132 rbm1_plugin.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
133 rbm2_plugin = plugin_wrapper(rbm2, sched = pretrain_layer2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
134 rbm2_plugin.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
135 rbm1_counter = pretrain_layer1.register(counter)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
136 rbm2_counter = pretrain_layer2.register(counter)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
137
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
138
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
139 # Dependency graph for pre-training layer 0
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
140 rbm1_plugin.act( on = [ pretrain_layer1.begin()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
141 Message('continue') ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
142 when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
143 rbm1_counter.act( on = rbm1_plugin.eod(), when = always() )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
144
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
145
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
146 # Dependency graph for pre-training layer 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
147 rbm2_plugin.act( on = pretrain_layer2.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
148 pretrain_layer2.stop( on = rbm2_plugin.eod(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
149
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
150
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
151 # Constructing fine-tunning plugins
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
152 learner = early_stopper.register(plugin_wrapper(logreg))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
153 learner.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
154 validation = early_stopper.register( plugin_wrapper(valid_err)))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
155 validation.listen(Message('init'), update_hyperparameters)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
156 clock = early_stopper.register( ca.generate_clock())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
157 early_stopper_plugin = early_stopper.register( early_stopper_plugin)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
158
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
159 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
160 def save_weights(self, message):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
161 cPickle.dump(logreg, open('model.pkl'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
162
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
163
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
164 learner.act( on = early_stopper.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
165 learner.act( on = learner.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
166 validation.act( on = clock.hour(), when = every(n = 1))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
167 early_stopper.act( on = validation.value(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
168 save_model.act( on = early_stopper.new_best_error(), when =always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
169
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
170 @FnPlugin
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
171 def kfolds_plugin(self,event):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
172 if not hasattr(self, 'n'):
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
173 self.n = -1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
174 self.splits = [ [ ( 0,40000),(40000,50000),(50000,60000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
175 [ (10000,50000),(50000,60000),( 0,10000) ],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
176 [ (20000,60000),( 0,10000),(10000,20000) ] ]
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
177 if self.n < 3:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
178 self.n += 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
179 msg = Message('new split')
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
180 msg.data = (data.get_hyperparam('split'),self.splits[self.n])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
181 self.fire(msg)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
182 else:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
183 self.fire(Message('terminate'))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
184
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
185
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
186 kfolds.register(kfolds_plugin)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
187 kfolds_plugin.act(kfolds.begin(), when = always())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
188 kfolds_plugin.act(valid0.end(), always() )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
189 valid0.act(Message('new split'), always() )
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
190
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
191 sched.include(kfolds)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
192
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
193 sched.run()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
194
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
195 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
196
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
197 Notes:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
198 when a mode is regstered to begin with a certain message, it will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
199 rebroadcast that message when it starts, with only switching the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
200 type from whatever it was to 'init'. It will also send all 'init' messages
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
201 of the mode in which is included ( or of the schedular).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
203 one might be able to shorten this by having Macros that creates modes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
204 and automatically register certain plugins to it; you can always
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
205 afterwards add plugins to any mode
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
206
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
207
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
208
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
209 Step 1
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
210 ======
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
211
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
212
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
213 You start with the dataset that you construct as the dataset committee
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
214 proposed to. You continue constructing your model by applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
215 transformation, more or less like you would in Theano. When constructing
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
216 your model you also get a graph "behind the scene". Note though that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
217 this graph is totally different then the one Theano would create!
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
218 Let start with an example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
219
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
220 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
221
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
222 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
223 data_x, data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
224 output = sigmoid(dotW_b(data_x,10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
225 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
226 learner = SGD(err)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
227 '''
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
228
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
229 This shows how to create the learner behind the logistic regression,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
230 but not the function that will compute the validation error or the test
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
231 error ( or any other statistics). Before going into the detail of what
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
232 all those transforms ( or the results after applying one) means, here
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
233 is another partial example for a SdA :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
234
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
235 .. code-block:: python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
236
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
237 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
238 ## Layer 1:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
239
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
240 data_x,data_y = GPU_transform(load_mnist())
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
241 noisy_data_x = gaussian_noise(data_x, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
242 hidden1 = tanh(dotW_b(data_x, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
243 reconstruct1 = reconstruct(hidden1.replace(data_x, noisy_data_x),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
244 noisy_data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
245 err1 = cross_entropy(reconstruct1, data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
246 learner1 = SGD(err1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
247
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
248 # Layer 2 :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
249 noisy_hidden1 = gaussian_noise(hidden1, amount = 0.1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
250 hidden2 = tanh(dotW_b(hidden1, n_units = 200))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
251 reconstruct2 = reconstruct(hidden2.replace(hidden1,noisy_hidden1),
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
252 noisy_hidden1)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
253 err2 = cross_entropy(reconstruct2, hidden)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
254 learner2 = SGD(err2)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
255
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
256 # Top layer:
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
257
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
258 output = sigmoid(dotW_b(hidden2, n_units = 10))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
259 err = cross_entropy(output, data_y)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
260 learner = SGD(err)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
261
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
262 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
263
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
264 What's going on here?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
265 ---------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
266
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
267 By calling different "transforms" (we could call them ops or functions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
268 you decide what the architecture does. What you get back from applying
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
269 any of these transforms, are nodes. You have different types of nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
270 (which I will enumerate a bit later) but they all offer a basic interface.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
271 That interface is the dataset API + a few more methods and/or attributes.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
272 There are also a few transform that work on the graph that I think will
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
273 be pretty useful :
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
274
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
275 * .replace(dict) -> method; replaces the subgraphs given as keys with
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
276 the ones given as values; throws an exception if it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
277 is impossible
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
278
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
279 * reconstruct(dict) -> transform; tries to reconstruct the nodes given as
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
280 keys starting from the nodes given as values by
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
281 going through the inverse of all transforms that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
282 are in between
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
283
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
284 * .tm, .tp -> methods; returns nodes that correspond to the value
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
285 at t-k or t+k
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
286 * recurrent_layer -> function; creates a special type of node that is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
287 recurrent; the node has two important attributes that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
288 need to be specified before calling the node iterator;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
289 those attributes are .t0 which represents the initial
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
290 value and .value which should describe the recurrent
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
291 relation
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
292 * add_constraints -> transform; adds a constraint to a given node
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
293 * data_listener -> function; creates a special node that listens for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
294 messages to get data; it should be used to decompose
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
295 the architecture in modules that can run on different
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
296 machines
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
297
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
298 * switch(hyperparam, dict) -> transform; a lazy switch that allows you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
299 do construct by hyper-parameters
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
300
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
301 * get_hyperparameter -> method; given a name it will return the first node
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
302 starting from top that is a hyper parameter and has
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
303 that name
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
304 * get_parameter -> method; given a name it will return the first node
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
305 starting from top that is a parameter and has that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
306 name
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
307
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
308
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
309
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
310 Because every node provides the dataset API it means you can iterate over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
311 any of the nodes. They will produce the original dataset transformed up
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
312 to that point.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
313
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
314 ** NOTES **
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
315 1. This is not like a symbolic graph. When adding a transform
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
316 you can get a warning straight forward. This is because you start from
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
317 the dataset and you always have access to some data. Though sometime
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
318 you would want to have the nodes lazy, i.e. not try to compute everything
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
319 until the graph is done.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
320
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
321 2. You can still have complex Theano expressions. Each node has a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
322 theano variable describing the graph up to that point + optionally
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
323 a compiled function over which you can iterate. We can use some
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
324 on_demand mechanism to compile when needed.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
325
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
326 What types of nodes do you have
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
327 --------------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
328
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
329 Note that this differentiation is more or less semantical and not
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
330 mandatory syntactical. Is just to help understanding the graph.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
331
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
332
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
333 * Data Nodes -- datasets are such nodes; the result of any
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
334 simple transform is also a data node ( like the result
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
335 of a sigmoid, or dotW_b)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
336 * Learner Nodes -- they are the same as data nodes, with the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
337 difference that they have side effects on the model;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
338 they update the weights
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
339 * Apply Nodes -- they are used to connect input variables to
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
340 the transformation/op node and output nodes
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
341 * Dependency Nodes -- very similar to apply nodes just that they connect
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
342 constraints subgraphs to a model graph
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
343 * Parameter Nodes -- when iterating over them they will only output
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
344 the values of the parameters;
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
345 * Hyper-parameter Nodes -- very similar to parameter nodes; this is a
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
346 semantical difference ( they are not updated by the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
347 any learner nodes)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
348 * Transform Nodes -- this nodes describe the mathematical function
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
349 and if there is one the inverse of that transform; there
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
350 would usually be two types of transforms; ones that use
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
351 theano and those that do not -- this is because those that
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
352 do can be composed
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
353
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
354 Each node is lazy, in the sense that unless you try to iterate on it, it
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
355 will not try to compute the next value.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
356
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
357
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
358 Isn't this too low level ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
359 --------------------------
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
360
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
361 I think that way of writing and decomposing your neural network is
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
362 efficient and useful when writing such networks. Of course when you
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
363 just want to run a classical SdA you shouldn't need to go through the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
364 trouble of writing all that. I think we should have Macors for this.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
365
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
366 * Macro -- syntactically it looks just like a transform (i.e. a python
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
367 function) only that it actually applies multiple transforms to the input
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
368 and might return several nodes (not just one).
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
369 Example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
370
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
371
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
372 learner, prediction, pretraining-learners = SdA(
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
373 input = data_x,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
374 target = data_y,
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
375 hiddens = [200,200],
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
376 noises = [0.1,0.1])
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
377
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
378
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
379 How do you deal with loops ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
380 ----------------------------
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
381
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
382 When implementing architectures you some time need to loop like for
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
383 RNN or CD, PCD etc. Adding loops in such a scheme is always hard.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
384 I borrowed the idea in the code below from PyBrain. You first construct
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
385 a shell layer that you call recurrent layer. Then you define the
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
386 functionality by giving the initial value and the recurrent step.
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
387 For example:
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
388
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
389 .. code-block:: python
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
390
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
391 '''
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
392 # sketch of writing a RNN
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
393 x = load_mnist()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
394 y = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
395 y.value = tanh(dotW(x, n=50) + dotW(y.tm(1),50))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
396 y.t0 = zeros( (50,))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
397 out = dotW(y,10)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
398
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
399
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
400 # sketch of writing CDk starting from x
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
401 x = recurrent_layer()
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
402 x.t0 = input_values
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
403 h = binomial_sample( sigmoid( dotW_b(x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
404 x.value = binomial_sample( sigmoid( reconstruct(h, x.tm(1))))
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
405 ## the assumption is that the inverse of sigmoid is the identity fn
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
406 pseudo_cost = free_energy(x.tp(k)) - free_energy(x.t0)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
407
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
408
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
409 '''
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
410
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
411 How do I deal with constraints ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
412 --------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
413
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
414 Use the add constrain. You are required to pass a transform with its
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
415 hyper-parameters initial values ?
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
416
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
417
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
418 How do I deal with other type of networs ?
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
419 ------------------------------------------
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
420
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
421 (opaque transforms)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
422
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
423 new_data = PCA(data_x)
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
424
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
425
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
426 svn_predictions = SVN(data_x)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
427 svn_learner = SVN_learner(svn_predictions)
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
428 # Note that for the SVN this might be just syntactic sugar; we have the two
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
429 # steps because we expect different interfaces for this nodes
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
430
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
431
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
432
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
433 Step 1.5
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
434 ========
1153
ae5ba6206fd3 a first draft of pseudo-code for logreg .. using version B (?) approach
Razvan Pascanu <r.pascanu@gmail.com>
parents:
diff changeset
435
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
436 There is a wrapper function called plugin. Once you call plugin over
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
437 any of the previous nodes you will get a plugin that has a certain
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
438 set of conventions
1154
f923dddf0bf7 a better version of the script
pascanur
parents: 1153
diff changeset
439
1202
7fff3d5c7694 ARCHITECTURE/LAYER: a incomplete story about the plug-ins and way of constructing models
pascanur
parents: 1174
diff changeset
440 ''''