Mercurial > pylearn

annotate doc/v2_planning/arch_src/plugin_JB.py @ 1217:c0515c0dfef9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Wrote a sphinx extension for a taglist directive that outputs a javascript tag list and related functions
author boulanni <nicolas_boulanger@hotmail.com>
date Wed, 22 Sep 2010 12:49:41 -0400
parents 478bb1f8215c
children 9fac28d80fb7
rev   line source
1212
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 """plugin_JB - draft of potential library architecture using iterators
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 This strategy makes use of a simple imperative language whose statements are python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 calls to create learning algorithms that can be manipulated and executed in several desirable
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 ways.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 The training procedure for a PCA module is easy to express:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 # allocate the relevant modules
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 dataset = Dataset(numpy.random.RandomState(123).randn(13,1))
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11 pca = PCA_Analysis()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 pca_batchsize=1000
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 # define the control-flow of the algorithm
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 train_pca = SEQ([
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16 BUFFER_REPEAT(pca_batchsize, CALL(dataset.next)),
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 FILT(pca.analyze)])
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 # run the program
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 train_pca.run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22 The CALL, SEQ, FILT, and BUFFER_REPEAT are control-flow elements. The control-flow elements I
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 defined so far are:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 - CALL - a basic statement, just calls a python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 - FILT - like call, but passes the return value of the last CALL or FILT to the python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27 - SEQ - a sequence of elements to run in order
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28 - REPEAT - do something N times (and return None or maybe the last CALL?)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 - BUFFER_REPEAT - do something N times and accumulate the return value from each iter
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 - LOOP - do something an infinite number of times
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31 - CHOOSE - like a switch statement (should rename to SWITCH)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 - WEAVE - interleave execution of multiple control-flow elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33 - POPEN - launch a process and return its status when it's complete
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 - PRINT - a shortcut for CALL(print_obj)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 We don't have many requirements per-se for the architecture, but I think this design respects
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 and realizes all of them.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 The advantages of this approach are:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41 - algorithms (including partially run ones) are COPYABLE, and SERIALIZABLE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43 - algorithms can be executed without seizing control of the python process (the run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 method does this, but if you look inside it you'll see it's a simple for loop)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 - it is easy to execute an algorithm step by step in a main loop that also checks for
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 network or filesystem events related to e.g. job management.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 - the library can provide learning algorithms via control-flow templates, and the user can
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 edit them (with search/replace calls) to include HOOKS, and DIAGNOSTIC plug-in
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 functionality
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 e.g. prog.find(CALL(cd1_update, layer=layer1)).replace_with(
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 SEQ([CALL(cd1_update, layer=layer1), CALL(my_debugfn)]))
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 - user can print the 'program code' of an algorithm built from library pieces
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 - program can be optimized automatically.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 - e.g. BUFFER(N, CALL(dataset.next)) could be replaced if dataset.next implements the
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 right attribute/protocol for 'bufferable' or something.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 - e.g. SEQ([a,b,c,d]) could be compiled to a single CALL to a Theano-compiled function
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64 if a, b, c, and d are calls to callable objects that export something like a
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 'theano_SEQ' interface
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70 __license__ = 'TODO'
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 __copyright__ = 'TODO'
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 import cPickle, copy, os, subprocess, sys, time
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 import numpy
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
76 ####################################################
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
77 # CONTROL-FLOW CONSTRUCTS
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79 class INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80 """Return value for Element.step"""
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
82 class ELEMENT(object):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
83 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
84 Base class for control flow elements (e.g. CALL, REPEAT, etc.)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 The design is that every element has a driver, that is another element, or the iterator
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 implementation in the ELEMENT class.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 the driver calls start when entering a new control element
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90 - this would be called once per e.g. outer loop iteration
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92 the driver calls step to advance the control element
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 - which returns INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94 - which returns any other object to indicate completion
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97 # subclasses should override these methods:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103 # subclasses should typically not override these:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 def run(self, arg=None, n_steps=float('inf')):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 self.start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 i = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 r = self.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 while r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109 i += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 #TODO make sure there is not an off-by-one error
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 if i > n_steps:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 break
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 r = self.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116 class BUFFER_REPEAT(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118 Accumulate a number of return values into one list / array.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
119
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120 The source of return values `src` is a control element that will be restarted repeatedly in
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121 order to fulfil the requiement of gathering N samples.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
123 TODO: support accumulating of tuples of arrays
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 def __init__(self, N, src, storage=None):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127 TODO: use preallocated `storage`
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
128 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
129 self.N = N
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
130 self.n = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
131 self.src = src
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
132 self.storage = storage
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
133 self.src.start(None)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
134 if self.storage != None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135 raise NotImplementedError()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 self.buf = [None] * self.N
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 self.n = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 r = self.src.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 if r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145 self.src.start(None) # restart our stream
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 self.buf[self.n] = r
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147 self.n += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148 if self.n == self.N:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
150 return self.buf
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
151 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 assert 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 class CALL(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
156 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
157 Control flow terminal - call a python function or method.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
158
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
159 Returns the return value of the call.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
160 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
161 def __init__(self, fn, *args, **kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162 self.fn = fn
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
163 self.args = args
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164 self.kwargs=kwargs
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 self.use_start_arg = kwargs.pop('use_start_arg', False)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 self.start_arg = arg
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169 return self
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
170 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
171 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
172 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 if self.use_start_arg:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174 if self.args:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
175 raise TypeError('cant get positional args both ways')
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
176 return self.fn(self.start_arg, **self.kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178 return self.fn(*self.args, **self.kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
179 def __getstate__(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
180 rval = dict(self.__dict__)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
181 if type(self.fn) is type(self.step): #instancemethod
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 fn = rval.pop('fn')
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 rval['i fn'] = fn.im_func, fn.im_self, fn.im_class
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
184 return rval
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
185 def __setstate__(self, dct):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
186 if 'i fn' in dct:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
187 dct['fn'] = type(self.step)(*dct.pop('i fn'))
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
188 self.__dict__.update(dct)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
189
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
190 def FILT(fn, **kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
191 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
192 Return a CALL object that uses the return value from the previous CALL as the first and
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
193 only positional argument.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
194 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
195 return CALL(fn, use_start_arg=True, **kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
196
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
197 def CHOOSE(which, options):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
198 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
199 Execute one out of a number of optional control flow paths
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
200 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
201 raise NotImplementedError()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
202
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
203 def LOOP(elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
204 #TODO: implement a true infinite loop
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
205 try:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
206 iter(elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
207 return REPEAT(sys.maxint, elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
208 except TypeError:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
209 return REPEAT(sys.maxint, [elements])
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
210
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
211 class REPEAT(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
212 def __init__(self, N, elements, pass_rvals=False):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
213 self.N = N
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
214 self.elements = elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
215 self.pass_rvals = pass_rvals
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
216
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
217 #TODO: check for N being callable
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
218 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219 self.n = 0 #loop iteration
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220 self.idx = 0 #element idx
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
222 self.elements[0].start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
223 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
224 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
225 r = self.elements[self.idx].step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
226 if r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
227 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
228 self.idx += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
229 if self.idx < len(self.elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
230 self.elements[self.idx].start(r)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
231 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
232 self.n += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
233 if self.n < self.N:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
234 self.idx = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
235 self.elements[self.idx].start(r)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
236 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
237 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
238 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
239 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
240
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
241 def SEQ(elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
242 return REPEAT(1, elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
243
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
244 class WEAVE(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
245 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
246 Interleave execution of a number of elements.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
247
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
248 TODO: allow a schedule (at least relative frequency) of elements from each program
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
249 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
250 def __init__(self, n_required, elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
251 self.elements = elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
252 if n_required == -1:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
253 self.n_required = len(elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
254 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
255 self.n_required = n_required
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
256 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
257 for el in self.elements:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
258 el.start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
259 self.elem_finished = [0] * len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
260 self.idx = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
261 self.finished= False
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
262 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
263 assert not self.finished # if this is triggered, we have a broken driver
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
264
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
265 #start with this check in case there were no elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
266 # it's possible for the number of finished elements to exceed the threshold
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
267 if sum(self.elem_finished) >= self.n_required:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
268 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
269 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
270
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
271 # step the active element
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
272 r = self.elements[self.idx].step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
273
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
274 if r is not INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
275 self.elem_finished[self.idx] = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
276
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
277 # check for completion
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
278 if sum(self.elem_finished) >= self.n_required:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
279 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
280 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
281
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
282 # advance to the next un-finished element
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
283 self.idx = (self.idx+1) % len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
284 while self.elem_finished[self.idx]:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
285 self.idx = (self.idx+1) % len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
286
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
287 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
288
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
289 class POPEN(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
290 def __init__(self, args):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
291 self.args = args
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
292 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
293 self.p = subprocess.Popen(self.args)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
294 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
295 r = self.p.poll()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
296 if r is None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
297 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
298 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
299
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
300 def PRINT(obj):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
301 return CALL(print_obj, obj)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
302
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
303 class SPAWN(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
304 SUCCESS = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
305 def __init__(self, data, prog):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
306 self.data = data
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
307 self.prog = prog
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
308 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
309 # pickle the (data, prog) pair
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
310 s = cPickle.dumps((self.data, self.prog))
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
311
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
312 # call python with a stub function that
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
313 # unpickles the data, prog pair and starts running the prog
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
314 self.rpipe, wpipe = os.pipe()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
315 code = 'import sys, plugin_JB; sys.exit(plugin_JB.SPAWN._main(%i))'%wpipe
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
316 self.p = subprocess.Popen(
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
317 ['python', '-c', code],
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
318 stdin=subprocess.PIPE)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
319 # send the data and prog to the other process
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
320 self.p.stdin.write(s)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
321 self.finished= False
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
322
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
323 #TODO: send over tgz of the modules this code needs
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
324
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
325 #TODO: When the client process is on a different machine, negotiate with the client
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
326 # process to determine which modules it needs, and send over the code for pure python
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
327 # ones. Make sure versions match for non-pure python ones.
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
328
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
329 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
330 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
331 r = self.p.poll()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
332 if r is None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
333 return INCOMPLETE # typical exit case
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
334 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
335 if r != self.SUCCESS:
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
336 print "UH OH", r # TODO - ???
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
337 rfile = os.fdopen(self.rpipe)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
338 # recv the revised of the data dictionary
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
339 data = cPickle.load(rfile)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
340 # modify the data dict in-place
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
341 # for new values to be visible to other components
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
342 self.data.update(data)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
343 rfile.close()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
344 #TODO: return something meaningful? like r?
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
345 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
346
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
347 @staticmethod
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
348 def _main(wpipe):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
349 #TODO: unpack and install tgz of the modules this code needs
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
350 data, prog = cPickle.load(sys.stdin)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
351 rval = prog.run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
352 os.write(wpipe, cPickle.dumps(data))
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
353 return SPAWN.SUCCESS
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
354 #os.close(wpipe)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
355
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
356
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
357 def print_obj(obj):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
358 print obj
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
359 def print_obj_attr(obj, attr):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
360 print getattr(obj, attr)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
361 def no_op(*args, **kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
362 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
363
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
364 def importable_fn(d):
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
365 d['new key'] = len(d)
478bb1f8215c plugin_JB - added SPAWN control element and demo program
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
366