pylearn: doc/v2_planning/arch_src/plugin

author	James Bergstra <bergstrj@iro.umontreal.ca>
date	Wed, 22 Sep 2010 01:37:55 -0400
parents
children	9fac28d80fb7

rev	line source
1212 478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	1 """plugin_JB - draft of potential library architecture using iterators
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	2
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	3 This strategy makes use of a simple imperative language whose statements are python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	4 calls to create learning algorithms that can be manipulated and executed in several desirable
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	5 ways.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	6
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	7 The training procedure for a PCA module is easy to express:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	8
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	9 # allocate the relevant modules
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	10 dataset = Dataset(numpy.random.RandomState(123).randn(13,1))
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	11 pca = PCA_Analysis()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	12 pca_batchsize=1000
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	13
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	14 # define the control-flow of the algorithm
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	15 train_pca = SEQ([
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	16 BUFFER_REPEAT(pca_batchsize, CALL(dataset.next)),
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	17 FILT(pca.analyze)])
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	18
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	19 # run the program
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	20 train_pca.run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	21
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	22 The CALL, SEQ, FILT, and BUFFER_REPEAT are control-flow elements. The control-flow elements I
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	23 defined so far are:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	24
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	25 - CALL - a basic statement, just calls a python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	26 - FILT - like call, but passes the return value of the last CALL or FILT to the python function
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	27 - SEQ - a sequence of elements to run in order
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	28 - REPEAT - do something N times (and return None or maybe the last CALL?)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	29 - BUFFER_REPEAT - do something N times and accumulate the return value from each iter
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	30 - LOOP - do something an infinite number of times
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	31 - CHOOSE - like a switch statement (should rename to SWITCH)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	32 - WEAVE - interleave execution of multiple control-flow elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	33 - POPEN - launch a process and return its status when it's complete
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	34 - PRINT - a shortcut for CALL(print_obj)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	35
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	36
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	37 We don't have many requirements per-se for the architecture, but I think this design respects
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	38 and realizes all of them.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	39 The advantages of this approach are:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	40
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	41 - algorithms (including partially run ones) are COPYABLE, and SERIALIZABLE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	42
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	43 - algorithms can be executed without seizing control of the python process (the run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	44 method does this, but if you look inside it you'll see it's a simple for loop)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	45
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	46 - it is easy to execute an algorithm step by step in a main loop that also checks for
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	47 network or filesystem events related to e.g. job management.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	48
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	49 - the library can provide learning algorithms via control-flow templates, and the user can
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	50 edit them (with search/replace calls) to include HOOKS, and DIAGNOSTIC plug-in
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	51 functionality
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	52
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	53 e.g. prog.find(CALL(cd1_update, layer=layer1)).replace_with(
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	54 SEQ([CALL(cd1_update, layer=layer1), CALL(my_debugfn)]))
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	55
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	56 - user can print the 'program code' of an algorithm built from library pieces
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	57
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	58 - program can be optimized automatically.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	59
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	60 - e.g. BUFFER(N, CALL(dataset.next)) could be replaced if dataset.next implements the
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	61 right attribute/protocol for 'bufferable' or something.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	62
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	63 - e.g. SEQ([a,b,c,d]) could be compiled to a single CALL to a Theano-compiled function
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	64 if a, b, c, and d are calls to callable objects that export something like a
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	65 'theano_SEQ' interface
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	66
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	67
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	68 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	69
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	70 __license__ = 'TODO'
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	71 __copyright__ = 'TODO'
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	72
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	73 import cPickle, copy, os, subprocess, sys, time
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	74 import numpy
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	75
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	76 ####################################################
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	77 # CONTROL-FLOW CONSTRUCTS
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	78
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	79 class INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	80 """Return value for Element.step"""
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	81
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	82 class ELEMENT(object):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	83 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	84 Base class for control flow elements (e.g. CALL, REPEAT, etc.)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	85
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	86 The design is that every element has a driver, that is another element, or the iterator
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	87 implementation in the ELEMENT class.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	88
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	89 the driver calls start when entering a new control element
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	90 - this would be called once per e.g. outer loop iteration
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	91
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	92 the driver calls step to advance the control element
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	93 - which returns INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	94 - which returns any other object to indicate completion
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	95 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	96
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	97 # subclasses should override these methods:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	98 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	99 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	100 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	101 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	102
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	103 # subclasses should typically not override these:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	104 def run(self, arg=None, n_steps=float('inf')):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	105 self.start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	106 i = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	107 r = self.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	108 while r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	109 i += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	110 #TODO make sure there is not an off-by-one error
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	111 if i > n_steps:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	112 break
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	113 r = self.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	114 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	115
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	116 class BUFFER_REPEAT(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	117 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	118 Accumulate a number of return values into one list / array.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	119
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	120 The source of return values `src` is a control element that will be restarted repeatedly in
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	121 order to fulfil the requiement of gathering N samples.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	122
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	123 TODO: support accumulating of tuples of arrays
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	124 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	125 def __init__(self, N, src, storage=None):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	126 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	127 TODO: use preallocated `storage`
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	128 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	129 self.N = N
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	130 self.n = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	131 self.src = src
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	132 self.storage = storage
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	133 self.src.start(None)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	134 if self.storage != None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	135 raise NotImplementedError()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	136 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	137 self.buf = [None] * self.N
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	138 self.n = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	139 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	140 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	141 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	142 r = self.src.step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	143 if r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	144 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	145 self.src.start(None) # restart our stream
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	146 self.buf[self.n] = r
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	147 self.n += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	148 if self.n == self.N:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	149 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	150 return self.buf
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	151 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	152 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	153 assert 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	154
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	155 class CALL(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	156 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	157 Control flow terminal - call a python function or method.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	158
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	159 Returns the return value of the call.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	160 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	161 def __init__(self, fn, args, *kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	162 self.fn = fn
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	163 self.args = args
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	164 self.kwargs=kwargs
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	165 self.use_start_arg = kwargs.pop('use_start_arg', False)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	166 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	167 self.start_arg = arg
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	168 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	169 return self
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	170 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	171 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	172 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	173 if self.use_start_arg:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	174 if self.args:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	175 raise TypeError('cant get positional args both ways')
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	176 return self.fn(self.start_arg, **self.kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	177 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	178 return self.fn(self.args, *self.kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	179 def __getstate__(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	180 rval = dict(self.__dict__)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	181 if type(self.fn) is type(self.step): #instancemethod
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	182 fn = rval.pop('fn')
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	183 rval['i fn'] = fn.im_func, fn.im_self, fn.im_class
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	184 return rval
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	185 def __setstate__(self, dct):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	186 if 'i fn' in dct:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	187 dct['fn'] = type(self.step)(*dct.pop('i fn'))
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	188 self.__dict__.update(dct)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	189
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	190 def FILT(fn, **kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	191 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	192 Return a CALL object that uses the return value from the previous CALL as the first and
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	193 only positional argument.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	194 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	195 return CALL(fn, use_start_arg=True, **kwargs)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	196
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	197 def CHOOSE(which, options):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	198 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	199 Execute one out of a number of optional control flow paths
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	200 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	201 raise NotImplementedError()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	202
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	203 def LOOP(elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	204 #TODO: implement a true infinite loop
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	205 try:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	206 iter(elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	207 return REPEAT(sys.maxint, elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	208 except TypeError:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	209 return REPEAT(sys.maxint, [elements])
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	210
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	211 class REPEAT(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	212 def __init__(self, N, elements, pass_rvals=False):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	213 self.N = N
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	214 self.elements = elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	215 self.pass_rvals = pass_rvals
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	216
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	217 #TODO: check for N being callable
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	218 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	219 self.n = 0 #loop iteration
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	220 self.idx = 0 #element idx
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	221 self.finished = False
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	222 self.elements[0].start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	223 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	224 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	225 r = self.elements[self.idx].step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	226 if r is INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	227 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	228 self.idx += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	229 if self.idx < len(self.elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	230 self.elements[self.idx].start(r)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	231 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	232 self.n += 1
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	233 if self.n < self.N:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	234 self.idx = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	235 self.elements[self.idx].start(r)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	236 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	237 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	238 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	239 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	240
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	241 def SEQ(elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	242 return REPEAT(1, elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	243
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	244 class WEAVE(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	245 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	246 Interleave execution of a number of elements.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	247
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	248 TODO: allow a schedule (at least relative frequency) of elements from each program
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	249 """
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	250 def __init__(self, n_required, elements):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	251 self.elements = elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	252 if n_required == -1:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	253 self.n_required = len(elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	254 else:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	255 self.n_required = n_required
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	256 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	257 for el in self.elements:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	258 el.start(arg)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	259 self.elem_finished = [0] * len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	260 self.idx = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	261 self.finished= False
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	262 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	263 assert not self.finished # if this is triggered, we have a broken driver
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	264
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	265 #start with this check in case there were no elements
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	266 # it's possible for the number of finished elements to exceed the threshold
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	267 if sum(self.elem_finished) >= self.n_required:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	268 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	269 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	270
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	271 # step the active element
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	272 r = self.elements[self.idx].step()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	273
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	274 if r is not INCOMPLETE:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	275 self.elem_finished[self.idx] = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	276
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	277 # check for completion
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	278 if sum(self.elem_finished) >= self.n_required:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	279 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	280 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	281
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	282 # advance to the next un-finished element
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	283 self.idx = (self.idx+1) % len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	284 while self.elem_finished[self.idx]:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	285 self.idx = (self.idx+1) % len(self.elements)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	286
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	287 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	288
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	289 class POPEN(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	290 def __init__(self, args):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	291 self.args = args
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	292 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	293 self.p = subprocess.Popen(self.args)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	294 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	295 r = self.p.poll()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	296 if r is None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	297 return INCOMPLETE
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	298 return r
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	299
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	300 def PRINT(obj):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	301 return CALL(print_obj, obj)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	302
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	303 class SPAWN(ELEMENT):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	304 SUCCESS = 0
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	305 def __init__(self, data, prog):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	306 self.data = data
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	307 self.prog = prog
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	308 def start(self, arg):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	309 # pickle the (data, prog) pair
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	310 s = cPickle.dumps((self.data, self.prog))
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	311
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	312 # call python with a stub function that
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	313 # unpickles the data, prog pair and starts running the prog
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	314 self.rpipe, wpipe = os.pipe()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	315 code = 'import sys, plugin_JB; sys.exit(plugin_JB.SPAWN._main(%i))'%wpipe
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	316 self.p = subprocess.Popen(
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	317 ['python', '-c', code],
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	318 stdin=subprocess.PIPE)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	319 # send the data and prog to the other process
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	320 self.p.stdin.write(s)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	321 self.finished= False
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	322
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	323 #TODO: send over tgz of the modules this code needs
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	324
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	325 #TODO: When the client process is on a different machine, negotiate with the client
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	326 # process to determine which modules it needs, and send over the code for pure python
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	327 # ones. Make sure versions match for non-pure python ones.
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	328
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	329 def step(self):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	330 assert not self.finished
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	331 r = self.p.poll()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	332 if r is None:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	333 return INCOMPLETE # typical exit case
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	334 self.finished = True
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	335 if r != self.SUCCESS:
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	336 print "UH OH", r # TODO - ???
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	337 rfile = os.fdopen(self.rpipe)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	338 # recv the revised of the data dictionary
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	339 data = cPickle.load(rfile)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	340 # modify the data dict in-place
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	341 # for new values to be visible to other components
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	342 self.data.update(data)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	343 rfile.close()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	344 #TODO: return something meaningful? like r?
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	345 return None
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	346
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	347 @staticmethod
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	348 def _main(wpipe):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	349 #TODO: unpack and install tgz of the modules this code needs
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	350 data, prog = cPickle.load(sys.stdin)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	351 rval = prog.run()
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	352 os.write(wpipe, cPickle.dumps(data))
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	353 return SPAWN.SUCCESS
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	354 #os.close(wpipe)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	355
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	356
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	357 def print_obj(obj):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	358 print obj
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	359 def print_obj_attr(obj, attr):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	360 print getattr(obj, attr)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	361 def no_op(args, *kwargs):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	362 pass
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	363
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	364 def importable_fn(d):
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	365 d['new key'] = len(d)
478bb1f8215c plugin_JB - added SPAWN control element and demo program James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	366

1212

478bb1f8215c plugin_JB - added SPAWN control element and demo program