Mercurial > pylearn

annotate doc/v2_planning/arch_src/checkpoint_JB.py @ 1340:04b988fb00b6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Reply to Razvan
author Olivier Delalleau <delallea@iro>
date Thu, 21 Oct 2010 16:18:52 -0400
parents 09ad2a4f663c
children
rev   line source
1336
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
1 import copy as copy_module
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
2
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 #TODO: use logging.info to report cache hits/ misses
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5 CO_VARARGS = 0x0004
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
6 CO_VARKEYWORDS = 0x0008
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8 class mem_db(dict):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9 # A key->document dictionary.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 # A "document" is itself a dictionary.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12 # A "document" can be a small or large object, but it cannot be partially retrieved.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
13
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14 # This simple data structure is used in pylearn to cache intermediate reults between
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 # several process invocations.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16 pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 class UNSPECIFIED(object):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 class CtrlObj(object):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 Job control API.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 This interface makes it easier to break a logical program into pieces that can be
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26 executed by several different processes, either serially or in parallel.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
28
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
29 The base class provides decorators to simplify some common cache patterns:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
30 - cache_pickle to cache arbitrary return values using the pickle mechanism
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
31
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
32 - cache_dict to cache dict return values directly using the document db
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
33
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
34 - cache_numpy to cache [single] numpy ndarray rvals in a way that supports memmapping of
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35 large arrays.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
36
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
37 Authors are encouraged to use these when they apply, but should feel free to implement
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
38 other cache logic when these standard ones are lacking using the CtorlObj.get() and
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
39 CtorlObj.set() methods.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
40
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
41
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
42 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
43
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
44 def __init__(self, rootdir, db, autosync):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
45 self.db = db
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
46 self.r_lookup = {}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
47 self.autosync=autosync
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
48
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
49 def get(self, key, default_val=UNSPECIFIED, copy=True):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50 # Default to return a COPY because a self.set() is required to make a change persistent.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
51 # Inplace changes that the CtrlObj does not know about (via self.set()) will not be saved.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
52 try:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
53 val = self.db[key]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
54 except:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55 if default_val is not UNSPECIFIED:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
56 # return default_val, but do not add it to the r_lookup object
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
57 # since looking up that key in the future would not retrieve default_val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
58 return default_val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59 else:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
60 raise
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
61 if copy:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
62 rval = copy_module.deepcopy(val)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63 else:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
64 rval = val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
65 self.r_lookup[id(rval)] = key
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
66 return rval
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
68 def get_key(self, val):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69 """Return the key that retrieved `val`.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
70
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
71 This is useful for specifying cache keys for unhashable (e.g. numpy) objects that
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72 happen to be stored in the db.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
73 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
74 return self.r_lookup[id(val)]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75 def set(self, key, val):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
76 vv = dict(val)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
77 if self.db.get(key, None) not in (val, None):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78 del_keys = [k for (k,v) in self.r_lookup.iteritems() if v == key]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
79 for k in del_keys:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80 del self.r_lookup[k]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81 self.db[key] = vv
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
82 def delete(self, key):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
83 del_keys = [k for (k,v) in self.r_lookup.iteritems() if v == key]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
84 for k in del_keys:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
85 del self.r_lookup[k]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
86 del self.db[key]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
87 def checkpoint(self):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
88 """Potentially pass control to another greenlet/tasklet that could potentially
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
89 serialize this (calling) greenlet/tasklet using cPickle.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
90 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
91 pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
92
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
93 def sync(self, pull=True, push=True):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
94 """Synchronise local changes with a master version (if applicable).
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
96 pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
97
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98 def open(self, filename):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
99 """Return a file-handle to a file that can be synced with a server"""
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
100 #todo - save references / proxies of the file objects returned here
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101 # and sync them with a server if they are closed
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
102 return open(os.path.join(self.rootdir, filename))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
103
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104 def open_unique(self, mode='wb', prefix='uniq_', suffix=''):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
105 #TODO: use the standard lib algo for this if you can find it.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
106 if suffix:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107 template = prefix+'%06i.'+suffix
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
108 else:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
109 template = prefix+'%06i'
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110 while True:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
111 fname = template%numpy.random.randint(999999)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112 path = os.path.join(self.rootdir, fname)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
113 try:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114 open(path).close()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115 except IOError: #file not found
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
116 return open(path, mode=mode), path
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
117
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
118 def memory_ctrl_obj():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
119 return CtrlObj(db=dict())
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
121 def directory_ctrl_obj(path, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
122 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
123
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
124 def mongo_ctrl_obj(connection_args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
125 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
126
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
127 def couchdb_ctrl_obj(connection_args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
128 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
129
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
130 def jobman_ctrl_obj(connection_args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
131 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
132
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
133
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
134 def _default_values(f):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
135 """Return a dictionary param -> default value of function `f`'s parameters"""
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
136 default_dict = {}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
137 func_defaults = f.func_defaults
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
138 if func_defaults:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
139 first_default_pos = f.func_code.co_argcount-len(f.func_defaults)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
140 params_with_defaults = f.func_code.co_varnames[first_default_pos:f.func_code.co_argcount]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141 rval = dict(zip(params_with_defaults, f.func_defaults))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
142 else:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
143 rval = {}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
144 return rval
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
145
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
146 def test_default_values():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
147
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
148 def f(a): pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
149 assert _default_values(f) == {}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
150
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
151 def f(a, b=1):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
152 aa = 5
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
153 assert _default_values(f) == dict(b=1)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
154
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
155 def f(a, b=1, c=2, *args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
156 e = b+c
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
157 return e
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
158 assert _default_values(f) == dict(b=1, c=2)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
159
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
160 def _arg_assignment(f, args, kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
161 # make a dictionary from args and kwargs that contains all the arguments to f and their
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
162 # values
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
163 assignment = dict()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
164
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
165 params = f.func_code.co_varnames[:f.func_code.co_argcount] #CORRECT?
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
166
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
167 f_accepts_varargs = f.func_code.co_flags & CO_VARARGS
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
168 f_accepts_kwargs = f.func_code.co_flags & CO_VARKEYWORDS
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
169
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
170 if f_accepts_varargs:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
171 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
172 if f_accepts_kwargs:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
173 raise NotImplementedError()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
174
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
175 # first add positional arguments
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
176 #TODO: what if f accepts a '*args' or similar?
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
177 assert len(args) <= f.func_code.co_argcount
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
178 for i, a in enumerate(args):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
179 assignment[f.func_code.co_varnames[i]] = a # CORRECT??
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
180
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
181 # next add kw arguments
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
182 for k,v in kwargs.iteritems():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
183 if k in assignment:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
184 #TODO: match Python error
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
185 raise TypeError('duplicate argument provided for parameter', k)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
186
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
187 if (not f_accepts_kwargs) and (k not in params):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
188 #TODO: match Python error
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
189 raise TypeError('invalid keyword argument', k)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
190
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
191 assignment[k] = v
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
192
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
193 # finally add default arguments for any remaining parameters
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
194 for k,v in _default_values(f).iteritems():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
195 if k in assignment:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
196 pass # this argument has [already] been specified
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
197 else:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
198 assignment[k] = v
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
199
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
200 # TODO
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
201 # check that the assignment covers all parameters without default values
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
202
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
203 # TODO
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
204 # check that the assignment includes no extra variables if f does not accept a '**'
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
205 # parameter.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
206
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
207 return assignment
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
208
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
209 def test_arg_assignment():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
210 #TODO: check cases that should cause errors
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
211 # - doubly-specified arguments,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
212 # - insufficient arguments
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
213
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
214 def f():pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
215 assert _arg_assignment(f, (), {}) == {}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
216 def f(a):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
217 assert _arg_assignment(f, (1,), {}) == {'a':1}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
218 def f(a):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
219 assert _arg_assignment(f, (), {'a':1}) == {'a':1}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
220
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
221 def f(a=1):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
222 assert _arg_assignment(f, (), {}) == {'a':1}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
223 def f(a=1):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
224 assert _arg_assignment(f, (2,), {}) == {'a':2}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
225 def f(a=1):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
226 assert _arg_assignment(f, (), {'a':2}) == {'a':2}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
227 def f(a=1):pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
228 assert _arg_assignment(f, (), {'a':2}) == {'a':2}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
229
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
230 def f(b, a=1): pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
231 assert _arg_assignment(f, (3,4), {}) == {'b':3, 'a':4}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
232 def f(b, a=1): pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
233 assert _arg_assignment(f, (3,), {'a':4}) == {'b':3, 'a':4}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
234 def f(b, a=1): pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
235 assert _arg_assignment(f, (), {'b':3,'a':4}) == {'b':3, 'a':4}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
236 def f(b, a=1): pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
237 assert _arg_assignment(f, (), {'b':3}) == {'b':3, 'a':1}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
238 def f(b, a=1): a0=6
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
239 assert _arg_assignment(f, (2,), {}) == {'b':2, 'a':1}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
240
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
241 if 0:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
242 def test_arg_assignment_w_varargs():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
243 def f(b, c=1, *a, **kw): z=5
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
244 assert _arg_assignment(f, (3,), {}) == {'b':3, 'c':1, 'a':(), 'kw':{}}
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
245
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
246
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
247 class CtrlObjCacheWrapper(object):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
248
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
249 @classmethod
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
250 def decorate(cls, *args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
251 self = cls(*args, **kwargs)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
252 def rval(f):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
253 self.f = f
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
254 return rval
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
255 def parse_args(self, args, kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
256 """Return key, f_args, f_kwargs, by removing ctrl-cache related flags.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
257
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
258 The key is None or a hashable pair that identifies all the arguments to the function.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
259 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
260 ctrl_args = dict(
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
261 ctrl = None,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
262 ctrl_ignore_cache=False,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
263 ctrl_force_shallow_recompute=False,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
264 ctrl_force_deep_recompute=False,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
265 )
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
266
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
267 # remove the ctrl and ctrl_* arguments
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
268 # because they are not meant to be passed to 'f'
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
269 ctrl_kwds = [(k,v) for (k,v) in kwargs.iteritems()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
270 if k.startswith('ctrl')]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
271 ctrl_args.update(dict(ctrl_kwds))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
272 f_kwds = [(k,v) for (k,v) in kwargs.iteritems()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
273 if not k.startswith('ctrl')]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
274
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
275 # assignment is a dictionary with a complete specification of the effective arguments to f
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
276 # including default values, varargs, and varkwargs.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
277 assignment = _arg_assignment(self.f, args, dict(f_kwds))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
278
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
279 assignment_items = assignment.items()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
280 assignment_items.sort() #canonical ordering for parameters
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
281
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
282 # replace argument values with explicitly provided keys
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
283 assignment_key = [(k, kwargs.get('ctrl_key_%s'%k, v))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
284 for (k,v) in assignment_items]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
285
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
286 rval_key = ('fn_cache', self.f, tuple(assignment_key))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
287 try:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
288 hash(rval_key)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
289 except:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
290 rval_key = None
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
291 return rval_key, assignment, {}, ctrl_args
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
292
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
293 def __doc__(self):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
294 #TODO: Add documentation from self.f
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
295 return """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
296 Optional magic kwargs:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
297 ctrl - use this handle for cache/checkpointing
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
298 ctrl_key_%(paramname)s - specify a key to use for a cache lookup of this parameter
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
299 ctrl_ignore_cache - completely ignore the cache (but checkpointing can still work)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
300 ctrl_force_shallow_recompute - refresh the cache (but not of sub-calls)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
301 ctrl_force_deep_recompute - recursively refresh the cache
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
302 ctrl_nocopy - skip the usual copy of a cached return value
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
303 """
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
304 def __call__(self, *args, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
305 # N.B.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
306 # ctrl_force_deep_recompute
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
307 # can work by inspecting the call stack
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
308 # if any parent frame has a special variable set (e.g. _child_frame_ctrl_force_deep_recompute)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
309 # then it means this is a ctrl_force_deep_recompute too.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
310 key, f_args, f_kwargs, ctrl_args = self.parse_args(args, kwargs)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
311
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
312 ctrl = ctrl_args['ctrl']
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
313 if ctrl is None or ctrl_args['ctrl_ignore_cache']:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
314 return self.f(*f_args, **f_kwargs)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
315 if key:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
316 try:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
317 return self.get_cached_val(ctrl, key)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
318 except KeyError:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
319 pass
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
320 f_rval = self.f(*f_args, **f_kwargs)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
321 if key:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
322 f_rval = self.cache_val(ctrl, key, f_rval)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
323 return f_rval
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
324
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
325 def get_cached_val(self, ctrl, key):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
326 return ctrl.get(key)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
327 def cache_val(self, ctrl, key, val):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
328 ctrl.set(key, val)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
329 return val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
330
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
331 class NumpyCacheCtrl(CtrlObjCacheWrapper):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
332 def get_cached_val(self, ctrl, key):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
333 filename = ctrl.get(key)['npy_filename']
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
334 return numpy.load(filename)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
335 def cache_val(self, ctrl, key, val):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
336 try:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
337 filename = ctrl.get(key)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
338 except KeyError:
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
339 handle, filename = ctrl.open_uniq()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
340 handle.close()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
341 ctrl.set(key, dict(npy_filename=filename))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
342 numpy.save(filename, val)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
343 return val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
344
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
345 class PickleCacheCtrl(CtrlObjCacheWrapper):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
346 def __init__(self, protocol=0, **kwargs):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
347 self.protocol=protocol
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
348 super(PickleCacheCtrl, self).__init__(**kwargs)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
349 def get_cached_val(self, ctrl, key):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
350 return cPickle.loads(ctrl.get(key)['cPickle_str'])
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
351 def cache_val(self, ctrl, key, val):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
352 ctrl.set(key, dict(cPickle_str=cPickle.dumps(val)))
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
353 return val
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
354
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
355 @NumpyCacheCtrl.decorate()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
356 def get_raw_data(rows, cols, seed=67273):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
357 return numpy.random.RandomState(seed).randn(rows, cols)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
358
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
359 @NumpyCacheCtrl.decorate()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
360 def get_whitened_dataset(X, pca, max_components=5):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
361 return X[:,:max_components]
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
362
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
363 @PickleCacheCtrl.decorate(protocol=-1)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
364 def get_pca(X, max_components=100):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
365 return dict(
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
366 mean=0,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
367 eigvals=numpy.ones(X.shape[1]),
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
368 eigvecs=numpy.identity(X.shape[1])
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
369 )
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
370
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
371 @PickleCacheCtrl.decorate(protocol=-1)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
372 def train_mean_var_model(data, ctrl):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
373 mean = numpy.zeros(data.shape[1])
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
374 meansq = numpy.zeros(data.shape[1])
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
375 for i in xrange(data.shape[0]):
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
376 alpha = 1.0 / (i+1)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
377 mean += (1-alpha) * mean + data[i] * alpha
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
378 meansq += (1-alpha) * meansq + (data[i]**2) * alpha
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
379 ctrl.checkpoint()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
380 return (mean, meansq)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
381
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
382 def test_run_experiment():
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
383
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
384 # Could use db, or filesystem, or both, etc.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
385 # There would be generic ones, but the experimenter should be very aware of what is being
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
386 # cached where, when, and how. This is how results are stored and retrieved after all.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
387 # Cluster-friendly jobs should not use local files directly, but should store cached
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
388 # computations and results to such a database.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
389 # Different jobs should avoid using the same keys in the database because coordinating
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
390 # writes is difficult, and conflicts will inevitably arise.
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
391 ctrl = memory_ctrl_obj()
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
392
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
393 raw_data = get_raw_data(ctrl=ctrl)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
394 raw_data_key = ctrl.get_key(raw_data)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
395
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
396 pca = get_pca(raw_data, max_components=30, ctrl=ctrl)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
397 whitened_data = get_whitened_dataset(raw_data, pca, ctrl=ctrl)
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
398
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
399 mean_var = train_mean_var_model(
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
400 data=whitened_data+66,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
401 ctrl=ctrl,
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
402 ctrl_key_data=whitened_data) #tell that the temporary is tied to whitened_data
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
403
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
404 mean, var = mean_var
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
405
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
406 #TODO: Test that the cache actually worked!!
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
407
09ad2a4f663c adding new idea to arch_src
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
408