pylearn: pylearn/dbdict/newstuff.py annotate

annotate pylearn/dbdict/newstuff.py @ 597:c4579524baa6

revised newstuff.resolve to do more importing

author	james@X40
date	Tue, 13 Jan 2009 16:00:24 -0500
parents	f8d29730f146
children	e763711472a5

rev	line source
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	1
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	2 from __future__ import with_statement
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	3
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	4 from collections import defaultdict
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	5 import re, sys, inspect, os, signal, tempfile, shutil, socket
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	6 import traceback
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	7
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	8 import sql
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	9
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	10
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	11 ################################################################################
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	12 ### misc
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	13 ################################################################################
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	14
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	15 class DD(dict):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	16 def __getattr__(self, attr):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	17 return self[attr]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	18 def __setattr__(self, attr, value):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	19 self[attr] = value
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	20 def __str__(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	21 return 'DD%s' % dict(self)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	22 def __repr__(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	23 return str(self)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	24
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	25 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	26 ### resolve
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	27 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	28
597 c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	29 def resolve(name, try_import=True):
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	30 """
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	31 Resolve a string of the form X.Y...Z to a python object by repeatedly using getattr, and
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	32 __import__ to introspect objects (in this case X, then Y, etc. until finally Z is loaded).
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	33 """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	34 symbols = name.split('.')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	35 builder = __import__(symbols[0])
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	36 for sym in symbols[1:]:
597 c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	37 try:
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	38 builder = getattr(builder, sym)
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	39 except AttributeError, e:
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	40 if try_import:
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	41 __import__(builder.__name__, fromlist=[sym])
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	42 builder = getattr(builder, sym)
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	43 else:
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	44 raise e
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	45 return builder
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	46
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	47 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	48 ### dictionary
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	49 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	50
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	51 def convert(obj):
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	52 try:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	53 return eval(obj, {}, {})
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	54 except (NameError, SyntaxError):
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	55 return obj
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	56
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	57 def flatten(obj):
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	58 """nested dictionary -> flat dictionary with '.' notation """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	59 d = {}
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	60 def helper(d, prefix, obj):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	61 if isinstance(obj, (str, int, float)):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	62 d[prefix] = obj #convert(obj)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	63 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	64 if isinstance(obj, dict):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	65 subd = obj
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	66 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	67 subd = obj.state()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	68 subd['__builder__'] = '%s.%s' % (obj.__module__, obj.__class__.__name__)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	69 for k, v in subd.iteritems():
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	70 pfx = '.'.join([prefix, k]) if prefix else k
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	71 helper(d, pfx, v)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	72 helper(d, '', obj)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	73 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	74
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	75 def expand(d):
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	76 """inverse of flatten()"""
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	77 #def dd():
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	78 #return DD(dd)
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	79 struct = DD()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	80 for k, v in d.iteritems():
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	81 if k == '':
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	82 raise NotImplementedError()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	83 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	84 keys = k.split('.')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	85 current = struct
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	86 for k2 in keys[:-1]:
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	87 current = current.setdefault(k2, DD())
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	88 current[keys[-1]] = v #convert(v)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	89 return struct
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	90
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	91 def realize(d):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	92 if not isinstance(d, dict):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	93 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	94 d = dict((k, realize(v)) for k, v in d.iteritems())
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	95 if '__builder__' in d:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	96 builder = resolve(d.pop('__builder__'))
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	97 return builder(**d)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	98 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	99
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	100 def make(d):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	101 return realize(expand(d))
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	102
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	103 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	104 ### errors
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	105 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	106
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	107 class UsageError(Exception):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	108 pass
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	109
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	110 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	111 ### parsing and formatting
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	112 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	113
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	114 def parse(*strings):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	115 d = {}
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	116 for string in strings:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	117 s1 = re.split(' = ', string, 1)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	118 s2 = re.split(' :: ', string, 1)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	119 if len(s1) == 1 and len(s2) == 1:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	120 raise UsageError('Expected a keyword argument in place of "%s"' % s1[0])
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	121 elif len(s1) == 2:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	122 k, v = s1
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	123 v = convert(v)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	124 elif len(s2) == 2:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	125 k, v = s2
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	126 k += '.__builder__'
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	127 d[k] = v
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	128 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	129
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	130 def format_d(d, sep = '\n', space = True):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	131 d = flatten(d)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	132 pattern = "%s = %r" if space else "%s=%r"
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	133 return sep.join(pattern % (k, v) for k, v in d.iteritems())
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	134
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	135 def format_help(topic):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	136 if topic is None:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	137 return 'No help.'
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	138 elif isinstance(topic, str):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	139 help = topic
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	140 elif hasattr(topic, 'help'):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	141 help = topic.help()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	142 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	143 help = topic.__doc__
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	144 if not help:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	145 return 'No help.'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	146
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	147 ss = map(str.rstrip, help.split('\n'))
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	148 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	149 baseline = min([len(line) - len(line.lstrip()) for line in ss if line])
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	150 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	151 return 'No help.'
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	152 s = '\n'.join([line[baseline:] for line in ss])
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	153 s = re.sub(string = s, pattern = '\n{2,}', repl = '\n\n')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	154 s = re.sub(string = s, pattern = '(^\n)\|(\n$)', repl = '')
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	155
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	156 return s
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	157
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	158 ################################################################################
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	159 ### single channels
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	160 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	161
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	162 # try:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	163 # import greenlet
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	164 # except:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	165 # try:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	166 # from py import greenlet
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	167 # except:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	168 # print >>sys.stderr, 'the greenlet module is unavailable'
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	169 # greenlet = None
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	170
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	171
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	172 class Channel(object):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	173
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	174 COMPLETE = property(lambda s:None,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	175 doc=("Experiments should return this value to "
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	176 "indicate that they are done (if not done, return `Incomplete`"))
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	177 INCOMPLETE = property(lambda s:True,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	178 doc=("Experiments should return this value to indicate that "
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	179 "they are not done (if done return `COMPLETE`)"))
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	180
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	181 START = property(lambda s: 0,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	182 doc="dbdict.status == START means a experiment is ready to run")
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	183 RUNNING = property(lambda s: 1,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	184 doc="dbdict.status == RUNNING means a experiment is running on dbdict_hostname")
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	185 DONE = property(lambda s: 2,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	186 doc="dbdict.status == DONE means a experiment has completed (not necessarily successfully)")
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	187
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	188 # Methods to be used by the experiment to communicate with the channel
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	189
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	190 def save(self):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	191 """
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	192 Save the experiment's state to the various media supported by
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	193 the Channel.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	194 """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	195 raise NotImplementedError()
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	196
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	197 def switch(self, message = None):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	198 """
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	199 Called from the experiment to give the control back to the channel.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	200 The following return values are meaningful:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	201 * 'stop' -> the experiment must stop as soon as possible. It may save what
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	202 it needs to save. This occurs when SIGTERM or SIGINT are sent (or in
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	203 user-defined circumstances).
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	204 switch() may give the control to the user. In this case, the user may
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	205 resume the experiment by calling switch() again. If an argument is given
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	206 by the user, it will be relayed to the experiment.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	207 """
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	208 pass
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	209
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	210 def __call__(self, message = None):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	211 return self.switch(message)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	212
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	213 def save_and_switch(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	214 self.save()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	215 self.switch()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	216
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	217 # Methods to run the experiment
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	218
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	219 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	220 pass
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	221
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	222 def __enter__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	223 pass
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	224
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	225 def __exit__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	226 pass
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	227
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	228 def run(self):
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	229 pass
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	230
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	231
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	232
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	233 class JobError(Exception):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	234 RUNNING = 0
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	235 DONE = 1
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	236 NOJOB = 2
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	237
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	238
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	239 class SingleChannel(Channel):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	240
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	241 def __init__(self, experiment, state):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	242 self.experiment = experiment
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	243 self.state = state
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	244 self.feedback = None
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	245
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	246 def switch(self, message = None):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	247 feedback = self.feedback
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	248 self.feedback = None
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	249 return feedback
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	250
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	251 def run(self, force = False):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	252 self.setup()
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	253
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	254 status = self.state.dbdict.get('status', self.START)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	255 if status is self.DONE and not force:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	256 # If you want to disregard this, use the --force flag (not yet implemented)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	257 raise JobError(JobError.RUNNING,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	258 'The job has already completed.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	259 elif status is self.RUNNING and not force:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	260 raise JobError(JobError.DONE,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	261 'The job is already running.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	262 self.state.dbdict.status = self.RUNNING
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	263
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	264 v = self.COMPLETE
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	265 with self:
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	266 try:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	267 v = self.experiment(self.state, self)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	268 finally:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	269 self.state.dbdict.status = self.DONE if v is self.COMPLETE else self.START
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	270
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	271 return v
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	272
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	273 def on_sigterm(self, signo, frame):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	274 # SIGTERM handler. It is the experiment function's responsibility to
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	275 # call switch() often enough to get this feedback.
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	276 self.feedback = 'stop'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	277
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	278 def __enter__(self):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	279 # install a SIGTERM handler that asks the experiment function to return
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	280 # the next time it will call switch()
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	281 self.prev_sigterm = signal.getsignal(signal.SIGTERM)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	282 self.prev_sigint = signal.getsignal(signal.SIGINT)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	283 signal.signal(signal.SIGTERM, self.on_sigterm)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	284 signal.signal(signal.SIGINT, self.on_sigterm)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	285 return self
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	286
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	287 def __exit__(self, type, value, tb_traceback, save = True):
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	288 if type:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	289 try:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	290 raise type, value, tb_traceback
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	291 except:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	292 traceback.print_exc()
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	293 signal.signal(signal.SIGTERM, self.prev_sigterm)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	294 signal.signal(signal.SIGINT, self.prev_sigint)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	295 self.prev_sigterm = None
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	296 self.prev_sigint = None
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	297 if save:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	298 self.save()
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	299 return True
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	300
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	301
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	302 class StandardChannel(SingleChannel):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	303
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	304 def __init__(self, path, experiment, state, redirect_stdout = False, redirect_stderr = False):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	305 super(StandardChannel, self).__init__(experiment, state)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	306 self.path = os.path.realpath(path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	307 self.redirect_stdout = redirect_stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	308 self.redirect_stderr = redirect_stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	309
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	310 def save(self):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	311 with open(os.path.join(self.path, 'current.conf'), 'w') as current:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	312 current.write(format_d(self.state))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	313 current.write('\n')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	314
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	315 def __enter__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	316 self.old_cwd = os.getcwd()
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	317 os.chdir(self.path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	318 if self.redirect_stdout:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	319 self.old_stdout = sys.stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	320 sys.stdout = open('stdout', 'a')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	321 if self.redirect_stderr:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	322 self.old_stderr = sys.stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	323 sys.stderr = open('stderr', 'a')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	324 return super(StandardChannel, self).__enter__()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	325
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	326 def __exit__(self, type, value, traceback):
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	327 rval = super(StandardChannel, self).__exit__(type, value, traceback, save = False)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	328 if self.redirect_stdout:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	329 sys.stdout.close()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	330 sys.stdout = self.old_stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	331 if self.redirect_stderr:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	332 sys.stderr.close()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	333 sys.stderr = self.old_stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	334 os.chdir(self.old_cwd)
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	335 self.save()
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	336 return rval
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	337
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	338 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	339 if not os.path.isdir(self.path):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	340 os.makedirs(self.path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	341 with self:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	342 origf = os.path.join(self.path, 'orig.conf')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	343 if not os.path.isfile(origf):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	344 with open(origf, 'w') as orig:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	345 orig.write(format_d(self.state))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	346 orig.write('\n')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	347 currentf = os.path.join(self.path, 'current.conf')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	348 if os.path.isfile(currentf):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	349 with open(currentf, 'r') as current:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	350 self.state = expand(parse(*map(str.strip, current.readlines())))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	351
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	352
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	353 class RSyncException(Exception):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	354 pass
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	355
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	356 class RSyncChannel(StandardChannel):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	357
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	358 def __init__(self, path, remote_path, experiment, state, redirect_stdout = False, redirect_stderr = False):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	359 super(RSyncChannel, self).__init__(path, experiment, state, redirect_stdout, redirect_stderr)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	360
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	361 ssh_prefix='ssh://'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	362 if remote_path.startswith(ssh_prefix):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	363 remote_path = remote_path[len(ssh_prefix):]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	364 colon_pos = remote_path.find(':')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	365 self.host = remote_path[:colon_pos]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	366 self.remote_path = remote_path[colon_pos+1:]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	367 else:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	368 self.host = ''
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	369 self.remote_path = os.path.realpath(remote_path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	370
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	371 def rsync(self, direction):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	372 """The directory at which experiment-related files are stored.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	373 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	374
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	375 path = self.path
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	376
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	377 remote_path = self.remote_path
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	378 if self.host:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	379 remote_path = ':'.join([self.host, remote_path])
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	380
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	381 # TODO: use something more portable than os.system
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	382 if direction == 'push':
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	383 rsync_cmd = 'rsync -ac "%s/" "%s/"' % (path, remote_path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	384 elif direction == 'pull':
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	385 rsync_cmd = 'rsync -ac "%s/" "%s/"' % (remote_path, path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	386 else:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	387 raise RSyncException('invalid direction', direction)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	388
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	389 rsync_rval = os.system(rsync_cmd)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	390 if rsync_rval != 0:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	391 raise RSyncException('rsync failure', (rsync_rval, rsync_cmd))
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	392
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	393 def touch(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	394 if self.host:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	395 host = self.host
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	396 touch_cmd = ('ssh %(host)s "mkdir -p \'%(path)s\'"' % dict(host = self.host,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	397 path = self.remote_path))
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	398 else:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	399 touch_cmd = ("mkdir -p '%(path)s'" % dict(path = self.remote_path))
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	400 # print "ECHO", touch_cmd
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	401 touch_rval = os.system(touch_cmd)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	402 if 0 != touch_rval:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	403 raise Exception('touch failure', (touch_rval, touch_cmd))
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	404
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	405 def pull(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	406 return self.rsync('pull')
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	407
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	408 def push(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	409 return self.rsync('push')
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	410
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	411 def save(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	412 super(RSyncChannel, self).save()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	413 self.push()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	414
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	415 def setup(self):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	416 self.touch()
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	417 self.pull()
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	418 super(RSyncChannel, self).setup()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	419
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	420
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	421 class DBRSyncChannel(RSyncChannel):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	422
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	423 RESTART_PRIORITY = 2.0
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	424
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	425 def __init__(self, username, password, hostname, dbname, tablename, path, remote_root, redirect_stdout = False, redirect_stderr = False):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	426 self.username, self.password, self.hostname, self.dbname, self.tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	427 = username, password, hostname, dbname, tablename
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	428
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	429 self.db = sql.postgres_serial(
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	430 user = self.username,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	431 password = self.password,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	432 host = self.hostname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	433 database = self.dbname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	434 table_prefix = self.tablename)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	435
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	436 self.dbstate = sql.book_dct_postgres_serial(self.db)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	437 if self.dbstate is None:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	438 raise JobError(JobError.NOJOB,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	439 'No job was found to run.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	440
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	441 try:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	442 state = expand(self.dbstate)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	443 experiment = resolve(state.dbdict.experiment)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	444 remote_path = os.path.join(remote_root, self.dbname, self.tablename, str(self.dbstate.id))
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	445 super(DBRSyncChannel, self).__init__(path, remote_path, experiment, state, redirect_stdout, redirect_stderr)
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	446 except:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	447 self.dbstate['dbdict.status'] = self.DONE
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	448 raise
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	449
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	450 def save(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	451 super(DBRSyncChannel, self).save()
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	452 self.dbstate.update(flatten(self.state))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	453
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	454 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	455 # Extract a single experiment from the table that is not already running.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	456 # set self.experiment and self.state
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	457 super(DBRSyncChannel, self).setup()
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	458 self.state.dbdict.sql.host_name = socket.gethostname()
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	459 self.state.dbdict.sql.host_workdir = self.path
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	460 self.dbstate.update(flatten(self.state))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	461
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	462 def run(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	463 # We pass the force flag as True because the status flag is
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	464 # already set to RUNNING by book_dct in __init__
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	465 v = super(DBRSyncChannel, self).run(force = True)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	466 if v is self.INCOMPLETE and self.state.dbdict.sql.priority != self.RESTART_PRIORITY:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	467 self.state.dbdict.sql.priority = self.RESTART_PRIORITY
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	468 self.save()
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	469 return v
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	470
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	471
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	472
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	473 ################################################################################
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	474 ### running
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	475 ################################################################################
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	476
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	477 import optparse
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	478 OptionParser = optparse.OptionParser
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	479 # class OptionParser(optparse.OptionParser):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	480 # def error(self, message):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	481 # pass
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	482
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	483 def parse_and_run(command, arguments):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	484 parser, runner = runner_registry.get(command, (None, None))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	485 if not runner:
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	486 raise UsageError('Unknown runner: "%s"' % command)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	487 if parser:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	488 options, arguments = parser.parse_args(arguments)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	489 else:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	490 options = optparse.Values()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	491 run(runner, [options] + arguments)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	492
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	493 def run(runner, arguments):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	494 argspec = inspect.getargspec(runner)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	495 minargs = len(argspec[0])-(len(argspec[3]) if argspec[3] else 0)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	496 maxargs = len(argspec[0])
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	497 if minargs > len(arguments) or maxargs < len(arguments) and not argspec[1]:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	498 s = format_help(runner)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	499 raise UsageError(s)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	500 runner(*arguments)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	501
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	502 runner_registry = dict()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	503
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	504
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	505 parser_cmdline = OptionParser(usage = '%prog cmdline [options] <experiment> <parameters>')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	506 parser_cmdline.add_option('-f', '--force', action = 'store_true', dest = 'force', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	507 help = 'force running the experiment even if it is already running or completed')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	508 parser_cmdline.add_option('--redirect-stdout', action = 'store_true', dest = 'redirect_stdout', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	509 help = 'redirect stdout to the workdir/stdout file')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	510 parser_cmdline.add_option('--redirect-stderr', action = 'store_true', dest = 'redirect_stderr', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	511 help = 'redirect stderr to the workdir/stdout file')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	512 parser_cmdline.add_option('-r', '--redirect', action = 'store_true', dest = 'redirect', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	513 help = 'redirect stdout and stderr to the workdir/stdout and workdir/stderr files')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	514 parser_cmdline.add_option('-w', '--workdir', action = 'store', dest = 'workdir', default = None,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	515 help = 'the working directory in which to run the experiment')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	516
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	517 def runner_cmdline(options, experiment, *strings):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	518 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	519 Start an experiment with parameters given on the command line.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	520
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	521 Usage: cmdline [options] <experiment> <parameters>
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	522
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	523 Run an experiment with parameters provided on the command
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	524 line. See the help topics for experiment and parameters for
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	525 syntax information.
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	526
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	527 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	528 dbdict-run cmdline mymodule.my_experiment \\
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	529 stopper::pylearn.stopper.nsteps \\ # use pylearn.stopper.nsteps
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	530 stopper.n=10000 \\ # the argument "n" of nsteps is 10000
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	531 lr=0.03
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	532 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	533 state = expand(parse(*strings))
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	534 state.setdefault('dbdict', DD()).experiment = experiment
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	535 experiment = resolve(experiment)
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	536 workdir = options.workdir or format_d(state, sep=',', space = False)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	537 channel = StandardChannel(workdir,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	538 experiment, state,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	539 redirect_stdout = options.redirect or options.redirect_stdout,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	540 redirect_stderr = options.redirect or options.redirect_stderr)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	541 channel.run(force = options.force)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	542
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	543 runner_registry['cmdline'] = (parser_cmdline, runner_cmdline)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	544
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	545
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	546
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	547 parser_sqlschedule = OptionParser(usage = '%prog sqlschedule [options] <tablepath> <experiment> <parameters>')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	548
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	549 def runner_sqlschedule(options, dbdescr, experiment, *strings):
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	550 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	551 Schedule a job to run using the sql command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	552
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	553 Usage: sqlschedule <tablepath> <experiment> <parameters>
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	554
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	555 See the experiment and parameters topics for more information about
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	556 these parameters.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	557
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	558 Assuming that a postgres database is running on `host`, contains a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	559 database called `dbname` and that `user` has the permissions to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	560 create, read and modify tables on that database, tablepath should
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	561 be of the following form:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	562 postgres://user:pass@host/dbname/tablename
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	563
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	564 If no table is named `tablename`, one will be created
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	565 automatically. The state corresponding to the experiment and
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	566 parameters specified in the command will be saved in the database,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	567 but no experiment will be run.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	568
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	569 To run an experiment scheduled using sqlschedule, see the sql
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	570 command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	571
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	572 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	573 dbdict-run sqlschedule postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	574 mymodule.my_experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	575 stopper::pylearn.stopper.nsteps \\ # use pylearn.stopper.nsteps
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	576 stopper.n=10000 \\ # the argument "n" of nsteps is 10000
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	577 lr=0.03
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	578 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	579
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	580 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	581 username, password, hostname, dbname, tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	582 = sql.parse_dbstring(dbdescr)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	583 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	584 raise UsageError('Wrong syntax for dbdescr')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	585
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	586 db = sql.postgres_serial(
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	587 user = username,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	588 password = password,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	589 host = hostname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	590 database = dbname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	591 table_prefix = tablename)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	592
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	593 state = parse(*strings)
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	594 try:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	595 resolve(experiment)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	596 except:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	597 raise UsageError('The first parameter to sqlschedule must be a valid, importable symbol.')
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	598 state['dbdict.experiment'] = experiment
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	599 sql.add_experiments_to_db([state], db, verbose = 1)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	600
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	601 runner_registry['sqlschedule'] = (parser_sqlschedule, runner_sqlschedule)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	602
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	603
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	604
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	605 parser_sql = OptionParser(usage = '%prog sql [options] <tablepath> <exproot>')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	606 parser_sql.add_option('-n', dest = 'n', type = 'int', default = 1,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	607 help = 'Run N experiments sequentially (default 1) '
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	608 '(if N is <= 0, runs as many experiments as possible).')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	609
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	610 def runner_sql(options, dbdescr, exproot):
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	611 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	612 Run jobs from a sql table.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	613
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	614 Usage: sql <tablepath> <exproot>
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	615
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	616 The jobs should be scheduled first with the sqlschedule command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	617
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	618 Assuming that a postgres database is running on `host`, contains a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	619 database called `dbname` and that `user` has the permissions to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	620 create, read and modify tables on that database, tablepath should
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	621 be of the following form:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	622 postgres://user:pass@host/dbname/tablename
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	623
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	624 exproot can be a local path or a remote path. Examples of exproots:
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	625 /some/local/path
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	626 ssh://some_host:/some/remote/path # relative to the filesystem root
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	627 ssh://some_host:other/remote/path # relative to the HOME on some_host
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	628
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	629 The exproot will contain a subdirectory hierarchy corresponding to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	630 the dbname, tablename and job id which is a unique integer.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	631
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	632 The sql runner will pick any job in the table which is not running
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	633 and is not done and will terminate when that job ends. You may call
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	634 the same command multiple times, sequentially or in parallel, to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	635 run as many unfinished jobs as have been scheduled in that table
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	636 with sqlschedule.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	637
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	638 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	639 dbdict-run sql \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	640 postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	641 ssh://central_host:myexperiments
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	642 """
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	643 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	644 username, password, hostname, dbname, tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	645 = sql.parse_dbstring(dbdescr)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	646 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	647 raise UsageError('Wrong syntax for dbdescr')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	648
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	649 n = options.n if options.n else -1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	650 nrun = 0
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	651 try:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	652 while n != 0:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	653 workdir = tempfile.mkdtemp()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	654 #print 'wdir', workdir
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	655 channel = DBRSyncChannel(username, password, hostname, dbname, tablename,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	656 workdir,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	657 exproot,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	658 redirect_stdout = True,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	659 redirect_stderr = True)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	660 channel.run()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	661 shutil.rmtree(workdir, ignore_errors=True)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	662 n -= 1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	663 nrun += 1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	664 except JobError, e:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	665 if e.args[0] == JobError.NOJOB:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	666 print 'No more jobs to run (run %i jobs)' % nrun
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	667
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	668 runner_registry['sql'] = (parser_sql, runner_sql)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	669
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	670
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	671
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	672
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	673
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	674 def runner_help(options, topic = None):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	675 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	676 Get help for a topic.
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	677
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	678 Usage: help <topic>
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	679 """
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	680 def bold(x):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	681 return '\033[1m%s\033[0m' % x
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	682 if topic is None:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	683 print bold('Topics: (use help <topic> for more info)')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	684 print 'example Example of defining and running an experiment.'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	685 print 'experiment How to define an experiment.'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	686 print 'parameters How to list the parameters for an experiment.'
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	687 print
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	688 print bold('Available commands: (use help <command> for more info)')
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	689 for name, command in sorted(runner_registry.iteritems()):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	690 print name.ljust(20), format_help(command).split('\n')[0]
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	691 return
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	692 elif topic == 'experiment':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	693 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	694
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	695 dbdict-run serves to run experiments. To define an experiment, you
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	696 only have to define a function respecting the following protocol in
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	697 a python file or module:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	698
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	699 def my_experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	700 # experiment code goes here
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	701
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	702 The return value of my_experiment may be channel.COMPLETE or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	703 channel.INCOMPLETE. If the latter is returned, the experiment may
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	704 be resumed at a later point. Note that the return value `None`
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	705 is interpreted as channel.COMPLETE.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	706
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	707 If a command defined by dbdict-run has an <experiment> parameter,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	708 that parameter must be a string such that it could be used in a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	709 python import statement to import the my_experiment function. For
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	710 example if you defined my_experiment in my_module.py, you can pass
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	711 'my_module.my_experiment' as the experiment parameter.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	712
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	713 When entering my_experiment, the current working directory will be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	714 set for you to a directory specially created for the experiment.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	715 The location and name of that directory vary depending on which
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	716 dbdict-run command you run. You may create logs, save files, pictures,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	717 results, etc. in it.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	718
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	719 state is an object containing the parameters given to the experiment.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	720 For example, if you run the followinc command:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	721
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	722 dbdict-run cmdline my_module.my_experiment a.x=6
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	723
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	724 `state.a.x` will contain the integer 6, and so will `state['a']['x']`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	725 If the state is changed, it will be saved when the experiment ends
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	726 or when channel.save() is called. The next time the experiment is run
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	727 with the same working directory, the modified state will be provided.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	728
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	729 It is not recommended to store large amounts of data in the state. It
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	730 should be limited to scalar or string parameters. Results such as
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	731 weight matrices should be stored in files in the working directory.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	732
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	733 channel is an object with the following important methods:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	734
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	735 - channel.switch() (or channel()) will give the control back to the
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	736 user, if it is appropriate to do so. If a call to channel.switch()
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	737 returns the string 'stop', it typically means that the signal
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	738 SIGTERM (or SIGINT) was received. Therefore, the experiment may be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	739 killed soon, so it should save and return True or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	740 channel.INCOMPLETE so it can be resumed later. This should be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	741 checked periodically or data loss may be incurred.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	742
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	743 - channel.save() will save the current state. It is automatically
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	744 called when the function returns, but it is a good idea to do it
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	745 periodically.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	746
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	747 - channel.save_and_switch() is an useful shortcut to do both operations
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	748 described above.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	749 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	750
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	751 elif topic == 'parameters':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	752 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	753 If a command takes <parameters> arguments, the arguments should each
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	754 take one of the following forms:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	755
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	756 key=value
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	757
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	758 Set a parameter with name `key` to `value`. The value will be casted
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	759 to an appropriate type automatically and it will be accessible to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	760 the experiment using `state.key`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	761
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	762 If `key` is a dotted name, the value will be set in nested
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	763 dictionaries corresponding to each part.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	764
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	765 Examples:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	766 a=1 state.a <- 1
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	767 b=2.3 state.b <- 2.3
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	768 c.d="hello" state.c.d <- "hello"
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	769
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	770 key::builder
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	771
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	772 This is equivalent to key.__builder__=builder.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	773
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	774 The builder should be a symbol that can be used with import or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	775 __import__ and it should be callable.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	776
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	777 If a key has a builder defined, the experiment code may easily make
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	778 an object out of it using the `make` function. `obj = make(state.key)`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	779 This will call the builder on the substate corresponding to state.key,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	780 as will be made clear in the example:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	781
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	782 Example:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	783 regexp::re.compile
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	784 regexp.pattern='a.*c'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	785
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	786 from pylearn.dbdict.newstuff import make
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	787 def experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	788 regexp = make(state.regexp) # regexp is now re.compile(pattern = 'a.*c')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	789 print regexp.sub('blahblah', 'hello abbbbc there')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	790
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	791 If the above experiment was called with the state produced by the
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	792 parameters in the example, it would print 'hello blahblah there'.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	793 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	794
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	795 elif topic == 'example':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	796 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	797 Example of an experiment that trains some model for 100000 iterations:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	798
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	799 # defined in: my_experiments.py
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	800 def experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	801 try:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	802 model = cPickle.load(open('model', 'r'))
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	803 except:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	804 model = my_model(state.some_param, state.other_param)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	805 state.n = 0
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	806 dataset = my_dataset(skipto = state.n)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	807 for i in xrange(100000 - state.n):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	808 model.update(dataset.next())
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	809 if i and i % 1000 == 0:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	810 if channel.save_and_switch() == 'stop':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	811 state.n += i + 1
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	812 rval = channel.INCOMPLETE
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	813 break
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	814 else:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	815 state.result = model.cost(some_test_set)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	816 rval = channel.COMPLETE
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	817 cPickle.dump(model, open('model', 'w'))
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	818 return rval
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	819
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	820 And then you could run it this way:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	821
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	822 dbdict-run cmdline my_experiments.experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	823 some_param=1 \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	824 other_param=0.4
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	825
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	826 Or this way:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	827
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	828 dbdict-run sqlschedule postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	829 my_experiments.experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	830 some_param=1 \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	831 other_param=0.4
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	832
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	833 dbdict-run sql postgres://user:pass@host/dbname/tablename exproot
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	834
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	835 You need to make sure that the module `my_experiments` is accessible
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	836 from python. You can check with the command
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	837
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	838 $ python -m my_experiments
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	839 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	840 else:
586 5020b12e87ee fixed broken help Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 584 diff changeset	841 helptext = runner_registry.get(topic, None)[1]
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	842 print format_help(helptext)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	843
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	844 runner_registry['help'] = (None, runner_help)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	845
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	846 ################################################################################
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	847 ### main
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	848 ################################################################################
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	849
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	850 def run_cmdline():
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	851 try:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	852 if len(sys.argv) <= 1:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	853 raise UsageError('Usage: %s <run_type> [<arguments>*]' % sys.argv[0])
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	854 cmd = None
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	855 args = []
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	856 for arg in sys.argv[1:]:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	857 if cmd is not None or arg.startswith('-'):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	858 args.append(arg)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	859 else:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	860 cmd = arg
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	861 parse_and_run(cmd, args)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	862 except UsageError, e:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	863 print 'Usage error:'
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	864 print e
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	865
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	866 if __name__ == '__main__':
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	867 run_cmdline()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	868
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	869

Mercurial > pylearn

annotate pylearn/dbdict/newstuff.py @ 597:c4579524baa6