pylearn: pylearn/dbdict/newstuff.py annotate

annotate pylearn/dbdict/newstuff.py @ 609:7cee8c7f0449

added -f flag to sqlschedule and sqlschedule_filemerge

author	Olivier Breuleux <breuleuo@iro.umontreal.ca>
date	Fri, 16 Jan 2009 15:53:22 -0500
parents	52a99d83f06d
children	e8cb4bde30a7

rev	line source
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	1
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	2 from __future__ import with_statement
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	3
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	4 from collections import defaultdict
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	5 import re, sys, inspect, os, signal, tempfile, shutil, socket
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	6 import traceback
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	7
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	8 import sql
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	9
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	10
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	11 ################################################################################
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	12 ### misc
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	13 ################################################################################
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	14
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	15 class DD(dict):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	16 def __getattr__(self, attr):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	17 return self[attr]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	18 def __setattr__(self, attr, value):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	19 self[attr] = value
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	20 def __str__(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	21 return 'DD%s' % dict(self)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	22 def __repr__(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	23 return str(self)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	24
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	25 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	26 ### resolve
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	27 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	28
597 c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	29 def resolve(name, try_import=True):
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	30 """
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	31 Resolve a string of the form X.Y...Z to a python object by repeatedly using getattr, and
c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	32 __import__ to introspect objects (in this case X, then Y, etc. until finally Z is loaded).
599 bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	33
597 c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	34 """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	35 symbols = name.split('.')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	36 builder = __import__(symbols[0])
599 bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	37 try:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	38 for sym in symbols[1:]:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	39 try:
597 c4579524baa6 revised newstuff.resolve to do more importing james@X40 parents: 590 diff changeset	40 builder = getattr(builder, sym)
599 bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	41 except AttributeError, e:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	42 if try_import:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	43 __import__(builder.__name__, fromlist=[sym])
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	44 builder = getattr(builder, sym)
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	45 else:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	46 raise e
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	47 except (AttributeError, ImportError), e:
bd777e960c7c misc small changes James Bergstra <bergstrj@iro.umontreal.ca> parents: 598 diff changeset	48 raise type(e)('Failed to resolve compound symbol %s' % name, e)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	49 return builder
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	50
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	51 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	52 ### dictionary
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	53 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	54
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	55 def convert(obj):
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	56 try:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	57 return eval(obj, {}, {})
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	58 except (NameError, SyntaxError):
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	59 return obj
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	60
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	61 def flatten(obj):
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	62 """nested dictionary -> flat dictionary with '.' notation """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	63 d = {}
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	64 def helper(d, prefix, obj):
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	65 if isinstance(obj, (str, int, float, list, tuple)):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	66 d[prefix] = obj #convert(obj)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	67 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	68 if isinstance(obj, dict):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	69 subd = obj
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	70 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	71 subd = obj.state()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	72 subd['__builder__'] = '%s.%s' % (obj.__module__, obj.__class__.__name__)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	73 for k, v in subd.iteritems():
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	74 pfx = '.'.join([prefix, k]) if prefix else k
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	75 helper(d, pfx, v)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	76 helper(d, '', obj)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	77 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	78
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	79 def expand(d):
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	80 """inverse of flatten()"""
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	81 #def dd():
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	82 #return DD(dd)
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	83 struct = DD()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	84 for k, v in d.iteritems():
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	85 if k == '':
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	86 raise NotImplementedError()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	87 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	88 keys = k.split('.')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	89 current = struct
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	90 for k2 in keys[:-1]:
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	91 current = current.setdefault(k2, DD())
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	92 current[keys[-1]] = v #convert(v)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	93 return struct
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	94
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	95 def realize(d):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	96 if not isinstance(d, dict):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	97 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	98 d = dict((k, realize(v)) for k, v in d.iteritems())
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	99 if '__builder__' in d:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	100 builder = resolve(d.pop('__builder__'))
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	101 return builder(**d)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	102 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	103
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	104 def make(d):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	105 return realize(expand(d))
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	106
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	107 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	108 ### errors
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	109 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	110
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	111 class UsageError(Exception):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	112 pass
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	113
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	114 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	115 ### parsing and formatting
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	116 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	117
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	118 def parse(*strings):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	119 d = {}
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	120 for string in strings:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	121 s1 = re.split(' = ', string, 1)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	122 s2 = re.split(' :: ', string, 1)
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	123 if len(s1) == 1 and len(s2) == 1:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	124 raise UsageError('Expected a keyword argument in place of "%s"' % s1[0])
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	125 elif len(s1) == 2:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	126 k, v = s1
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	127 v = convert(v)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	128 elif len(s2) == 2:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	129 k, v = s2
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	130 k += '.__builder__'
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	131 d[k] = v
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	132 return d
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	133
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	134 def format_d(d, sep = '\n', space = True):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	135 d = flatten(d)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	136 pattern = "%s = %r" if space else "%s=%r"
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	137 return sep.join(pattern % (k, v) for k, v in d.iteritems())
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	138
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	139 def format_help(topic):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	140 if topic is None:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	141 return 'No help.'
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	142 elif isinstance(topic, str):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	143 help = topic
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	144 elif hasattr(topic, 'help'):
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	145 help = topic.help()
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	146 else:
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	147 help = topic.__doc__
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	148 if not help:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	149 return 'No help.'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	150
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	151 ss = map(str.rstrip, help.split('\n'))
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	152 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	153 baseline = min([len(line) - len(line.lstrip()) for line in ss if line])
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	154 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	155 return 'No help.'
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	156 s = '\n'.join([line[baseline:] for line in ss])
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	157 s = re.sub(string = s, pattern = '\n{2,}', repl = '\n\n')
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	158 s = re.sub(string = s, pattern = '(^\n)\|(\n$)', repl = '')
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	159
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	160 return s
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	161
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	162 ################################################################################
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	163 ### single channels
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	164 ################################################################################
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	165
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	166 # try:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	167 # import greenlet
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	168 # except:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	169 # try:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	170 # from py import greenlet
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	171 # except:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	172 # print >>sys.stderr, 'the greenlet module is unavailable'
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	173 # greenlet = None
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	174
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	175
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	176 class Channel(object):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	177
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	178 COMPLETE = property(lambda s:None,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	179 doc=("Experiments should return this value to "
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	180 "indicate that they are done (if not done, return `Incomplete`"))
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	181 INCOMPLETE = property(lambda s:True,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	182 doc=("Experiments should return this value to indicate that "
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	183 "they are not done (if done return `COMPLETE`)"))
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	184
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	185 START = property(lambda s: 0,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	186 doc="dbdict.status == START means a experiment is ready to run")
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	187 RUNNING = property(lambda s: 1,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	188 doc="dbdict.status == RUNNING means a experiment is running on dbdict_hostname")
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	189 DONE = property(lambda s: 2,
6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	190 doc="dbdict.status == DONE means a experiment has completed (not necessarily successfully)")
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	191
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	192 # Methods to be used by the experiment to communicate with the channel
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	193
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	194 def save(self):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	195 """
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	196 Save the experiment's state to the various media supported by
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	197 the Channel.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	198 """
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	199 raise NotImplementedError()
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	200
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	201 def switch(self, message = None):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	202 """
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	203 Called from the experiment to give the control back to the channel.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	204 The following return values are meaningful:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	205 * 'stop' -> the experiment must stop as soon as possible. It may save what
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	206 it needs to save. This occurs when SIGTERM or SIGINT are sent (or in
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	207 user-defined circumstances).
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	208 switch() may give the control to the user. In this case, the user may
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	209 resume the experiment by calling switch() again. If an argument is given
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	210 by the user, it will be relayed to the experiment.
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	211 """
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	212 pass
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	213
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	214 def __call__(self, message = None):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	215 return self.switch(message)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	216
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	217 def save_and_switch(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	218 self.save()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	219 self.switch()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	220
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	221 # Methods to run the experiment
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	222
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	223 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	224 pass
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	225
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	226 def __enter__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	227 pass
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	228
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	229 def __exit__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	230 pass
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	231
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	232 def run(self):
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	233 pass
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	234
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	235
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	236
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	237 class JobError(Exception):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	238 RUNNING = 0
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	239 DONE = 1
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	240 NOJOB = 2
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	241
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	242
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	243 class SingleChannel(Channel):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	244
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	245 def __init__(self, experiment, state):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	246 self.experiment = experiment
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	247 self.state = state
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	248 self.feedback = None
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	249
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	250 def switch(self, message = None):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	251 feedback = self.feedback
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	252 self.feedback = None
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	253 return feedback
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	254
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	255 def run(self, force = False):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	256 self.setup()
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	257
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	258 status = self.state.dbdict.get('status', self.START)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	259 if status is self.DONE and not force:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	260 # If you want to disregard this, use the --force flag (not yet implemented)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	261 raise JobError(JobError.RUNNING,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	262 'The job has already completed.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	263 elif status is self.RUNNING and not force:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	264 raise JobError(JobError.DONE,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	265 'The job is already running.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	266 self.state.dbdict.status = self.RUNNING
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	267
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	268 v = self.COMPLETE
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	269 with self:
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	270 try:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	271 v = self.experiment(self.state, self)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	272 finally:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	273 self.state.dbdict.status = self.DONE if v is self.COMPLETE else self.START
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	274
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	275 return v
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	276
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	277 def on_sigterm(self, signo, frame):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	278 # SIGTERM handler. It is the experiment function's responsibility to
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	279 # call switch() often enough to get this feedback.
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	280 self.feedback = 'stop'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	281
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	282 def __enter__(self):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	283 # install a SIGTERM handler that asks the experiment function to return
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	284 # the next time it will call switch()
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	285 self.prev_sigterm = signal.getsignal(signal.SIGTERM)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	286 self.prev_sigint = signal.getsignal(signal.SIGINT)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	287 signal.signal(signal.SIGTERM, self.on_sigterm)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	288 signal.signal(signal.SIGINT, self.on_sigterm)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	289 return self
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	290
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	291 def __exit__(self, type, value, tb_traceback, save = True):
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	292 if type:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	293 try:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	294 raise type, value, tb_traceback
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	295 except:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	296 traceback.print_exc()
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	297 signal.signal(signal.SIGTERM, self.prev_sigterm)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	298 signal.signal(signal.SIGINT, self.prev_sigint)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	299 self.prev_sigterm = None
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	300 self.prev_sigint = None
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	301 if save:
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	302 self.save()
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	303 return True
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	304
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	305
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	306 class StandardChannel(SingleChannel):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	307
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	308 def __init__(self, path, experiment, state, redirect_stdout = False, redirect_stderr = False):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	309 super(StandardChannel, self).__init__(experiment, state)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	310 self.path = os.path.realpath(path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	311 self.redirect_stdout = redirect_stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	312 self.redirect_stderr = redirect_stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	313
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	314 def save(self):
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	315 with open(os.path.join(self.path, 'current.conf'), 'w') as current:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	316 current.write(format_d(self.state))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	317 current.write('\n')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	318
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	319 def __enter__(self):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	320 self.old_cwd = os.getcwd()
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	321 os.chdir(self.path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	322 if self.redirect_stdout:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	323 self.old_stdout = sys.stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	324 sys.stdout = open('stdout', 'a')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	325 if self.redirect_stderr:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	326 self.old_stderr = sys.stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	327 sys.stderr = open('stderr', 'a')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	328 return super(StandardChannel, self).__enter__()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	329
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	330 def __exit__(self, type, value, traceback):
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	331 rval = super(StandardChannel, self).__exit__(type, value, traceback, save = False)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	332 if self.redirect_stdout:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	333 sys.stdout.close()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	334 sys.stdout = self.old_stdout
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	335 if self.redirect_stderr:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	336 sys.stderr.close()
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	337 sys.stderr = self.old_stderr
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	338 os.chdir(self.old_cwd)
590 f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	339 self.save()
f8d29730f146 fixed the behavior when raising errors and redirecting stderr Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 589 diff changeset	340 return rval
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	341
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	342 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	343 if not os.path.isdir(self.path):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	344 os.makedirs(self.path)
567 d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	345 with self:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	346 origf = os.path.join(self.path, 'orig.conf')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	347 if not os.path.isfile(origf):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	348 with open(origf, 'w') as orig:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	349 orig.write(format_d(self.state))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	350 orig.write('\n')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	351 currentf = os.path.join(self.path, 'current.conf')
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	352 if os.path.isfile(currentf):
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	353 with open(currentf, 'r') as current:
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	354 self.state = expand(parse(*map(str.strip, current.readlines())))
d88c35e8f83a another checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 566 diff changeset	355
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	356
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	357 class RSyncException(Exception):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	358 pass
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	359
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	360 class RSyncChannel(StandardChannel):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	361
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	362 def __init__(self, path, remote_path, experiment, state, redirect_stdout = False, redirect_stderr = False):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	363 super(RSyncChannel, self).__init__(path, experiment, state, redirect_stdout, redirect_stderr)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	364
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	365 ssh_prefix='ssh://'
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	366 if remote_path.startswith(ssh_prefix):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	367 remote_path = remote_path[len(ssh_prefix):]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	368 colon_pos = remote_path.find(':')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	369 self.host = remote_path[:colon_pos]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	370 self.remote_path = remote_path[colon_pos+1:]
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	371 else:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	372 self.host = ''
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	373 self.remote_path = os.path.realpath(remote_path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	374
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	375 def rsync(self, direction):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	376 """The directory at which experiment-related files are stored.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	377 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	378
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	379 path = self.path
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	380
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	381 remote_path = self.remote_path
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	382 if self.host:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	383 remote_path = ':'.join([self.host, remote_path])
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	384
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	385 # TODO: use something more portable than os.system
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	386 if direction == 'push':
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	387 rsync_cmd = 'rsync -ac "%s/" "%s/"' % (path, remote_path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	388 elif direction == 'pull':
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	389 rsync_cmd = 'rsync -ac "%s/" "%s/"' % (remote_path, path)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	390 else:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	391 raise RSyncException('invalid direction', direction)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	392
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	393 rsync_rval = os.system(rsync_cmd)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	394 if rsync_rval != 0:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	395 raise RSyncException('rsync failure', (rsync_rval, rsync_cmd))
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	396
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	397 def touch(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	398 if self.host:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	399 host = self.host
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	400 touch_cmd = ('ssh %(host)s "mkdir -p \'%(path)s\'"' % dict(host = self.host,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	401 path = self.remote_path))
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	402 else:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	403 touch_cmd = ("mkdir -p '%(path)s'" % dict(path = self.remote_path))
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	404 # print "ECHO", touch_cmd
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	405 touch_rval = os.system(touch_cmd)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	406 if 0 != touch_rval:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	407 raise Exception('touch failure', (touch_rval, touch_cmd))
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	408
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	409 def pull(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	410 return self.rsync('pull')
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	411
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	412 def push(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	413 return self.rsync('push')
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	414
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	415 def save(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	416 super(RSyncChannel, self).save()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	417 self.push()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	418
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	419 def setup(self):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	420 self.touch()
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	421 self.pull()
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	422 super(RSyncChannel, self).setup()
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	423
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	424
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	425 class DBRSyncChannel(RSyncChannel):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	426
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	427 RESTART_PRIORITY = 2.0
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	428
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	429 def __init__(self, username, password, hostname, dbname, tablename, path, remote_root, redirect_stdout = False, redirect_stderr = False):
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	430 self.username, self.password, self.hostname, self.dbname, self.tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	431 = username, password, hostname, dbname, tablename
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	432
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	433 self.db = sql.postgres_serial(
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	434 user = self.username,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	435 password = self.password,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	436 host = self.hostname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	437 database = self.dbname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	438 table_prefix = self.tablename)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	439
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	440 self.dbstate = sql.book_dct_postgres_serial(self.db)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	441 if self.dbstate is None:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	442 raise JobError(JobError.NOJOB,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	443 'No job was found to run.')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	444
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	445 try:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	446 state = expand(self.dbstate)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	447 experiment = resolve(state.dbdict.experiment)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	448 remote_path = os.path.join(remote_root, self.dbname, self.tablename, str(self.dbstate.id))
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	449 super(DBRSyncChannel, self).__init__(path, remote_path, experiment, state, redirect_stdout, redirect_stderr)
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	450 except:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	451 self.dbstate['dbdict.status'] = self.DONE
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	452 raise
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	453
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	454 def save(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	455 super(DBRSyncChannel, self).save()
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	456 self.dbstate.update(flatten(self.state))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	457
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	458 def setup(self):
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	459 # Extract a single experiment from the table that is not already running.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	460 # set self.experiment and self.state
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	461 super(DBRSyncChannel, self).setup()
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	462 self.state.dbdict.sql.host_name = socket.gethostname()
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	463 self.state.dbdict.sql.host_workdir = self.path
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	464 self.dbstate.update(flatten(self.state))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	465
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	466 def run(self):
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	467 # We pass the force flag as True because the status flag is
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	468 # already set to RUNNING by book_dct in __init__
572 9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	469 v = super(DBRSyncChannel, self).run(force = True)
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	470 if v is self.INCOMPLETE and self.state.dbdict.sql.priority != self.RESTART_PRIORITY:
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	471 self.state.dbdict.sql.priority = self.RESTART_PRIORITY
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	472 self.save()
9f5891cd4048 added host name and workdir to the db Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 571 diff changeset	473 return v
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	474
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	475
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	476
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	477 ################################################################################
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	478 ### running
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	479 ################################################################################
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	480
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	481 import optparse
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	482 OptionParser = optparse.OptionParser
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	483 # class OptionParser(optparse.OptionParser):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	484 # def error(self, message):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	485 # pass
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	486
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	487 def parse_and_run(command, arguments):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	488 parser, runner = runner_registry.get(command, (None, None))
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	489 if not runner:
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	490 raise UsageError('Unknown runner: "%s"' % command)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	491 if parser:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	492 options, arguments = parser.parse_args(arguments)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	493 else:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	494 options = optparse.Values()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	495 run(runner, [options] + arguments)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	496
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	497 def run(runner, arguments):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	498 argspec = inspect.getargspec(runner)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	499 minargs = len(argspec[0])-(len(argspec[3]) if argspec[3] else 0)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	500 maxargs = len(argspec[0])
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	501 if minargs > len(arguments) or maxargs < len(arguments) and not argspec[1]:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	502 s = format_help(runner)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	503 raise UsageError(s)
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	504 runner(*arguments)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	505
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	506 runner_registry = dict()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	507
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	508
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	509 parser_cmdline = OptionParser(usage = '%prog cmdline [options] <experiment> <parameters>')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	510 parser_cmdline.add_option('-f', '--force', action = 'store_true', dest = 'force', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	511 help = 'force running the experiment even if it is already running or completed')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	512 parser_cmdline.add_option('--redirect-stdout', action = 'store_true', dest = 'redirect_stdout', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	513 help = 'redirect stdout to the workdir/stdout file')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	514 parser_cmdline.add_option('--redirect-stderr', action = 'store_true', dest = 'redirect_stderr', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	515 help = 'redirect stderr to the workdir/stdout file')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	516 parser_cmdline.add_option('-r', '--redirect', action = 'store_true', dest = 'redirect', default = False,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	517 help = 'redirect stdout and stderr to the workdir/stdout and workdir/stderr files')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	518 parser_cmdline.add_option('-w', '--workdir', action = 'store', dest = 'workdir', default = None,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	519 help = 'the working directory in which to run the experiment')
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	520 parser_cmdline.add_option('-n', '--dry-run', action = 'store_true', dest = 'dry_run', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	521 help = 'use this option to run the whole experiment in a temporary working directory (cleaned after use)')
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	522
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	523 def runner_cmdline(options, experiment, *strings):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	524 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	525 Start an experiment with parameters given on the command line.
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	526
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	527 Usage: cmdline [options] <experiment> <parameters>
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	528
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	529 Run an experiment with parameters provided on the command
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	530 line. See the help topics for experiment and parameters for
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	531 syntax information.
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	532
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	533 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	534 dbdict-run cmdline mymodule.my_experiment \\
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	535 stopper::pylearn.stopper.nsteps \\ # use pylearn.stopper.nsteps
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	536 stopper.n=10000 \\ # the argument "n" of nsteps is 10000
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	537 lr=0.03
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	538 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	539 state = expand(parse(*strings))
589 6a7f3d83c72b dbdict stuff James Bergstra <bergstrj@iro.umontreal.ca> parents: 586 diff changeset	540 state.setdefault('dbdict', DD()).experiment = experiment
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	541 experiment = resolve(experiment)
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	542 if options.workdir and options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	543 raise UsageError('Please use only one of: --workdir, --dry-run.')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	544 if options.workdir:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	545 workdir = options.workdir
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	546 elif options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	547 workdir = tempfile.mkdtemp()
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	548 else:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	549 workdir = format_d(state, sep=',', space = False)
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	550 channel = StandardChannel(workdir,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	551 experiment, state,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	552 redirect_stdout = options.redirect or options.redirect_stdout,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	553 redirect_stderr = options.redirect or options.redirect_stderr)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	554 channel.run(force = options.force)
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	555 if options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	556 shutil.rmtree(workdir, ignore_errors=True)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	557
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	558 runner_registry['cmdline'] = (parser_cmdline, runner_cmdline)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	559
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	560
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	561
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	562
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	563 parser_filemerge = OptionParser(usage = '%prog filemerge [options] <experiment> <file> <file2> ...')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	564 parser_filemerge.add_option('-f', '--force', action = 'store_true', dest = 'force', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	565 help = 'force running the experiment even if it is already running or completed')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	566 parser_filemerge.add_option('--redirect-stdout', action = 'store_true', dest = 'redirect_stdout', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	567 help = 'redirect stdout to the workdir/stdout file')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	568 parser_filemerge.add_option('--redirect-stderr', action = 'store_true', dest = 'redirect_stderr', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	569 help = 'redirect stderr to the workdir/stdout file')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	570 parser_filemerge.add_option('-r', '--redirect', action = 'store_true', dest = 'redirect', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	571 help = 'redirect stdout and stderr to the workdir/stdout and workdir/stderr files')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	572 parser_filemerge.add_option('-w', '--workdir', action = 'store', dest = 'workdir', default = None,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	573 help = 'the working directory in which to run the experiment')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	574 parser_filemerge.add_option('-n', '--dry-run', action = 'store_true', dest = 'dry_run', default = False,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	575 help = 'use this option to run the whole experiment in a temporary working directory (cleaned after use)')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	576
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	577 def runner_filemerge(options, experiment, mainfile, *other_files):
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	578 """
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	579 Start an experiment with parameters given in files.
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	580
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	581 Usage: filemerge [options] <experiment> <file> <file2> ...
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	582
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	583 Run an experiment with parameters provided in plain text files.
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	584 A single experiment will be run with the union of all the
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	585 parameters listed in the files.
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	586
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	587 Example:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	588 <in file blah1.txt>
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	589 text.first = "hello"
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	590 text.second = "world"
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	591
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	592 <in file blah2.txt>
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	593 number = 12
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	594 numbers.a = 55
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	595 numbers.b = 56
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	596
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	597 Given these files, the following command using filemerge:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	598 $ dbdict-run filemerge mymodule.my_experiment blah1.txt blah2.txt
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	599
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	600 is equivalent to this one using cmdline:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	601 $ dbdict-run cmdline mymodule.my_experiment \\
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	602 text.first=hello text.second=world \\
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	603 number=12 numbers.a=55 numbers.b=56
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	604 """
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	605 with open(mainfile) as f:
594 86a63770af24 fixed small bug with filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 593 diff changeset	606 _state = parse(*map(str.strip, f.readlines()))
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	607 for file in other_files:
595 7ddb286162f6 corrected filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 594 diff changeset	608 if '=' in file:
7ddb286162f6 corrected filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 594 diff changeset	609 _state.update(parse(file))
7ddb286162f6 corrected filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 594 diff changeset	610 else:
7ddb286162f6 corrected filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 594 diff changeset	611 with open(file) as f:
7ddb286162f6 corrected filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 594 diff changeset	612 _state.update(parse(*map(str.strip, f.readlines())))
594 86a63770af24 fixed small bug with filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 593 diff changeset	613 state = expand(_state)
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	614 state.setdefault('dbdict', DD()).experiment = experiment
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	615 experiment = resolve(experiment)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	616 if options.workdir and options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	617 raise UsageError('Please use only one of: --workdir, --dry-run.')
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	618 if options.workdir:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	619 workdir = options.workdir
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	620 elif options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	621 workdir = tempfile.mkdtemp()
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	622 else:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	623 workdir = format_d(state, sep=',', space = False)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	624 channel = StandardChannel(workdir,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	625 experiment, state,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	626 redirect_stdout = options.redirect or options.redirect_stdout,
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	627 redirect_stderr = options.redirect or options.redirect_stderr)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	628 channel.run(force = options.force)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	629 if options.dry_run:
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	630 shutil.rmtree(workdir, ignore_errors=True)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	631
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	632 runner_registry['filemerge'] = (parser_filemerge, runner_filemerge)
5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	633
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	634
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	635
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	636
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	637 parser_sqlschedule = OptionParser(usage = '%prog sqlschedule [options] <tablepath> <experiment> <parameters>')
609 7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	638 parser_sqlschedule.add_option('-f', '--force', action = 'store_true', dest = 'force', default = False,
7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	639 help = 'force adding the experiment to the database even if it is already there')
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	640
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	641 def runner_sqlschedule(options, dbdescr, experiment, *strings):
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	642 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	643 Schedule a job to run using the sql command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	644
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	645 Usage: sqlschedule <tablepath> <experiment> <parameters>
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	646
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	647 See the experiment and parameters topics for more information about
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	648 these parameters.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	649
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	650 Assuming that a postgres database is running on `host`, contains a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	651 database called `dbname` and that `user` has the permissions to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	652 create, read and modify tables on that database, tablepath should
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	653 be of the following form:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	654 postgres://user:pass@host/dbname/tablename
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	655
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	656 If no table is named `tablename`, one will be created
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	657 automatically. The state corresponding to the experiment and
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	658 parameters specified in the command will be saved in the database,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	659 but no experiment will be run.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	660
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	661 To run an experiment scheduled using sqlschedule, see the sql
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	662 command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	663
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	664 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	665 dbdict-run sqlschedule postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	666 mymodule.my_experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	667 stopper::pylearn.stopper.nsteps \\ # use pylearn.stopper.nsteps
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	668 stopper.n=10000 \\ # the argument "n" of nsteps is 10000
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	669 lr=0.03
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	670 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	671
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	672 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	673 username, password, hostname, dbname, tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	674 = sql.parse_dbstring(dbdescr)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	675 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	676 raise UsageError('Wrong syntax for dbdescr')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	677
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	678 db = sql.postgres_serial(
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	679 user = username,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	680 password = password,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	681 host = hostname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	682 database = dbname,
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	683 table_prefix = tablename)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	684
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	685 state = parse(*strings)
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	686 try:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	687 resolve(experiment)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	688 except:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	689 raise UsageError('The first parameter to sqlschedule must be a valid, importable symbol.')
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	690 state['dbdict.experiment'] = experiment
609 7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	691 sql.add_experiments_to_db([state], db, verbose = 1, add_dups = options.force)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	692
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	693 runner_registry['sqlschedule'] = (parser_sqlschedule, runner_sqlschedule)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	694
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	695
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	696
603 84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	697 parser_sqlschedule_filemerge = OptionParser(usage = '%prog sqlschedule_filemerge [options] <tablepath> <experiment> <parameters\|files>')
609 7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	698 parser_sqlschedule_filemerge.add_option('-f', '--force', action = 'store_true', dest = 'force', default = False,
7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	699 help = 'force adding the experiment to the database even if it is already there')
603 84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	700
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	701 def runner_sqlschedule_filemerge(options, dbdescr, experiment, mainfile, *other_files):
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	702 """
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	703 Schedule a job to run using the sql command using parameter files.
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	704
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	705 This command is to sqlschedule what the filemerge command is to
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	706 cmdline.
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	707 """
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	708
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	709 try:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	710 username, password, hostname, dbname, tablename \
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	711 = sql.parse_dbstring(dbdescr)
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	712 except:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	713 raise UsageError('Wrong syntax for dbdescr')
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	714
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	715 db = sql.postgres_serial(
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	716 user = username,
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	717 password = password,
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	718 host = hostname,
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	719 database = dbname,
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	720 table_prefix = tablename)
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	721
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	722 with open(mainfile) as f:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	723 _state = parse(*map(str.strip, f.readlines()))
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	724 for file in other_files:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	725 if '=' in file:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	726 _state.update(parse(file))
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	727 else:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	728 with open(file) as f:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	729 _state.update(parse(*map(str.strip, f.readlines())))
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	730 state = _state
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	731
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	732 try:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	733 resolve(experiment)
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	734 except:
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	735 raise UsageError('The first parameter to sqlschedule_filemerge must be a valid, importable symbol.')
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	736 state['dbdict.experiment'] = experiment
609 7cee8c7f0449 added -f flag to sqlschedule and sqlschedule_filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 604 diff changeset	737 sql.add_experiments_to_db([state], db, verbose = 1, add_dups = options.force)
603 84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	738
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	739 runner_registry['sqlschedule_filemerge'] = (parser_sqlschedule_filemerge, runner_sqlschedule_filemerge)
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	740
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	741
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	742
84e3af9d0110 added sqlschedule_filemerge to newstuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 595 diff changeset	743
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	744 parser_sql = OptionParser(usage = '%prog sql [options] <tablepath> <exproot>')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	745 parser_sql.add_option('-n', dest = 'n', type = 'int', default = 1,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	746 help = 'Run N experiments sequentially (default 1) '
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	747 '(if N is <= 0, runs as many experiments as possible).')
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	748
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	749 def runner_sql(options, dbdescr, exproot):
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	750 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	751 Run jobs from a sql table.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	752
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	753 Usage: sql <tablepath> <exproot>
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	754
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	755 The jobs should be scheduled first with the sqlschedule command.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	756
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	757 Assuming that a postgres database is running on `host`, contains a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	758 database called `dbname` and that `user` has the permissions to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	759 create, read and modify tables on that database, tablepath should
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	760 be of the following form:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	761 postgres://user:pass@host/dbname/tablename
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	762
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	763 exproot can be a local path or a remote path. Examples of exproots:
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	764 /some/local/path
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	765 ssh://some_host:/some/remote/path # relative to the filesystem root
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	766 ssh://some_host:other/remote/path # relative to the HOME on some_host
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	767
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	768 The exproot will contain a subdirectory hierarchy corresponding to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	769 the dbname, tablename and job id which is a unique integer.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	770
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	771 The sql runner will pick any job in the table which is not running
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	772 and is not done and will terminate when that job ends. You may call
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	773 the same command multiple times, sequentially or in parallel, to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	774 run as many unfinished jobs as have been scheduled in that table
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	775 with sqlschedule.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	776
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	777 Example use:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	778 dbdict-run sql \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	779 postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	780 ssh://central_host:myexperiments
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	781 """
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	782 try:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	783 username, password, hostname, dbname, tablename \
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	784 = sql.parse_dbstring(dbdescr)
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	785 except:
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	786 raise UsageError('Wrong syntax for dbdescr')
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	787
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	788 n = options.n if options.n else -1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	789 nrun = 0
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	790 try:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	791 while n != 0:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	792 workdir = tempfile.mkdtemp()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	793 #print 'wdir', workdir
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	794 channel = DBRSyncChannel(username, password, hostname, dbname, tablename,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	795 workdir,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	796 exproot,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	797 redirect_stdout = True,
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	798 redirect_stderr = True)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	799 channel.run()
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	800 shutil.rmtree(workdir, ignore_errors=True)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	801 n -= 1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	802 nrun += 1
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	803 except JobError, e:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	804 if e.args[0] == JobError.NOJOB:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	805 print 'No more jobs to run (run %i jobs)' % nrun
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	806
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	807 runner_registry['sql'] = (parser_sql, runner_sql)
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	808
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	809
1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	810
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	811
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	812
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	813 def runner_help(options, topic = None):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	814 """
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	815 Get help for a topic.
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	816
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	817 Usage: help <topic>
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	818 """
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	819 def bold(x):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	820 return '\033[1m%s\033[0m' % x
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	821 if topic is None:
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	822 print bold('Topics: (use help <topic> for more info)')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	823 print 'example Example of defining and running an experiment.'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	824 print 'experiment How to define an experiment.'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	825 print 'parameters How to list the parameters for an experiment.'
568 1f036d934ad9 improvements to dbdict interface Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 567 diff changeset	826 print
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	827 print bold('Available commands: (use help <command> for more info)')
593 5398a7a1eca7 added --dry-run (-n) option to cmdline, added filemerge Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 590 diff changeset	828 for name, (parser, command) in sorted(runner_registry.iteritems()):
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	829 print name.ljust(20), format_help(command).split('\n')[0]
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	830 return
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	831 elif topic == 'experiment':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	832 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	833
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	834 dbdict-run serves to run experiments. To define an experiment, you
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	835 only have to define a function respecting the following protocol in
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	836 a python file or module:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	837
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	838 def my_experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	839 # experiment code goes here
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	840
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	841 The return value of my_experiment may be channel.COMPLETE or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	842 channel.INCOMPLETE. If the latter is returned, the experiment may
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	843 be resumed at a later point. Note that the return value `None`
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	844 is interpreted as channel.COMPLETE.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	845
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	846 If a command defined by dbdict-run has an <experiment> parameter,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	847 that parameter must be a string such that it could be used in a
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	848 python import statement to import the my_experiment function. For
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	849 example if you defined my_experiment in my_module.py, you can pass
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	850 'my_module.my_experiment' as the experiment parameter.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	851
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	852 When entering my_experiment, the current working directory will be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	853 set for you to a directory specially created for the experiment.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	854 The location and name of that directory vary depending on which
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	855 dbdict-run command you run. You may create logs, save files, pictures,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	856 results, etc. in it.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	857
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	858 state is an object containing the parameters given to the experiment.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	859 For example, if you run the followinc command:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	860
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	861 dbdict-run cmdline my_module.my_experiment a.x=6
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	862
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	863 `state.a.x` will contain the integer 6, and so will `state['a']['x']`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	864 If the state is changed, it will be saved when the experiment ends
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	865 or when channel.save() is called. The next time the experiment is run
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	866 with the same working directory, the modified state will be provided.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	867
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	868 It is not recommended to store large amounts of data in the state. It
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	869 should be limited to scalar or string parameters. Results such as
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	870 weight matrices should be stored in files in the working directory.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	871
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	872 channel is an object with the following important methods:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	873
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	874 - channel.switch() (or channel()) will give the control back to the
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	875 user, if it is appropriate to do so. If a call to channel.switch()
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	876 returns the string 'stop', it typically means that the signal
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	877 SIGTERM (or SIGINT) was received. Therefore, the experiment may be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	878 killed soon, so it should save and return True or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	879 channel.INCOMPLETE so it can be resumed later. This should be
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	880 checked periodically or data loss may be incurred.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	881
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	882 - channel.save() will save the current state. It is automatically
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	883 called when the function returns, but it is a good idea to do it
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	884 periodically.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	885
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	886 - channel.save_and_switch() is an useful shortcut to do both operations
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	887 described above.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	888 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	889
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	890 elif topic == 'parameters':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	891 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	892 If a command takes <parameters> arguments, the arguments should each
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	893 take one of the following forms:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	894
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	895 key=value
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	896
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	897 Set a parameter with name `key` to `value`. The value will be casted
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	898 to an appropriate type automatically and it will be accessible to
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	899 the experiment using `state.key`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	900
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	901 If `key` is a dotted name, the value will be set in nested
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	902 dictionaries corresponding to each part.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	903
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	904 Examples:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	905 a=1 state.a <- 1
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	906 b=2.3 state.b <- 2.3
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	907 c.d="hello" state.c.d <- "hello"
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	908
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	909 key::builder
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	910
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	911 This is equivalent to key.__builder__=builder.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	912
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	913 The builder should be a symbol that can be used with import or
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	914 __import__ and it should be callable.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	915
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	916 If a key has a builder defined, the experiment code may easily make
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	917 an object out of it using the `make` function. `obj = make(state.key)`.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	918 This will call the builder on the substate corresponding to state.key,
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	919 as will be made clear in the example:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	920
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	921 Example:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	922 regexp::re.compile
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	923 regexp.pattern='a.*c'
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	924
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	925 from pylearn.dbdict.newstuff import make
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	926 def experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	927 regexp = make(state.regexp) # regexp is now re.compile(pattern = 'a.*c')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	928 print regexp.sub('blahblah', 'hello abbbbc there')
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	929
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	930 If the above experiment was called with the state produced by the
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	931 parameters in the example, it would print 'hello blahblah there'.
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	932 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	933
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	934 elif topic == 'example':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	935 helptext = """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	936 Example of an experiment that trains some model for 100000 iterations:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	937
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	938 # defined in: my_experiments.py
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	939 def experiment(state, channel):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	940 try:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	941 model = cPickle.load(open('model', 'r'))
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	942 except:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	943 model = my_model(state.some_param, state.other_param)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	944 state.n = 0
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	945 dataset = my_dataset(skipto = state.n)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	946 for i in xrange(100000 - state.n):
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	947 model.update(dataset.next())
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	948 if i and i % 1000 == 0:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	949 if channel.save_and_switch() == 'stop':
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	950 state.n += i + 1
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	951 rval = channel.INCOMPLETE
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	952 break
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	953 else:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	954 state.result = model.cost(some_test_set)
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	955 rval = channel.COMPLETE
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	956 cPickle.dump(model, open('model', 'w'))
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	957 return rval
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	958
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	959 And then you could run it this way:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	960
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	961 dbdict-run cmdline my_experiments.experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	962 some_param=1 \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	963 other_param=0.4
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	964
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	965 Or this way:
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	966
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	967 dbdict-run sqlschedule postgres://user:pass@host/dbname/tablename \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	968 my_experiments.experiment \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	969 some_param=1 \\
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	970 other_param=0.4
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	971
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	972 dbdict-run sql postgres://user:pass@host/dbname/tablename exproot
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	973
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	974 You need to make sure that the module `my_experiments` is accessible
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	975 from python. You can check with the command
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	976
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	977 $ python -m my_experiments
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	978 """
ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	979 else:
586 5020b12e87ee fixed broken help Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 584 diff changeset	980 helptext = runner_registry.get(topic, None)[1]
576 ef424abb7458 bugfixes and a lot of documentation Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 572 diff changeset	981 print format_help(helptext)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	982
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	983 runner_registry['help'] = (None, runner_help)
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	984
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	985 ################################################################################
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	986 ### main
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	987 ################################################################################
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	988
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	989 def run_cmdline():
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	990 try:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	991 if len(sys.argv) <= 1:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	992 raise UsageError('Usage: %s <run_type> [<arguments>*]' % sys.argv[0])
584 9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	993 cmd = None
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	994 args = []
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	995 for arg in sys.argv[1:]:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	996 if cmd is not None or arg.startswith('-'):
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	997 args.append(arg)
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	998 else:
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	999 cmd = arg
9f5691d5587d added command line options Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 579 diff changeset	1000 parse_and_run(cmd, args)
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	1001 except UsageError, e:
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	1002 print 'Usage error:'
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	1003 print e
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	1004
566 a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	1005 if __name__ == '__main__':
a71971ccc1e4 checkpoint Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: 565 diff changeset	1006 run_cmdline()
565 0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	1007
0ac4927e9d97 new stuff Olivier Breuleux <breuleuo@iro.umontreal.ca> parents: diff changeset	1008

Mercurial > pylearn

annotate pylearn/dbdict/newstuff.py @ 609:7cee8c7f0449