Mercurial > pylearn

annotate bin/pkldu.py @ 1497:3d4615ee96a4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added recommendation to use "except Exception:" rather than "except:"
author Olivier Delalleau <delallea@iro>
date Thu, 01 Sep 2011 13:35:02 -0400
parents 509d6669429d
children
rev   line source
1459
509d6669429d pkldu - changed /bin/env to /usr/bin/env which is more standard I hope.
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1437
diff changeset
1 #!/usr/bin/env python
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
2 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
3 Script to analyze disk usage of pickled files. See usage.
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
4 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
5 __authors__ = "Ian Goodfellow, Razvan Pascanu"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
6 __copyright__ = "(c) 2010, Universite de Montreal"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
7 __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
8
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
9 import cPickle, optparse, time, sys
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
10
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
11
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
12 usage = """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
13 pkldu [OPTIONS] file indices
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
14
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
15 First argument of the program is the file to analyze. Following arguments
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
16 help you indexing in the object. For example :
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
17 pkldu.py foo.pkl .my_field [my_key] 3
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
18
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
19 will load an object obj from foo.pkl and analyze obj.my_field["my_key"][3]
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
20 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
21
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
22 space_units = [(' B', 1),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
23 ('kB', 2**10),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
24 ('MB', 2**20),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
25 ('GB', 2**30),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
26 ('TB', 2**40)]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
27
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
28 time_units = [('s', 1),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
29 ('m', 60),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
30 ('h', 3600) ]
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
31
1433
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
32 def load(filepath):
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
33 f = open(filepath,'rb')
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
34 obj = cPickle.load(f)
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
35 f.close()
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
36 return obj
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
37
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
38 def format_string(s, maxlen):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
39 if len(s) > maxlen:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
40 s = s[:maxlen]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
41 return s + ' '*(maxlen - len(s))
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
42
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
43 def prettyprint(size, units, human_readable = False):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
44 unit_name = units[0][0]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
45 rval = size
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
46 if human_readable:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
47 for unit, val in units:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
48 if float(size)/val > 1:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
49 unit_name = unit
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
50 rval = float(size)/val
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
51 return (rval, unit_name)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
52
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
53
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
54 def analyze(options, filepath, indices):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
55
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
56 orig_obj = load(filepath)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
57 cycle_check = {}
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
58 obj_name = 'root_obj'
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
59 cycle_check[id(orig_obj)] = obj_name
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
60
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
61 for field in indices:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
62 if field.startswith('['):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
63 assert field.endswith(']')
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
64 obj_name += '[' + field[1:-1] + ']'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
65 orig_obj = orig_obj[field[1:-1]]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
66 elif field.startswith('.'):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
67 obj_name += '.' + field
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
68 orig_obj = getattr(orig_obj,field[1:])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
69 else:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
70 obj_name + '[' + field + ']'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
71 orig_obj = orig_obj[eval(field)]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
72 if id(orig_obj) in cycle_check:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
73 print ( "You're going in circles, "+obj_name+" is the same as "
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
74 +cycle_check[id(orig_obj)])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
75 quit()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
76 cycle_check[id(orig_obj)] = obj_name
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
77
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
78 s = cPickle.dumps(orig_obj)
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
79 prev_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
80 print 'original object : \t\t\t\t%6.2f %s'%prettyprint(prev_bytes,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
81 space_units,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
82 options.human)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
83
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
84 t1 = time.time()
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
85 x = cPickle.loads(s)
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
86 t2 = time.time()
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
87 prev_t = t2 - t1
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
88 print 'load time: %6.2f %s'%prettyprint(prev_t, time_units,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
89 options.human)
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
90
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
91 print_entries = []
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
92 if isinstance(orig_obj, dict):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
93 print 'Object is a dictionary'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
94 keys = [ key for key in orig_obj.keys() ]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
95 for key in keys:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
96 key_name = format_string(key, 40)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
97 s = cPickle.dumps(orig_obj[key])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
98 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
99 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
100 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
101 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
102 new_t = t2 - t1
1437
4b27456d3bce renamed output as key for dictionary ( from field). Now for dictionary we have keys, for tuples/list we have entries and for object fields
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1436
diff changeset
103 print_entry = 'key: %40s %6.2f %s ( loads in %6.2f %s)'%(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
104 (key_name,) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
105 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
106 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
107 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
108 print 'Processed', key_name
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
109 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
110 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
111 print print_entry
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
112
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
113 elif isinstance(orig_obj, (tuple, list)):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
114 print 'Object is a list/tuple of ', len(orig_obj), 'elements'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
115 for idx, v in enumerate(orig_obj):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
116 s = cPickle.dumps(v)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
117 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
118 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
119 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
120 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
121 new_t = t2 - t1
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
122 print_entry = 'entry: %03d \t\t\t\t %6.2f %s ( loads in %6.2f %s)' %(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
123 (idx,)+
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
124 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
125 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
126
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
127 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
128 print 'Processed entry number ', idx
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
129 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
130 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
131 print print_entry
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
132 else:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
133 print 'Object is a '+str(type(orig_obj))
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
134 for field in dir(orig_obj):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
135 field_name = format_string( field, 40)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
136 if field.startswith('__') and not options.reserved:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
137 # We skip reserved fields
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
138 break
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
139 try:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
140 s = cPickle.dumps(getattr(orig_obj, field))
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
141 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
142 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
143 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
144 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
145 new_t = t2 - t1
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
146 print_entry ='field: %40s %6.2f %s ( loads in %6.2f %s)' %(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
147 (field_name,)+
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
148 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
149 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
150
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
151 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
152 print 'Processed field ', field_name
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
153 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
154 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
155 print print_entry
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
156 except:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
157 print 'Could not pickle field', field_name
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
158 if options.order in ('desc','asc'):
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
159 reverse = False
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
160 if options.order == 'desc':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
161 reverse = True
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
162 print_entries = sorted(print_entries
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
163 , key = lambda x:x[0]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
164 , reverse = reverse)
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
165 for entry in print_entries:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
166 print entry[1]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
167
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
168
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
169 def process_options():
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
170
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
171 parser = optparse.OptionParser(usage)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
172
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
173 parser.add_option( '-H'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
174 , "--human-readable"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
175 , dest = 'human'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
176 , action="store_true"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
177 , default=False
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
178 , help = (' If information should be presented in '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
179 'human readable format')
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
180 )
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
181
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
182 parser.add_option( '-r'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
183 , "--reserved-fields"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
184 , dest = 'reserved'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
185 , action="store_true"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
186 , default=False
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
187 , help = (' If information about python reserved '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
188 ' fields (i.e. starting with `__`) '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
189 ' should be displayed' )
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
190 )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
191
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
192
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
193 parser.add_option( '-o'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
194 , "--order-fields"
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
195 , dest = 'order'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
196 , default= 'none'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
197 , help = (' Order fields acording the their size.'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
198 ' Possible values are {none, desc, asc}')
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
199 )
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
200
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
201 return parser.parse_args()
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
202
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
203
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
204 if __name__ == '__main__':
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
205 (options,args) = process_options()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
206 analyze(options, args[0], args[1:])