Mercurial > pylearn

annotate bin/pkldu.py @ 1530:08b3e827575a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Change class path as Theano want to hide the Deprecated Module interface.
author Frederic Bastien <nouiz@nouiz.org>
date Fri, 12 Jul 2013 13:46:45 -0400
parents 509d6669429d
children
rev   line source
1459
509d6669429d pkldu - changed /bin/env to /usr/bin/env which is more standard I hope.
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1437
diff changeset
1 #!/usr/bin/env python
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
2 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
3 Script to analyze disk usage of pickled files. See usage.
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
4 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
5 __authors__ = "Ian Goodfellow, Razvan Pascanu"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
6 __copyright__ = "(c) 2010, Universite de Montreal"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
7 __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
8
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
9 import cPickle, optparse, time, sys
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
10
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
11
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
12 usage = """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
13 pkldu [OPTIONS] file indices
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
14
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
15 First argument of the program is the file to analyze. Following arguments
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
16 help you indexing in the object. For example :
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
17 pkldu.py foo.pkl .my_field [my_key] 3
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
18
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
19 will load an object obj from foo.pkl and analyze obj.my_field["my_key"][3]
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
20 """
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
21
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
22 space_units = [(' B', 1),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
23 ('kB', 2**10),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
24 ('MB', 2**20),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
25 ('GB', 2**30),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
26 ('TB', 2**40)]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
27
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
28 time_units = [('s', 1),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
29 ('m', 60),
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
30 ('h', 3600) ]
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
31
1433
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
32 def load(filepath):
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
33 f = open(filepath,'rb')
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
34 obj = cPickle.load(f)
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
35 f.close()
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
36 return obj
14ba52c38f07 removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents: 1423
diff changeset
37
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
38 def format_string(s, maxlen):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
39 if len(s) > maxlen:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
40 s = s[:maxlen]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
41 return s + ' '*(maxlen - len(s))
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
42
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
43 def prettyprint(size, units, human_readable = False):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
44 unit_name = units[0][0]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
45 rval = size
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
46 if human_readable:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
47 for unit, val in units:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
48 if float(size)/val > 1:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
49 unit_name = unit
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
50 rval = float(size)/val
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
51 return (rval, unit_name)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
52
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
53
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
54 def analyze(options, filepath, indices):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
55
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
56 orig_obj = load(filepath)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
57 cycle_check = {}
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
58 obj_name = 'root_obj'
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
59 cycle_check[id(orig_obj)] = obj_name
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
60
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
61 for field in indices:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
62 if field.startswith('['):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
63 assert field.endswith(']')
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
64 obj_name += '[' + field[1:-1] + ']'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
65 orig_obj = orig_obj[field[1:-1]]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
66 elif field.startswith('.'):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
67 obj_name += '.' + field
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
68 orig_obj = getattr(orig_obj,field[1:])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
69 else:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
70 obj_name + '[' + field + ']'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
71 orig_obj = orig_obj[eval(field)]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
72 if id(orig_obj) in cycle_check:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
73 print ( "You're going in circles, "+obj_name+" is the same as "
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
74 +cycle_check[id(orig_obj)])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
75 quit()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
76 cycle_check[id(orig_obj)] = obj_name
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
77
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
78 s = cPickle.dumps(orig_obj)
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
79 prev_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
80 print 'original object : \t\t\t\t%6.2f %s'%prettyprint(prev_bytes,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
81 space_units,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
82 options.human)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
83
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
84 t1 = time.time()
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
85 x = cPickle.loads(s)
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
86 t2 = time.time()
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
87 prev_t = t2 - t1
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
88 print 'load time: %6.2f %s'%prettyprint(prev_t, time_units,
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
89 options.human)
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
90
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
91 print_entries = []
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
92 if isinstance(orig_obj, dict):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
93 print 'Object is a dictionary'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
94 keys = [ key for key in orig_obj.keys() ]
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
95 for key in keys:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
96 key_name = format_string(key, 40)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
97 s = cPickle.dumps(orig_obj[key])
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
98 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
99 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
100 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
101 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
102 new_t = t2 - t1
1437
4b27456d3bce renamed output as key for dictionary ( from field). Now for dictionary we have keys, for tuples/list we have entries and for object fields
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1436
diff changeset
103 print_entry = 'key: %40s %6.2f %s ( loads in %6.2f %s)'%(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
104 (key_name,) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
105 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
106 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
107 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
108 print 'Processed', key_name
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
109 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
110 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
111 print print_entry
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
112
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
113 elif isinstance(orig_obj, (tuple, list)):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
114 print 'Object is a list/tuple of ', len(orig_obj), 'elements'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
115 for idx, v in enumerate(orig_obj):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
116 s = cPickle.dumps(v)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
117 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
118 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
119 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
120 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
121 new_t = t2 - t1
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
122 print_entry = 'entry: %03d \t\t\t\t %6.2f %s ( loads in %6.2f %s)' %(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
123 (idx,)+
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
124 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
125 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
126
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
127 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
128 print 'Processed entry number ', idx
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
129 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
130 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
131 print print_entry
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
132 else:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
133 print 'Object is a '+str(type(orig_obj))
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
134 for field in dir(orig_obj):
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
135 field_name = format_string( field, 40)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
136 if field.startswith('__') and not options.reserved:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
137 # We skip reserved fields
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
138 break
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
139 try:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
140 s = cPickle.dumps(getattr(orig_obj, field))
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
141 new_bytes = len(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
142 t1 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
143 x = cPickle.loads(s)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
144 t2 = time.time()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
145 new_t = t2 - t1
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
146 print_entry ='field: %40s %6.2f %s ( loads in %6.2f %s)' %(
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
147 (field_name,)+
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
148 prettyprint(new_bytes, space_units, options.human) +
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
149 prettyprint(new_t, time_units, options.human) )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
150
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
151 if options.order is not 'none':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
152 print 'Processed field ', field_name
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
153 print_entries += [(new_bytes, print_entry)]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
154 else:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
155 print print_entry
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
156 except:
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
157 print 'Could not pickle field', field_name
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
158 if options.order in ('desc','asc'):
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
159 reverse = False
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
160 if options.order == 'desc':
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
161 reverse = True
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
162 print_entries = sorted(print_entries
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
163 , key = lambda x:x[0]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
164 , reverse = reverse)
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
165 for entry in print_entries:
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
166 print entry[1]
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
167
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
168
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
169 def process_options():
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
170
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
171 parser = optparse.OptionParser(usage)
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
172
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
173 parser.add_option( '-H'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
174 , "--human-readable"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
175 , dest = 'human'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
176 , action="store_true"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
177 , default=False
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
178 , help = (' If information should be presented in '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
179 'human readable format')
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
180 )
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
181
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
182 parser.add_option( '-r'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
183 , "--reserved-fields"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
184 , dest = 'reserved'
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
185 , action="store_true"
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
186 , default=False
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
187 , help = (' If information about python reserved '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
188 ' fields (i.e. starting with `__`) '
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
189 ' should be displayed' )
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
190 )
1436
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
191
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
192
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
193 parser.add_option( '-o'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
194 , "--order-fields"
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
195 , dest = 'order'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
196 , default= 'none'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
197 , help = (' Order fields acording the their size.'
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
198 ' Possible values are {none, desc, asc}')
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
199 )
35b56d794d09 added option to order descending /ascending the fields acording to their size
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1435
diff changeset
200
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
201 return parser.parse_args()
1423
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
202
ea5d27727869 added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff changeset
203
1435
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
204 if __name__ == '__main__':
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
205 (options,args) = process_options()
3dd64c115657 revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents: 1434
diff changeset
206 analyze(options, args[0], args[1:])