Mercurial > pylearn
annotate bin/pkldu.py @ 1435:3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
author | Razvan Pascanu <r.pascanu@gmail.com> |
---|---|
date | Tue, 22 Feb 2011 11:23:32 -0500 |
parents | f88508a42a38 |
children | 35b56d794d09 |
rev | line source |
---|---|
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
1 #!/bin/env python |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
2 """ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
3 Script to analyze disk usage of pickled files. See usage. |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
4 """ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
5 __authors__ = "Ian Goodfellow, Razvan Pascanu" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
6 __copyright__ = "(c) 2010, Universite de Montreal" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
7 __contact__ = "Razvan Pascanu <r.pascanu@gmail>" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
8 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
9 import cPickle, optparse, time, sys |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
10 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
11 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
12 usage = """ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
13 pkldu [OPTIONS] file indices |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
14 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
15 First argument of the program is the file to analyze. Following arguments |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
16 help you indexing in the object. For example : |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
17 pkldu.py foo.pkl .my_field [my_key] 3 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
18 |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
19 will load an object obj from foo.pkl and analyze obj.my_field["my_key"][3] |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
20 """ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
21 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
22 space_units = [(' B', 1), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
23 ('kB', 2**10), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
24 ('MB', 2**20), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
25 ('GB', 2**30), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
26 ('TB', 2**40)] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
27 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
28 time_units = [('s', 1), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
29 ('m', 60), |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
30 ('h', 3600) ] |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
31 |
1433
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
32 def load(filepath): |
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
33 f = open(filepath,'rb') |
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
34 obj = cPickle.load(f) |
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
35 f.close() |
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
36 return obj |
14ba52c38f07
removed import to file that don't exist in this repo.
Frederic Bastien <nouiz@nouiz.org>
parents:
1423
diff
changeset
|
37 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
38 def format_string(s, maxlen): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
39 if len(s) > maxlen: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
40 s = s[:maxlen] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
41 return s + ' '*(maxlen - len(s)) |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
42 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
43 def prettyprint(size, units, human_readable = False): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
44 unit_name = units[0][0] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
45 rval = size |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
46 if human_readable: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
47 for unit, val in units: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
48 if float(size)/val > 1: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
49 unit_name = unit |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
50 rval = float(size)/val |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
51 return (rval, unit_name) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
52 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
53 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
54 def analyze(options, filepath, indices): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
55 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
56 orig_obj = load(filepath) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
57 cycle_check = {} |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
58 obj_name = 'root_obj' |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
59 cycle_check[id(orig_obj)] = obj_name |
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
60 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
61 for field in indices: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
62 if field.startswith('['): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
63 assert field.endswith(']') |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
64 obj_name += '[' + field[1:-1] + ']' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
65 orig_obj = orig_obj[field[1:-1]] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
66 elif field.startswith('.'): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
67 obj_name += '.' + field |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
68 orig_obj = getattr(orig_obj,field[1:]) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
69 else: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
70 obj_name + '[' + field + ']' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
71 orig_obj = orig_obj[eval(field)] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
72 if id(orig_obj) in cycle_check: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
73 print ( "You're going in circles, "+obj_name+" is the same as " |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
74 +cycle_check[id(orig_obj)]) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
75 quit() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
76 cycle_check[id(orig_obj)] = obj_name |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
77 |
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
78 s = cPickle.dumps(orig_obj) |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
79 prev_bytes = len(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
80 print 'original object : \t\t\t\t%6.2f %s'%prettyprint(prev_bytes, |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
81 space_units, |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
82 options.human) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
83 |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
84 t1 = time.time() |
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
85 x = cPickle.loads(s) |
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
86 t2 = time.time() |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
87 prev_t = t2 - t1 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
88 print 'load time: %6.2f %s'%prettyprint(prev_t, time_units, |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
89 options.human) |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
90 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
91 if isinstance(orig_obj, dict): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
92 print 'Object is a dictionary' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
93 keys = [ key for key in orig_obj.keys() ] |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
94 for key in keys: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
95 key_name = format_string(key, 40) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
96 s = cPickle.dumps(orig_obj[key]) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
97 new_bytes = len(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
98 t1 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
99 x = cPickle.loads(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
100 t2 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
101 new_t = t2 - t1 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
102 print 'field: %40s %6.2f %s ( loads in %6.2f %s)'%( |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
103 (key_name,) + |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
104 prettyprint(new_bytes, space_units, options.human) + |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
105 prettyprint(new_t, time_units, options.human) ) |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
106 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
107 elif isinstance(orig_obj, (tuple, list)): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
108 print 'Object is a list/tuple of ', len(orig_obj), 'elements' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
109 for idx, v in enumerate(orig_obj): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
110 s = cPickle.dumps(v) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
111 new_bytes = len(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
112 t1 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
113 x = cPickle.loads(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
114 t2 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
115 new_t = t2 - t1 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
116 print 'entry: %03d \t\t\t\t %6.2f %s ( loads in %6.2f %s)' %( |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
117 (idx,)+ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
118 prettyprint(new_bytes, space_units, options.human) + |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
119 prettyprint(new_t, time_units, options.human) ) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
120 else: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
121 print 'Object is a '+str(type(orig_obj)) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
122 for field in dir(orig_obj): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
123 field_name = format_string( field, 40) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
124 if field.startswith('__') and not options.reserved: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
125 # We skip reserved fields |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
126 break |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
127 try: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
128 s = cPickle.dumps(getattr(orig_obj, field)) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
129 new_bytes = len(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
130 t1 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
131 x = cPickle.loads(s) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
132 t2 = time.time() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
133 new_t = t2 - t1 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
134 print 'field: %40s %6.2f %s ( loads in %6.2f %s)' %( |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
135 (field_name,)+ |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
136 prettyprint(new_bytes, space_units, options.human) + |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
137 prettyprint(new_t, time_units, options.human) ) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
138 except: |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
139 print 'Could not pickle field', field_name |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
140 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
141 def process_options(): |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
142 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
143 parser = optparse.OptionParser(usage) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
144 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
145 parser.add_option( '-H' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
146 , "--human-readable" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
147 , dest = 'human' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
148 , action="store_true" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
149 , default=False |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
150 , help = (' If information should be presented in ' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
151 'human readable format') |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
152 ) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
153 |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
154 parser.add_option( '-r' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
155 , "--reserved-fields" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
156 , dest = 'reserved' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
157 , action="store_true" |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
158 , default=False |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
159 , help = (' If information about python reserved ' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
160 ' fields (i.e. starting with `__`) ' |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
161 ' should be displayed' ) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
162 ) |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
163 return parser.parse_args() |
1423
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
164 |
ea5d27727869
added pickle disk usage inspection utility 'pkldu'
Ian Goodfellow
parents:
diff
changeset
|
165 |
1435
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
166 if __name__ == '__main__': |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
167 (options,args) = process_options() |
3dd64c115657
revised version of pkldu that is a bit more structured code wise and outputs in human readable units
Razvan Pascanu <r.pascanu@gmail.com>
parents:
1434
diff
changeset
|
168 analyze(options, args[0], args[1:]) |