Mercurial > ift6266

annotate utils/seriestables/test_series.py @ 613:5e481b224117

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix the reading of PNIST dataset following Dumi compression of the data.
author Frederic Bastien <nouiz@nouiz.org>
date Thu, 06 Jan 2011 13:57:05 -0500
parents bfe20d63f88c
children
rev   line source
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
1 import tempfile
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
2
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
3 import numpy
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
4 import numpy.random
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
5
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
6 from jobman import DD
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
7
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
8 import tables
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
9
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
10 from series import *
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
11 import series
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
12
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
13 #################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
14 # Utils
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
15
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
16 def compare_floats(f1,f2):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
17 if f1-f2 < 1e-3:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
18 return True
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
19 return False
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
20
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
21 def compare_lists(it1, it2, floats=False):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
22 if len(it1) != len(it2):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
23 return False
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
24
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
25 for el1, el2 in zip(it1, it2):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
26 if floats:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
27 if not compare_floats(el1,el2):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
28 return False
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
29 elif el1 != el2:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
30 return False
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
31
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
32 return True
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
33
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
34 #################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
35 # Basic Series class tests
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
36
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
37 def test_Series_types():
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
38 pass
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
39
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
40 #################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
41 # ErrorSeries tests
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
42
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
43 def test_ErrorSeries_common_case(h5f=None):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
44 if not h5f:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
45 h5f_path = tempfile.NamedTemporaryFile().name
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
46 h5f = tables.openFile(h5f_path, "w")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
47
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
48 validation_error = series.ErrorSeries(error_name="validation_error", table_name="validation_error",
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
49 hdf5_file=h5f, index_names=('epoch','minibatch'),
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
50 title="Validation error indexed by epoch and minibatch")
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
51
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
52 # (1,1), (1,2) etc. are (epoch, minibatch) index
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
53 validation_error.append((1,1), 32.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
54 validation_error.append((1,2), 30.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
55 validation_error.append((2,1), 28.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
56 validation_error.append((2,2), 26.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
57
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
58 h5f.close()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
59
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
60 h5f = tables.openFile(h5f_path, "r")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
61
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
62 table = h5f.getNode('/', 'validation_error')
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
63
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
64 assert compare_lists(table.cols.epoch[:], [1,1,2,2])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
65 assert compare_lists(table.cols.minibatch[:], [1,2,1,2])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
66 assert compare_lists(table.cols.validation_error[:], [32.0, 30.0, 28.0, 26.0])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
67
226
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
68 def test_ErrorSeries_no_index(h5f=None):
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
69 if not h5f:
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
70 h5f_path = tempfile.NamedTemporaryFile().name
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
71 h5f = tables.openFile(h5f_path, "w")
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
72
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
73 validation_error = series.ErrorSeries(error_name="validation_error",
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
74 table_name="validation_error",
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
75 hdf5_file=h5f,
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
76 # empty tuple
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
77 index_names=tuple(),
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
78 title="Validation error with no index")
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
79
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
80 # (1,1), (1,2) etc. are (epoch, minibatch) index
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
81 validation_error.append(tuple(), 32.0)
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
82 validation_error.append(tuple(), 30.0)
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
83 validation_error.append(tuple(), 28.0)
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
84 validation_error.append(tuple(), 26.0)
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
85
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
86 h5f.close()
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
87
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
88 h5f = tables.openFile(h5f_path, "r")
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
89
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
90 table = h5f.getNode('/', 'validation_error')
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
91
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
92 assert compare_lists(table.cols.validation_error[:], [32.0, 30.0, 28.0, 26.0])
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
93 assert not ("epoch" in dir(table.cols))
bfe20d63f88c Test pour cas sans index, après question de James
fsavard
parents: 224
diff changeset
94
224
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
95 def test_ErrorSeries_notimestamp(h5f=None):
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
96 if not h5f:
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
97 h5f_path = tempfile.NamedTemporaryFile().name
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
98 h5f = tables.openFile(h5f_path, "w")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
99
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
100 validation_error = series.ErrorSeries(error_name="validation_error", table_name="validation_error",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
101 hdf5_file=h5f, index_names=('epoch','minibatch'),
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
102 title="Validation error indexed by epoch and minibatch",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
103 store_timestamp=False)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
104
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
105 # (1,1), (1,2) etc. are (epoch, minibatch) index
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
106 validation_error.append((1,1), 32.0)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
107
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
108 h5f.close()
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
109
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
110 h5f = tables.openFile(h5f_path, "r")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
111
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
112 table = h5f.getNode('/', 'validation_error')
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
113
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
114 assert compare_lists(table.cols.epoch[:], [1])
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
115 assert not ("timestamp" in dir(table.cols))
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
116 assert "cpuclock" in dir(table.cols)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
117
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
118 def test_ErrorSeries_nocpuclock(h5f=None):
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
119 if not h5f:
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
120 h5f_path = tempfile.NamedTemporaryFile().name
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
121 h5f = tables.openFile(h5f_path, "w")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
122
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
123 validation_error = series.ErrorSeries(error_name="validation_error", table_name="validation_error",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
124 hdf5_file=h5f, index_names=('epoch','minibatch'),
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
125 title="Validation error indexed by epoch and minibatch",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
126 store_cpuclock=False)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
127
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
128 # (1,1), (1,2) etc. are (epoch, minibatch) index
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
129 validation_error.append((1,1), 32.0)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
130
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
131 h5f.close()
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
132
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
133 h5f = tables.openFile(h5f_path, "r")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
134
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
135 table = h5f.getNode('/', 'validation_error')
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
136
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
137 assert compare_lists(table.cols.epoch[:], [1])
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
138 assert not ("cpuclock" in dir(table.cols))
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
139 assert "timestamp" in dir(table.cols)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
140
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
141 def test_AccumulatorSeriesWrapper_common_case(h5f=None):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
142 if not h5f:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
143 h5f_path = tempfile.NamedTemporaryFile().name
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
144 h5f = tables.openFile(h5f_path, "w")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
145
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
146 validation_error = ErrorSeries(error_name="accumulated_validation_error",
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
147 table_name="accumulated_validation_error",
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
148 hdf5_file=h5f,
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
149 index_names=('epoch','minibatch'),
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
150 title="Validation error, summed every 3 minibatches, indexed by epoch and minibatch")
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
151
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
152 accumulator = AccumulatorSeriesWrapper(base_series=validation_error,
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
153 reduce_every=3, reduce_function=numpy.sum)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
154
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
155 # (1,1), (1,2) etc. are (epoch, minibatch) index
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
156 accumulator.append((1,1), 32.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
157 accumulator.append((1,2), 30.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
158 accumulator.append((2,1), 28.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
159 accumulator.append((2,2), 26.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
160 accumulator.append((3,1), 24.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
161 accumulator.append((3,2), 22.0)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
162
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
163 h5f.close()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
164
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
165 h5f = tables.openFile(h5f_path, "r")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
166
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
167 table = h5f.getNode('/', 'accumulated_validation_error')
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
168
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
169 assert compare_lists(table.cols.epoch[:], [2,3])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
170 assert compare_lists(table.cols.minibatch[:], [1,2])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
171 assert compare_lists(table.cols.accumulated_validation_error[:], [90.0,72.0], floats=True)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
172
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
173 def test_BasicStatisticsSeries_common_case(h5f=None):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
174 if not h5f:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
175 h5f_path = tempfile.NamedTemporaryFile().name
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
176 h5f = tables.openFile(h5f_path, "w")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
177
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
178 stats_series = BasicStatisticsSeries(table_name="b_vector_statistics",
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
179 hdf5_file=h5f, index_names=('epoch','minibatch'),
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
180 title="Basic statistics for b vector indexed by epoch and minibatch")
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
181
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
182 # (1,1), (1,2) etc. are (epoch, minibatch) index
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
183 stats_series.append((1,1), [0.15, 0.20, 0.30])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
184 stats_series.append((1,2), [-0.18, 0.30, 0.58])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
185 stats_series.append((2,1), [0.18, -0.38, -0.68])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
186 stats_series.append((2,2), [0.15, 0.02, 1.9])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
187
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
188 h5f.close()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
189
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
190 h5f = tables.openFile(h5f_path, "r")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
191
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
192 table = h5f.getNode('/', 'b_vector_statistics')
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
193
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
194 assert compare_lists(table.cols.epoch[:], [1,1,2,2])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
195 assert compare_lists(table.cols.minibatch[:], [1,2,1,2])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
196 assert compare_lists(table.cols.mean[:], [0.21666667, 0.23333333, -0.29333332, 0.69], floats=True)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
197 assert compare_lists(table.cols.min[:], [0.15000001, -0.18000001, -0.68000001, 0.02], floats=True)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
198 assert compare_lists(table.cols.max[:], [0.30, 0.58, 0.18, 1.9], floats=True)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
199 assert compare_lists(table.cols.std[:], [0.06236095, 0.31382939, 0.35640177, 0.85724366], floats=True)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
200
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
201 def test_SharedParamsStatisticsWrapper_commoncase(h5f=None):
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
202 import numpy.random
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
203
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
204 if not h5f:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
205 h5f_path = tempfile.NamedTemporaryFile().name
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
206 h5f = tables.openFile(h5f_path, "w")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
207
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
208 stats = SharedParamsStatisticsWrapper(new_group_name="params", base_group="/",
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
209 arrays_names=('b1','b2','b3'), hdf5_file=h5f,
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
210 index_names=('epoch','minibatch'))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
211
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
212 b1 = DD({'value':numpy.random.rand(5)})
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
213 b2 = DD({'value':numpy.random.rand(5)})
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
214 b3 = DD({'value':numpy.random.rand(5)})
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
215 stats.append((1,1), [b1,b2,b3])
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
216
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
217 h5f.close()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
218
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
219 h5f = tables.openFile(h5f_path, "r")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
220
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
221 b1_table = h5f.getNode('/params', 'b1')
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
222 b3_table = h5f.getNode('/params', 'b3')
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
223
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
224 assert b1_table.cols.mean[0] - numpy.mean(b1.value) < 1e-3
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
225 assert b3_table.cols.mean[0] - numpy.mean(b3.value) < 1e-3
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
226 assert b1_table.cols.min[0] - numpy.min(b1.value) < 1e-3
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
227 assert b3_table.cols.min[0] - numpy.min(b3.value) < 1e-3
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
228
224
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
229 def test_SharedParamsStatisticsWrapper_notimestamp(h5f=None):
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
230 import numpy.random
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
231
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
232 if not h5f:
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
233 h5f_path = tempfile.NamedTemporaryFile().name
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
234 h5f = tables.openFile(h5f_path, "w")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
235
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
236 stats = SharedParamsStatisticsWrapper(new_group_name="params", base_group="/",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
237 arrays_names=('b1','b2','b3'), hdf5_file=h5f,
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
238 index_names=('epoch','minibatch'),
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
239 store_timestamp=False)
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
240
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
241 b1 = DD({'value':numpy.random.rand(5)})
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
242 b2 = DD({'value':numpy.random.rand(5)})
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
243 b3 = DD({'value':numpy.random.rand(5)})
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
244 stats.append((1,1), [b1,b2,b3])
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
245
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
246 h5f.close()
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
247
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
248 h5f = tables.openFile(h5f_path, "r")
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
249
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
250 b1_table = h5f.getNode('/params', 'b1')
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
251 b3_table = h5f.getNode('/params', 'b3')
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
252
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
253 assert b1_table.cols.mean[0] - numpy.mean(b1.value) < 1e-3
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
254 assert b3_table.cols.mean[0] - numpy.mean(b3.value) < 1e-3
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
255 assert b1_table.cols.min[0] - numpy.min(b1.value) < 1e-3
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
256 assert b3_table.cols.min[0] - numpy.min(b3.value) < 1e-3
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
257
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
258 assert not ('timestamp' in dir(b1_table.cols))
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas
fsavard
parents: 220
diff changeset
259
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
260 def test_get_desc():
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
261 h5f_path = tempfile.NamedTemporaryFile().name
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
262 h5f = tables.openFile(h5f_path, "w")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
263
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
264 desc = series._get_description_with_n_ints_n_floats(("col1","col2"), ("col3","col4"))
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
265
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
266 mytable = h5f.createTable('/', 'mytable', desc)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
267
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
268 # just make sure the columns are there... otherwise this will throw an exception
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
269 mytable.cols.col1
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
270 mytable.cols.col2
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
271 mytable.cols.col3
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
272 mytable.cols.col4
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
273
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
274 try:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
275 # this should fail... LocalDescription must be local to get_desc_etc
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
276 test = LocalDescription
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
277 assert False
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
278 except:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
279 assert True
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
280
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
281 assert True
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
282
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
283 def test_index_to_tuple_floaterror():
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
284 try:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
285 series._index_to_tuple(5.1)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
286 assert False
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
287 except TypeError:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
288 assert True
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
289
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
290 def test_index_to_tuple_arrayok():
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
291 tpl = series._index_to_tuple([1,2,3])
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
292 assert type(tpl) == tuple and tpl[1] == 2 and tpl[2] == 3
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
293
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
294 def test_index_to_tuple_intbecomestuple():
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
295 tpl = series._index_to_tuple(32)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
296
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
297 assert type(tpl) == tuple and tpl == (32,)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
298
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
299 def test_index_to_tuple_longbecomestuple():
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
300 tpl = series._index_to_tuple(928374928374928L)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
301
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
302 assert type(tpl) == tuple and tpl == (928374928374928L,)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
303
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
304 if __name__ == '__main__':
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
305 import tempfile
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
306 test_get_desc()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
307 test_ErrorSeries_common_case()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
308 test_BasicStatisticsSeries_common_case()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
309 test_AccumulatorSeriesWrapper_common_case()
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
310 test_SharedParamsStatisticsWrapper_commoncase()
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
311