Mercurial > ift6266

annotate utils/seriestables/series.py @ 220:e172ef73cdc5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
author fsavard
date Thu, 11 Mar 2010 10:48:54 -0500
parents 4c137f16b013
children 02d9c1279dd8
rev   line source
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
1 import tables
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
2
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
3 import numpy
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
4 import time
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
5
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
6 ##############################################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
7 # Utility functions to create IsDescription objects (pytables data types)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
8
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
9 '''
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
10 The way these "IsDescription constructor" work is simple: write the
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
11 code as if it were in a file, then exec()ute it, leaving us with
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
12 a local-scoped LocalDescription which may be used to call createTable.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
13
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
14 It's a small hack, but it's necessary as the names of the columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
15 are retrieved based on the variable name, which we can't programmatically set
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
16 otherwise.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
17 '''
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
18
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
19 def _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock, pos=0):
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
20 toexec = ""
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
21
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
22 if store_timestamp:
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
23 toexec += "\ttimestamp = tables.Time32Col(pos="+str(pos)+")\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
24 pos += 1
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
25
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
26 if store_cpuclock:
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
27 toexec += "\tcpuclock = tables.Float64Col(pos="+str(pos)+")\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
28 pos += 1
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
29
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
30 return toexec, pos
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
31
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
32 def _get_description_n_ints(int_names, int_width=64, pos=0):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
33 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
34 Begins construction of a class inheriting from IsDescription
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
35 to construct an HDF5 table with index columns named with int_names.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
36
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
37 See Series().__init__ to see how those are used.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
38 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
39 int_constructor = "tables.Int64Col"
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
40 if int_width == 32:
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
41 int_constructor = "tables.Int32Col"
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
42
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
43 toexec = ""
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
44
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
45 for n in int_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
46 toexec += "\t" + n + " = " + int_constructor + "(pos=" + str(pos) + ")\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
47 pos += 1
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
48
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
49 return toexec, pos
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
50
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
51 def _get_description_with_n_ints_n_floats(int_names, float_names,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
52 int_width=64, float_width=32,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
53 store_timestamp=True, store_cpuclock=True):
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
54 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
55 Constructs a class to be used when constructing a table with PyTables.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
56
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
57 This is useful to construct a series with an index with multiple levels.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
58 E.g. if you want to index your "validation error" with "epoch" first, then
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
59 "minibatch_index" second, you'd use two "int_names".
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
60
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
61 Parameters
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
62 ----------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
63 int_names : tuple of str
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
64 Names of the int (e.g. index) columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
65 float_names : tuple of str
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
66 Names of the float (e.g. error) columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
67 int_width : {'32', '64'}
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
68 Type of ints.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
69 float_width : {'32', '64'}
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
70 Type of floats.
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
71 store_timestamp : bool
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
72 See __init__ of Series
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
73 store_cpuclock : bool
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
74 See __init__ of Series
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
75
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
76 Returns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
77 -------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
78 A class object, to pass to createTable()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
79 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
80
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
81 toexec = "class LocalDescription(tables.IsDescription):\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
82
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
83 toexec_, pos = _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
84 toexec += toexec_
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
85
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
86 toexec_, pos = _get_description_n_ints(int_names, int_width=int_width, pos=pos)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
87 toexec += toexec_
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
88
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
89 float_constructor = "tables.Float32Col"
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
90 if float_width == 64:
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
91 float_constructor = "tables.Float64Col"
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
92
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
93 for n in float_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
94 toexec += "\t" + n + " = " + float_constructor + "(pos=" + str(pos) + ")\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
95 pos += 1
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
96
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
97 exec(toexec)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
98
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
99 return LocalDescription
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
100
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
101 ##############################################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
102 # Series classes
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
103
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
104 # Shortcut to allow passing a single int as index, instead of a tuple
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
105 def _index_to_tuple(index):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
106 if type(index) == tuple:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
107 return index
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
108
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
109 if type(index) == list:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
110 index = tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
111 return index
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
112
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
113 try:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
114 if index % 1 > 0.001 and index % 1 < 0.999:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
115 raise
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
116 idx = long(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
117 return (idx,)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
118 except:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
119 raise TypeError("index must be a tuple of integers, or at least a single integer")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
120
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
121 class Series():
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
122 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
123 Base Series class, with minimal arguments and type checks.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
124
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
125 Yet cannot be used by itself (it's append() method raises an error)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
126 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
127
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
128 def __init__(self, table_name, hdf5_file, index_names=('epoch',),
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
129 title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
130 store_timestamp=True, store_cpuclock=True):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
131 """Basic arguments each Series must get.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
132
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
133 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
134 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
135 table_name : str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
136 Name of the table to create under group "hd5_group" (other
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
137 parameter). No spaces, ie. follow variable naming restrictions.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
138 hdf5_file : open HDF5 file
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
139 File opened with openFile() in PyTables (ie. return value of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
140 openFile).
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
141 index_names : tuple of str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
142 Columns to use as index for elements in the series, other
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
143 example would be ('epoch', 'minibatch'). This would then allow
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
144 you to call append(index, element) with index made of two ints,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
145 one for epoch index, one for minibatch index in epoch.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
146 title : str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
147 Title to attach to this table as metadata. Can contain spaces
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
148 and be longer then the table_name.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
149 hdf5_group : str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
150 Path of the group (kind of a file) in the HDF5 file under which
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
151 to create the table.
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
152 store_timestamp : bool
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
153 Whether to create a column for timestamps and store them with
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
154 each record.
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
155 store_cpuclock : bool
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
156 Whether to create a column for cpu clock and store it with
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
157 each record.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
158 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
159
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
160 #########################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
161 # checks
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
162
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
163 if type(table_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
164 raise TypeError("table_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
165 if table_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
166 raise ValueError("table_name must not be empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
167
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
168 if not isinstance(hdf5_file, tables.file.File):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
169 raise TypeError("hdf5_file must be an open HDF5 file (use tables.openFile)")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
170 #if not ('w' in hdf5_file.mode or 'a' in hdf5_file.mode):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
171 # raise ValueError("hdf5_file must be opened in write or append mode")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
172
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
173 if type(index_names) != tuple:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
174 raise TypeError("index_names must be a tuple of strings." + \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
175 "If you have only one element in the tuple, don't forget " +\
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
176 "to add a comma, e.g. ('epoch',).")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
177 for name in index_names:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
178 if type(name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
179 raise TypeError("index_names must only contain strings, but also"+\
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
180 "contains a "+str(type(name))+".")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
181
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
182 if type(title) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
183 raise TypeError("title must be a string, even if empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
184
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
185 if type(hdf5_group) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
186 raise TypeError("hdf5_group must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
187
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
188 if type(store_timestamp) != bool:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
189 raise TypeError("store_timestamp must be a bool")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
190
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
191 if type(store_cpuclock) != bool:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
192 raise TypeError("store_timestamp must be a bool")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
193
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
194 #########################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
195
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
196 self.table_name = table_name
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
197 self.hdf5_file = hdf5_file
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
198 self.index_names = index_names
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
199 self.title = title
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
200 self.hdf5_group = hdf5_group
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
201
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
202 self.store_timestamp = store_timestamp
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
203 self.store_cpuclock = store_cpuclock
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
204
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
205 def append(self, index, element):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
206 raise NotImplementedError
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
207
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
208 def _timestamp_cpuclock(self, newrow):
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
209 newrow["timestamp"] = time.time()
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
210 newrow["cpuclock"] = time.clock()
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
211
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
212 class DummySeries():
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
213 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
214 To put in a series dictionary instead of a real series, to do nothing
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
215 when we don't want a given series to be saved.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
216
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
217 E.g. if we'd normally have a "training_error" series in a dictionary
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
218 of series, the training loop would have something like this somewhere:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
219
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
220 series["training_error"].append((15,), 20.0)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
221
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
222 but if we don't want to save the training errors this time, we simply
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
223 do
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
224
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
225 series["training_error"] = DummySeries()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
226 """
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
227 def append(self, index, element):
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
228 pass
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
229
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
230 class ErrorSeries(Series):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
231 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
232 Most basic Series: saves a single float (called an Error as this is
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
233 the most common use case I foresee) along with an index (epoch, for
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
234 example) and timestamp/cpu.clock for each of these floats.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
235 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
236
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
237 def __init__(self, error_name, table_name,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
238 hdf5_file, index_names=('epoch',),
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
239 title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
240 store_timestamp=True, store_cpuclock=True):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
241 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
242 For most parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
243
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
244 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
245 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
246 error_name : str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
247 In the HDF5 table, column name for the error float itself.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
248 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
249
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
250 # most type/value checks are performed in Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
251 Series.__init__(self, table_name, hdf5_file, index_names, title,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
252 store_timestamp=store_timestamp,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
253 store_cpuclock=store_cpuclock)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
254
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
255 if type(error_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
256 raise TypeError("error_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
257 if error_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
258 raise ValueError("error_name must not be empty")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
259
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
260 self.error_name = error_name
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
261
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
262 self._create_table()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
263
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
264 def _create_table(self):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
265 table_description = _get_description_with_n_ints_n_floats( \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
266 self.index_names, (self.error_name,))
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
267
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
268 self._table = self.hdf5_file.createTable(self.hdf5_group,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
269 self.table_name,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
270 table_description,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
271 title=self.title)
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
272
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
273
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
274 def append(self, index, error):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
275 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
276 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
277 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
278 index : tuple of int
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
279 Following index_names passed to __init__, e.g. (12, 15) if
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
280 index_names were ('epoch', 'minibatch_size').
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
281 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
282 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
283 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
284
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
285 error : float
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
286 Next error in the series.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
287 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
288 index = _index_to_tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
289
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
290 if len(index) != len(self.index_names):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
291 raise ValueError("index provided does not have the right length (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
292 + str(len(self.index_names)) + " got " + str(len(index)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
293
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
294 # other checks are implicit when calling newrow[..] =,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
295 # which should throw an error if not of the right type
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
296
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
297 newrow = self._table.row
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
298
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
299 # Columns for index in table are based on index_names
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
300 for col_name, value in zip(self.index_names, index):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
301 newrow[col_name] = value
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
302 newrow[self.error_name] = error
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
303
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
304 # adds timestamp and cpuclock to newrow if necessary
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
305 self._timestamp_cpuclock(newrow)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
306
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
307 newrow.append()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
308
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
309 self.hdf5_file.flush()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
310
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
311 # Does not inherit from Series because it does not itself need to
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
312 # access the hdf5_file and does not need a series_name (provided
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
313 # by the base_series.)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
314 class AccumulatorSeriesWrapper():
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
315 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
316 Wraps a Series by accumulating objects passed its Accumulator.append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
317 method and "reducing" (e.g. calling numpy.mean(list)) once in a while,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
318 every "reduce_every" calls in fact.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
319 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
320
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
321 def __init__(self, base_series, reduce_every, reduce_function=numpy.mean):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
322 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
323 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
324 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
325 base_series : Series
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
326 This object must have an append(index, value) function.
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
327
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
328 reduce_every : int
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
329 Apply the reduction function (e.g. mean()) every time we get this
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
330 number of elements. E.g. if this is 100, then every 100 numbers
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
331 passed to append(), we'll take the mean and call append(this_mean)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
332 on the BaseSeries.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
333
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
334 reduce_function : function
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
335 Must take as input an array of "elements", as passed to (this
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
336 accumulator's) append(). Basic case would be to take an array of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
337 floats and sum them into one float, for example.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
338 """
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
339 self.base_series = base_series
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
340 self.reduce_function = reduce_function
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
341 self.reduce_every = reduce_every
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
342
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
343 self._buffer = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
344
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
345
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
346 def append(self, index, element):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
347 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
348 Parameters
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
349 ----------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
350 index : tuple of int
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
351 The index used is the one of the last element reduced. E.g. if
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
352 you accumulate over the first 1000 minibatches, the index
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
353 passed to the base_series.append() function will be 1000.
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
354 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
355 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
356 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
357
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
358 element : float
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
359 Element that will be accumulated.
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
360 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
361 self._buffer.append(element)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
362
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
363 if len(self._buffer) == self.reduce_every:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
364 reduced = self.reduce_function(self._buffer)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
365 self.base_series.append(index, reduced)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
366 self._buffer = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
367
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
368 # The >= case should never happen, except if lists
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
369 # were appended, which should be a red flag.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
370 assert len(self._buffer) < self.reduce_every
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
371
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
372 # Outside of class to fix an issue with exec in Python 2.6.
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
373 # My sorries to the god of pretty code.
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
374 def _BasicStatisticsSeries_construct_table_toexec(index_names, store_timestamp, store_cpuclock):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
375 toexec = "class LocalDescription(tables.IsDescription):\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
376
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
377 toexec_, pos = _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
378 toexec += toexec_
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
379
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
380 toexec_, pos = _get_description_n_ints(index_names, pos=pos)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
381 toexec += toexec_
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
382
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
383 toexec += "\tmean = tables.Float32Col(pos=" + str(pos) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
384 toexec += "\tmin = tables.Float32Col(pos=" + str(pos+1) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
385 toexec += "\tmax = tables.Float32Col(pos=" + str(pos+2) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
386 toexec += "\tstd = tables.Float32Col(pos=" + str(pos+3) + ")\n"
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
387
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
388 # This creates "LocalDescription", which we may then use
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
389 exec(toexec)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
390
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
391 return LocalDescription
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
392
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
393 # Defaults functions for BasicStatsSeries. These can be replaced.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
394 _basic_stats_functions = {'mean': lambda(x): numpy.mean(x),
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
395 'min': lambda(x): numpy.min(x),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
396 'max': lambda(x): numpy.max(x),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
397 'std': lambda(x): numpy.std(x)}
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
398
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
399 class BasicStatisticsSeries(Series):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
400
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
401 def __init__(self, table_name, hdf5_file,
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
402 stats_functions=_basic_stats_functions,
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
403 index_names=('epoch',), title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
404 store_timestamp=True, store_cpuclock=True):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
405 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
406 For most parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
407
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
408 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
409 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
410 series_name : str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
411 Not optional here. Will be prepended with "Basic statistics for "
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
412
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
413 stats_functions : dict, optional
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
414 Dictionary with a function for each key "mean", "min", "max",
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
415 "std". The function must take whatever is passed to append(...)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
416 and return a single number (float).
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
417 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
418
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
419 # Most type/value checks performed in Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
420 Series.__init__(self, table_name, hdf5_file, index_names, title,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
421 store_timestamp=store_timestamp,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
422 store_cpuclock=store_cpuclock)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
423
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
424 if type(hdf5_group) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
425 raise TypeError("hdf5_group must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
426
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
427 if type(stats_functions) != dict:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
428 # just a basic check. We'll suppose caller knows what he's doing.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
429 raise TypeError("stats_functions must be a dict")
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
430
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
431 self.hdf5_group = hdf5_group
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
432
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
433 self.stats_functions = stats_functions
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
434
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
435 self._create_table()
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
436
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
437 def _create_table(self):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
438 table_description = \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
439 _BasicStatisticsSeries_construct_table_toexec( \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
440 self.index_names,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
441 self.store_timestamp, self.store_cpuclock)
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
442
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
443 self._table = self.hdf5_file.createTable(self.hdf5_group,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
444 self.table_name, table_description)
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
445
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
446 def append(self, index, array):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
447 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
448 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
449 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
450 index : tuple of int
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
451 Following index_names passed to __init__, e.g. (12, 15)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
452 if index_names were ('epoch', 'minibatch_size')
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
453 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
454 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
455 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
456
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
457 array
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
458 Is of whatever type the stats_functions passed to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
459 __init__ can take. Default is anything numpy.mean(),
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
460 min(), max(), std() can take.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
461 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
462 index = _index_to_tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
463
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
464 if len(index) != len(self.index_names):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
465 raise ValueError("index provided does not have the right length (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
466 + str(len(self.index_names)) + " got " + str(len(index)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
467
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
468 newrow = self._table.row
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
469
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
470 for col_name, value in zip(self.index_names, index):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
471 newrow[col_name] = value
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
472
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
473 newrow["mean"] = self.stats_functions['mean'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
474 newrow["min"] = self.stats_functions['min'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
475 newrow["max"] = self.stats_functions['max'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
476 newrow["std"] = self.stats_functions['std'](array)
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
477
218
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
478 self._timestamp_cpuclock(newrow)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin
fsavard
parents: 213
diff changeset
479
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
480 newrow.append()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
481
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
482 self.hdf5_file.flush()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
483
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
484 class SeriesArrayWrapper():
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
485 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
486 Simply redistributes any number of elements to sub-series to respective
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
487 append()s.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
488
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
489 To use if you have many elements to append in similar series, e.g. if you
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
490 have an array containing [train_error, valid_error, test_error], and 3
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
491 corresponding series, this allows you to simply pass this array of 3
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
492 values to append() instead of passing each element to each individual
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
493 series in turn.
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
494 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
495
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
496 def __init__(self, base_series_list):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
497 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
498 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
499 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
500 base_series_list : array or tuple of Series
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
501 You must have previously created and configured each of those
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
502 series, then put them in an array. This array must follow the
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
503 same order as the array passed as ``elements`` parameter of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
504 append().
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
505 """
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
506 self.base_series_list = base_series_list
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
507
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
508 def append(self, index, elements):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
509 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
510 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
511 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
512 index : tuple of int
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
513 See for example ErrorSeries.append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
514
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
515 elements : array or tuple
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
516 Array or tuple of elements that will be passed down to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
517 the base_series passed to __init__, in the same order.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
518 """
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
519 if len(elements) != len(self.base_series_list):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
520 raise ValueError("not enough or too much elements provided (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
521 + str(len(self.base_series_list)) + " got " + str(len(elements)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
522
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
523 for series, el in zip(self.base_series_list, elements):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
524 series.append(index, el)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
525
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
526 class SharedParamsStatisticsWrapper(SeriesArrayWrapper):
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
527 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
528 Save mean, min/max, std of shared parameters place in an array.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
529
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
530 Here "shared" means "theano.shared", which means elements of the
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
531 array will have a .value to use for numpy.mean(), etc.
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
532
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
533 This inherits from SeriesArrayWrapper, which provides the append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
534 method.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
535 '''
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
536
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
537 def __init__(self, arrays_names, new_group_name, hdf5_file,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
538 base_group='/', index_names=('epoch',), title=""):
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
539 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
540 For other parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
541
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
542 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
543 ----------
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
544 array_names : array or tuple of str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
545 Name of each array, in order of the array passed to append(). E.g.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
546 ('layer1_b', 'layer1_W', 'layer2_b', 'layer2_W')
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
547
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
548 new_group_name : str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
549 Name of a new HDF5 group which will be created under base_group to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
550 store the new series.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
551
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
552 base_group : str
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
553 Path of the group under which to create the new group which will
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
554 store the series.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
555
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
556 title : str
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
557 Here the title is attached to the new group, not a table.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
558 """
220
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
559
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
560 # most other checks done when calling BasicStatisticsSeries
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
561 if type(new_group_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
562 raise TypeError("new_group_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
563 if new_group_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
564 raise ValueError("new_group_name must not be empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes.
fsavard
parents: 218
diff changeset
565
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
566 base_series_list = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
567
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
568 new_group = hdf5_file.createGroup(base_group, new_group_name, title=title)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
569
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
570 stats_functions = {'mean': lambda(x): numpy.mean(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
571 'min': lambda(x): numpy.min(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
572 'max': lambda(x): numpy.max(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
573 'std': lambda(x): numpy.std(x.value)}
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
574
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
575 for name in arrays_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
576 base_series_list.append(
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
577 BasicStatisticsSeries(
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
578 table_name=name,
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
579 hdf5_file=hdf5_file,
210
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
580 index_names=index_names,
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series
savardf
parents: 208
diff changeset
581 stats_functions=stats_functions,
208
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
582 hdf5_group=new_group._v_pathname))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
583
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
584 SeriesArrayWrapper.__init__(self, base_series_list)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
585
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae.
fsavard
parents:
diff changeset
586