ift6266: utils/seriestables/series.py annotate

annotate utils/seriestables/series.py @ 441:1272dc84a30c

merge

author	Arnaud Bergeron <abergeron@gmail.com>
date	Mon, 03 May 2010 13:55:03 -0400
parents	0515a8901c6a
children

rev	line source
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	1 import tables
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	2
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	3 import numpy
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	4 import time
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	5
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	6 ##############################################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	7 # Utility functions to create IsDescription objects (pytables data types)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	8
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	9 '''
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	10 The way these "IsDescription constructor" work is simple: write the
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	11 code as if it were in a file, then exec()ute it, leaving us with
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	12 a local-scoped LocalDescription which may be used to call createTable.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	13
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	14 It's a small hack, but it's necessary as the names of the columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	15 are retrieved based on the variable name, which we can't programmatically set
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	16 otherwise.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	17 '''
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	18
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	19 def _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock, pos=0):
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	20 toexec = ""
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	21
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	22 if store_timestamp:
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	23 toexec += "\ttimestamp = tables.Time32Col(pos="+str(pos)+")\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	24 pos += 1
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	25
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	26 if store_cpuclock:
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	27 toexec += "\tcpuclock = tables.Float64Col(pos="+str(pos)+")\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	28 pos += 1
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	29
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	30 return toexec, pos
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	31
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	32 def _get_description_n_ints(int_names, int_width=64, pos=0):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	33 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	34 Begins construction of a class inheriting from IsDescription
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	35 to construct an HDF5 table with index columns named with int_names.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	36
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	37 See Series().__init__ to see how those are used.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	38 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	39 int_constructor = "tables.Int64Col"
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	40 if int_width == 32:
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	41 int_constructor = "tables.Int32Col"
221 02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	42 elif not int_width in (32, 64):
02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	43 raise "int_width must be left unspecified, or should equal 32 or 64"
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	44
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	45 toexec = ""
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	46
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	47 for n in int_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	48 toexec += "\t" + n + " = " + int_constructor + "(pos=" + str(pos) + ")\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	49 pos += 1
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	50
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	51 return toexec, pos
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	52
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	53 def _get_description_with_n_ints_n_floats(int_names, float_names,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	54 int_width=64, float_width=32,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	55 store_timestamp=True, store_cpuclock=True):
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	56 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	57 Constructs a class to be used when constructing a table with PyTables.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	58
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	59 This is useful to construct a series with an index with multiple levels.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	60 E.g. if you want to index your "validation error" with "epoch" first, then
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	61 "minibatch_index" second, you'd use two "int_names".
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	62
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	63 Parameters
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	64 ----------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	65 int_names : tuple of str
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	66 Names of the int (e.g. index) columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	67 float_names : tuple of str
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	68 Names of the float (e.g. error) columns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	69 int_width : {'32', '64'}
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	70 Type of ints.
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	71 float_width : {'32', '64'}
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	72 Type of floats.
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	73 store_timestamp : bool
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	74 See __init__ of Series
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	75 store_cpuclock : bool
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	76 See __init__ of Series
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	77
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	78 Returns
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	79 -------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	80 A class object, to pass to createTable()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	81 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	82
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	83 toexec = "class LocalDescription(tables.IsDescription):\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	84
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	85 toexec_, pos = _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	86 toexec += toexec_
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	87
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	88 toexec_, pos = _get_description_n_ints(int_names, int_width=int_width, pos=pos)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	89 toexec += toexec_
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	90
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	91 float_constructor = "tables.Float32Col"
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	92 if float_width == 64:
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	93 float_constructor = "tables.Float64Col"
221 02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	94 elif not float_width in (32, 64):
02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	95 raise "float_width must be left unspecified, or should equal 32 or 64"
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	96
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	97 for n in float_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	98 toexec += "\t" + n + " = " + float_constructor + "(pos=" + str(pos) + ")\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	99 pos += 1
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	100
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	101 exec(toexec)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	102
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	103 return LocalDescription
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	104
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	105 ##############################################################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	106 # Series classes
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	107
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	108 # Shortcut to allow passing a single int as index, instead of a tuple
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	109 def _index_to_tuple(index):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	110 if type(index) == tuple:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	111 return index
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	112
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	113 if type(index) == list:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	114 index = tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	115 return index
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	116
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	117 try:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	118 if index % 1 > 0.001 and index % 1 < 0.999:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	119 raise
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	120 idx = long(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	121 return (idx,)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	122 except:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	123 raise TypeError("index must be a tuple of integers, or at least a single integer")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	124
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	125 class Series():
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	126 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	127 Base Series class, with minimal arguments and type checks.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	128
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	129 Yet cannot be used by itself (it's append() method raises an error)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	130 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	131
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	132 def __init__(self, table_name, hdf5_file, index_names=('epoch',),
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	133 title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	134 store_timestamp=True, store_cpuclock=True):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	135 """Basic arguments each Series must get.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	136
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	137 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	138 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	139 table_name : str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	140 Name of the table to create under group "hd5_group" (other
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	141 parameter). No spaces, ie. follow variable naming restrictions.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	142 hdf5_file : open HDF5 file
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	143 File opened with openFile() in PyTables (ie. return value of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	144 openFile).
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	145 index_names : tuple of str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	146 Columns to use as index for elements in the series, other
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	147 example would be ('epoch', 'minibatch'). This would then allow
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	148 you to call append(index, element) with index made of two ints,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	149 one for epoch index, one for minibatch index in epoch.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	150 title : str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	151 Title to attach to this table as metadata. Can contain spaces
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	152 and be longer then the table_name.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	153 hdf5_group : str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	154 Path of the group (kind of a file) in the HDF5 file under which
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	155 to create the table.
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	156 store_timestamp : bool
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	157 Whether to create a column for timestamps and store them with
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	158 each record.
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	159 store_cpuclock : bool
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	160 Whether to create a column for cpu clock and store it with
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	161 each record.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	162 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	163
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	164 #########################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	165 # checks
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	166
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	167 if type(table_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	168 raise TypeError("table_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	169 if table_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	170 raise ValueError("table_name must not be empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	171
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	172 if not isinstance(hdf5_file, tables.file.File):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	173 raise TypeError("hdf5_file must be an open HDF5 file (use tables.openFile)")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	174 #if not ('w' in hdf5_file.mode or 'a' in hdf5_file.mode):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	175 # raise ValueError("hdf5_file must be opened in write or append mode")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	176
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	177 if type(index_names) != tuple:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	178 raise TypeError("index_names must be a tuple of strings." + \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	179 "If you have only one element in the tuple, don't forget " +\
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	180 "to add a comma, e.g. ('epoch',).")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	181 for name in index_names:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	182 if type(name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	183 raise TypeError("index_names must only contain strings, but also"+\
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	184 "contains a "+str(type(name))+".")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	185
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	186 if type(title) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	187 raise TypeError("title must be a string, even if empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	188
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	189 if type(hdf5_group) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	190 raise TypeError("hdf5_group must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	191
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	192 if type(store_timestamp) != bool:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	193 raise TypeError("store_timestamp must be a bool")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	194
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	195 if type(store_cpuclock) != bool:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	196 raise TypeError("store_timestamp must be a bool")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	197
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	198 #########################################
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	199
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	200 self.table_name = table_name
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	201 self.hdf5_file = hdf5_file
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	202 self.index_names = index_names
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	203 self.title = title
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	204 self.hdf5_group = hdf5_group
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	205
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	206 self.store_timestamp = store_timestamp
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	207 self.store_cpuclock = store_cpuclock
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	208
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	209 def append(self, index, element):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	210 raise NotImplementedError
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	211
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	212 def _timestamp_cpuclock(self, newrow):
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	213 if self.store_timestamp:
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	214 newrow["timestamp"] = time.time()
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	215
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	216 if self.store_cpuclock:
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	217 newrow["cpuclock"] = time.clock()
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	218
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	219 class DummySeries():
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	220 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	221 To put in a series dictionary instead of a real series, to do nothing
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	222 when we don't want a given series to be saved.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	223
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	224 E.g. if we'd normally have a "training_error" series in a dictionary
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	225 of series, the training loop would have something like this somewhere:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	226
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	227 series["training_error"].append((15,), 20.0)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	228
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	229 but if we don't want to save the training errors this time, we simply
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	230 do
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	231
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	232 series["training_error"] = DummySeries()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	233 """
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	234 def append(self, index, element):
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	235 pass
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	236
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	237 class ErrorSeries(Series):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	238 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	239 Most basic Series: saves a single float (called an Error as this is
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	240 the most common use case I foresee) along with an index (epoch, for
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	241 example) and timestamp/cpu.clock for each of these floats.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	242 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	243
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	244 def __init__(self, error_name, table_name,
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	245 hdf5_file, index_names=('epoch',),
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	246 title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	247 store_timestamp=True, store_cpuclock=True):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	248 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	249 For most parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	250
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	251 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	252 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	253 error_name : str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	254 In the HDF5 table, column name for the error float itself.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	255 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	256
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	257 # most type/value checks are performed in Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	258 Series.__init__(self, table_name, hdf5_file, index_names, title,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	259 store_timestamp=store_timestamp,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	260 store_cpuclock=store_cpuclock)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	261
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	262 if type(error_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	263 raise TypeError("error_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	264 if error_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	265 raise ValueError("error_name must not be empty")
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	266
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	267 self.error_name = error_name
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	268
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	269 self._create_table()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	270
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	271 def _create_table(self):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	272 table_description = _get_description_with_n_ints_n_floats( \
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	273 self.index_names, (self.error_name,),
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	274 store_timestamp=self.store_timestamp,
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	275 store_cpuclock=self.store_cpuclock)
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	276
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	277 self._table = self.hdf5_file.createTable(self.hdf5_group,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	278 self.table_name,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	279 table_description,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	280 title=self.title)
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	281
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	282
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	283 def append(self, index, error):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	284 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	285 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	286 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	287 index : tuple of int
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	288 Following index_names passed to __init__, e.g. (12, 15) if
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	289 index_names were ('epoch', 'minibatch_size').
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	290 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	291 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	292 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	293
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	294 error : float
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	295 Next error in the series.
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	296 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	297 index = _index_to_tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	298
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	299 if len(index) != len(self.index_names):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	300 raise ValueError("index provided does not have the right length (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	301 + str(len(self.index_names)) + " got " + str(len(index)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	302
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	303 # other checks are implicit when calling newrow[..] =,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	304 # which should throw an error if not of the right type
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	305
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	306 newrow = self._table.row
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	307
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	308 # Columns for index in table are based on index_names
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	309 for col_name, value in zip(self.index_names, index):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	310 newrow[col_name] = value
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	311 newrow[self.error_name] = error
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	312
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	313 # adds timestamp and cpuclock to newrow if necessary
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	314 self._timestamp_cpuclock(newrow)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	315
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	316 newrow.append()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	317
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	318 self.hdf5_file.flush()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	319
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	320 # Does not inherit from Series because it does not itself need to
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	321 # access the hdf5_file and does not need a series_name (provided
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	322 # by the base_series.)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	323 class AccumulatorSeriesWrapper():
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	324 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	325 Wraps a Series by accumulating objects passed its Accumulator.append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	326 method and "reducing" (e.g. calling numpy.mean(list)) once in a while,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	327 every "reduce_every" calls in fact.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	328 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	329
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	330 def __init__(self, base_series, reduce_every, reduce_function=numpy.mean):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	331 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	332 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	333 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	334 base_series : Series
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	335 This object must have an append(index, value) function.
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	336
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	337 reduce_every : int
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	338 Apply the reduction function (e.g. mean()) every time we get this
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	339 number of elements. E.g. if this is 100, then every 100 numbers
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	340 passed to append(), we'll take the mean and call append(this_mean)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	341 on the BaseSeries.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	342
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	343 reduce_function : function
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	344 Must take as input an array of "elements", as passed to (this
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	345 accumulator's) append(). Basic case would be to take an array of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	346 floats and sum them into one float, for example.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	347 """
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	348 self.base_series = base_series
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	349 self.reduce_function = reduce_function
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	350 self.reduce_every = reduce_every
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	351
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	352 self._buffer = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	353
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	354
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	355 def append(self, index, element):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	356 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	357 Parameters
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	358 ----------
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	359 index : tuple of int
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	360 The index used is the one of the last element reduced. E.g. if
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	361 you accumulate over the first 1000 minibatches, the index
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	362 passed to the base_series.append() function will be 1000.
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	363 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	364 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	365 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	366
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	367 element : float
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	368 Element that will be accumulated.
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	369 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	370 self._buffer.append(element)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	371
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	372 if len(self._buffer) == self.reduce_every:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	373 reduced = self.reduce_function(self._buffer)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	374 self.base_series.append(index, reduced)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	375 self._buffer = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	376
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	377 # The >= case should never happen, except if lists
221 02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	378 # were appended by accessing _buffer externally (when it's
02d9c1279dd8 Encore quelques typechecks à SeriesTables. fsavard parents: 220 diff changeset	379 # intended to be private), which should be a red flag.
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	380 assert len(self._buffer) < self.reduce_every
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	381
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	382 # Outside of class to fix an issue with exec in Python 2.6.
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	383 # My sorries to the god of pretty code.
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	384 def _BasicStatisticsSeries_construct_table_toexec(index_names, store_timestamp, store_cpuclock):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	385 toexec = "class LocalDescription(tables.IsDescription):\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	386
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	387 toexec_, pos = _get_description_timestamp_cpuclock_columns(store_timestamp, store_cpuclock)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	388 toexec += toexec_
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	389
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	390 toexec_, pos = _get_description_n_ints(index_names, pos=pos)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	391 toexec += toexec_
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	392
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	393 toexec += "\tmean = tables.Float32Col(pos=" + str(pos) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	394 toexec += "\tmin = tables.Float32Col(pos=" + str(pos+1) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	395 toexec += "\tmax = tables.Float32Col(pos=" + str(pos+2) + ")\n"
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	396 toexec += "\tstd = tables.Float32Col(pos=" + str(pos+3) + ")\n"
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	397
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	398 # This creates "LocalDescription", which we may then use
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	399 exec(toexec)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	400
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	401 return LocalDescription
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	402
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	403 # Defaults functions for BasicStatsSeries. These can be replaced.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	404 _basic_stats_functions = {'mean': lambda(x): numpy.mean(x),
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	405 'min': lambda(x): numpy.min(x),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	406 'max': lambda(x): numpy.max(x),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	407 'std': lambda(x): numpy.std(x)}
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	408
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	409 class BasicStatisticsSeries(Series):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	410
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	411 def __init__(self, table_name, hdf5_file,
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	412 stats_functions=_basic_stats_functions,
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	413 index_names=('epoch',), title="", hdf5_group='/',
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	414 store_timestamp=True, store_cpuclock=True):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	415 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	416 For most parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	417
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	418 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	419 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	420 series_name : str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	421 Not optional here. Will be prepended with "Basic statistics for "
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	422
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	423 stats_functions : dict, optional
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	424 Dictionary with a function for each key "mean", "min", "max",
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	425 "std". The function must take whatever is passed to append(...)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	426 and return a single number (float).
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	427 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	428
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	429 # Most type/value checks performed in Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	430 Series.__init__(self, table_name, hdf5_file, index_names, title,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	431 store_timestamp=store_timestamp,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	432 store_cpuclock=store_cpuclock)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	433
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	434 if type(hdf5_group) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	435 raise TypeError("hdf5_group must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	436
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	437 if type(stats_functions) != dict:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	438 # just a basic check. We'll suppose caller knows what he's doing.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	439 raise TypeError("stats_functions must be a dict")
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	440
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	441 self.hdf5_group = hdf5_group
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	442
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	443 self.stats_functions = stats_functions
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	444
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	445 self._create_table()
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	446
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	447 def _create_table(self):
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	448 table_description = \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	449 _BasicStatisticsSeries_construct_table_toexec( \
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	450 self.index_names,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	451 self.store_timestamp, self.store_cpuclock)
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	452
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	453 self._table = self.hdf5_file.createTable(self.hdf5_group,
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	454 self.table_name, table_description)
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	455
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	456 def append(self, index, array):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	457 """
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	458 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	459 ----------
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	460 index : tuple of int
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	461 Following index_names passed to __init__, e.g. (12, 15)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	462 if index_names were ('epoch', 'minibatch_size')
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	463 A single int (not tuple) is acceptable if index_names has a single
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	464 element.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	465 An array will be casted to a tuple, as a convenience.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	466
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	467 array
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	468 Is of whatever type the stats_functions passed to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	469 __init__ can take. Default is anything numpy.mean(),
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	470 min(), max(), std() can take.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	471 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	472 index = _index_to_tuple(index)
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	473
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	474 if len(index) != len(self.index_names):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	475 raise ValueError("index provided does not have the right length (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	476 + str(len(self.index_names)) + " got " + str(len(index)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	477
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	478 newrow = self._table.row
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	479
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	480 for col_name, value in zip(self.index_names, index):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	481 newrow[col_name] = value
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	482
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	483 newrow["mean"] = self.stats_functions['mean'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	484 newrow["min"] = self.stats_functions['min'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	485 newrow["max"] = self.stats_functions['max'](array)
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	486 newrow["std"] = self.stats_functions['std'](array)
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	487
218 4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	488 self._timestamp_cpuclock(newrow)
4c137f16b013 Modifications pour stocker des timestamps/cpuclock avec chaque rangée créée, selon suggestion de Yoshua ce matin fsavard parents: 213 diff changeset	489
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	490 newrow.append()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	491
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	492 self.hdf5_file.flush()
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	493
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	494 class SeriesArrayWrapper():
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	495 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	496 Simply redistributes any number of elements to sub-series to respective
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	497 append()s.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	498
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	499 To use if you have many elements to append in similar series, e.g. if you
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	500 have an array containing [train_error, valid_error, test_error], and 3
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	501 corresponding series, this allows you to simply pass this array of 3
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	502 values to append() instead of passing each element to each individual
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	503 series in turn.
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	504 """
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	505
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	506 def __init__(self, base_series_list):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	507 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	508 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	509 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	510 base_series_list : array or tuple of Series
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	511 You must have previously created and configured each of those
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	512 series, then put them in an array. This array must follow the
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	513 same order as the array passed as ``elements`` parameter of
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	514 append().
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	515 """
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	516 self.base_series_list = base_series_list
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	517
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	518 def append(self, index, elements):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	519 """
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	520 Parameters
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	521 ----------
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	522 index : tuple of int
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	523 See for example ErrorSeries.append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	524
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	525 elements : array or tuple
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	526 Array or tuple of elements that will be passed down to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	527 the base_series passed to __init__, in the same order.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	528 """
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	529 if len(elements) != len(self.base_series_list):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	530 raise ValueError("not enough or too much elements provided (expected " \
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	531 + str(len(self.base_series_list)) + " got " + str(len(elements)))
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	532
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	533 for series, el in zip(self.base_series_list, elements):
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	534 series.append(index, el)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	535
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	536 class SharedParamsStatisticsWrapper(SeriesArrayWrapper):
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	537 '''
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	538 Save mean, min/max, std of shared parameters place in an array.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	539
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	540 Here "shared" means "theano.shared", which means elements of the
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	541 array will have a .value to use for numpy.mean(), etc.
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	542
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	543 This inherits from SeriesArrayWrapper, which provides the append()
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	544 method.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	545 '''
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	546
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	547 def __init__(self, arrays_names, new_group_name, hdf5_file,
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	548 base_group='/', index_names=('epoch',), title="",
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	549 store_timestamp=True, store_cpuclock=True):
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	550 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	551 For other parameters, see Series.__init__
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	552
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	553 Parameters
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	554 ----------
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	555 array_names : array or tuple of str
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	556 Name of each array, in order of the array passed to append(). E.g.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	557 ('layer1_b', 'layer1_W', 'layer2_b', 'layer2_W')
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	558
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	559 new_group_name : str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	560 Name of a new HDF5 group which will be created under base_group to
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	561 store the new series.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	562
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	563 base_group : str
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	564 Path of the group under which to create the new group which will
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	565 store the series.
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	566
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	567 title : str
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	568 Here the title is attached to the new group, not a table.
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	569
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	570 store_timestamp : bool
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	571 Here timestamp and cpuclock are stored in each table
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	572
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	573 store_cpuclock : bool
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	574 Here timestamp and cpuclock are stored in each table
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	575 """
220 e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	576
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	577 # most other checks done when calling BasicStatisticsSeries
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	578 if type(new_group_name) != str:
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	579 raise TypeError("new_group_name must be a string")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	580 if new_group_name == "":
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	581 raise ValueError("new_group_name must not be empty")
e172ef73cdc5 Ajouté un paquet de type/value checks à SeriesTables, et finalisé les docstrings. Ajouté 3-4 tests. Légers refactorings ici et là sans conséquences externes. fsavard parents: 218 diff changeset	582
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	583 base_series_list = []
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	584
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	585 new_group = hdf5_file.createGroup(base_group, new_group_name, title=title)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	586
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	587 stats_functions = {'mean': lambda(x): numpy.mean(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	588 'min': lambda(x): numpy.min(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	589 'max': lambda(x): numpy.max(x.value),
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	590 'std': lambda(x): numpy.std(x.value)}
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	591
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	592 for name in arrays_names:
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	593 base_series_list.append(
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	594 BasicStatisticsSeries(
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	595 table_name=name,
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	596 hdf5_file=hdf5_file,
210 dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	597 index_names=index_names,
dc0d77c8a878 Commented table_series code, changed ParamsStatisticsArray to take shared params instead, create DummySeries to use when we don't want to save a named series savardf parents: 208 diff changeset	598 stats_functions=stats_functions,
224 0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	599 hdf5_group=new_group._v_pathname,
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	600 store_timestamp=store_timestamp,
0515a8901c6a Corrigé un bug avec store_timestamp/cpuclock, et tests pour éviter ce cas fsavard parents: 221 diff changeset	601 store_cpuclock=store_cpuclock))
208 acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	602
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	603 SeriesArrayWrapper.__init__(self, base_series_list)
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	604
acb942530923 Completely rewrote my series module, now based on HDF5 and PyTables (in a separate directory called 'tables_series' for retrocompatibility of running code). Minor (inconsequential) changes to stacked_dae. fsavard parents: diff changeset	605

Mercurial > ift6266

annotate utils/seriestables/series.py @ 441:1272dc84a30c