ift6266: datasets/ftfile.py annotate

annotate datasets/ftfile.py @ 239:42005ec87747

Mergé (manuellement) les changements de Sylvain pour utiliser le code de dataset d'Arnaud, à cette différence près que je n'utilse pas les givens. J'ai probablement une approche différente pour limiter la taille du dataset dans mon débuggage, aussi.

author	fsavard
date	Mon, 15 Mar 2010 18:30:21 -0400
parents	1faae5079522
children	966272e7f14b

rev	line source
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	1 from pylearn.io.filetensor import _read_header, _prod
178 938bd350dbf0 Make the datasets iterators return theano shared slices with the appropriate types. Arnaud Bergeron <abergeron@gmail.com> parents: 177 diff changeset	2 import numpy, theano
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	3 from dataset import DataSet
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	4 from dsetiter import DataIterator
178 938bd350dbf0 Make the datasets iterators return theano shared slices with the appropriate types. Arnaud Bergeron <abergeron@gmail.com> parents: 177 diff changeset	5 from itertools import izip, imap
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	6
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	7 class FTFile(object):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	8 def __init__(self, fname, scale=1, dtype=None):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	9 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	10 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	11 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	12 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	13 self.file = open(fname, 'rb')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	14 self.magic_t, self.elsize, _, self.dim, _ = _read_header(self.file, False)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	15 self.size = self.dim[0]
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	16 self.scale = scale
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	17 self.dtype = dtype
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	18
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	19 def skip(self, num):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	20 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	21 Skips `num` items in the file.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	22
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	23 If `num` is negative, skips size-num.
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	24
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	25 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	26 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	27 >>> f.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	28 58646
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	29 >>> f.elsize
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	30 4
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	31 >>> f.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	32 20
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	33 >>> f.skip(1000)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	34 >>> f.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	35 4020
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	36 >>> f.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	37 57646
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	38 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	39 >>> f.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	40 58646
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	41 >>> f.file.tell()
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	42 20
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	43 >>> f.skip(-1000)
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	44 >>> f.file.tell()
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	45 230604
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	46 >>> f.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	47 1000
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	48 """
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	49 if num < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	50 num += self.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	51 if num < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	52 raise ValueError('Skipping past the start of the file')
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	53 if num >= self.size:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	54 self.size = 0
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	55 else:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	56 self.size -= num
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	57 f_start = self.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	58 self.file.seek(f_start + (self.elsize * _prod(self.dim[1:]) * num))
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	59
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	60 def read(self, num):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	61 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	62 Reads `num` elements from the file and return the result as a
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	63 numpy matrix. Last read is truncated.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	64
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	65 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	66 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	67 >>> f.read(1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	68 array([6], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	69 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	70 array([7, 4, 7, 5, 6, 4, 8, 0, 9, 6], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	71 >>> f.skip(58630)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	72 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	73 array([9, 2, 4, 2, 8], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	74 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	75 array([], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	76 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	77 >>> f.read(1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	78 array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	79 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	80 if num > self.size:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	81 num = self.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	82 self.dim[0] = num
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	83 self.size -= num
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	84 res = numpy.fromfile(self.file, dtype=self.magic_t, count=_prod(self.dim)).reshape(self.dim)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	85 if self.dtype is not None:
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	86 res = res.astype(self.dtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	87 if self.scale != 1:
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	88 res /= self.scale
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	89 return res
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	90
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	91 class FTSource(object):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	92 def __init__(self, file, skip=0, size=None, dtype=None, scale=1):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	93 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	94 Create a data source from a possible subset of a .ft file.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	95
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	96 Parameters:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	97 `file` (string) -- the filename
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	98 `skip` (int, optional) -- amount of examples to skip from
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	99 the start of the file. If
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	100 negative, skips filesize - skip.
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	101 `size` (int, optional) -- truncates number of examples
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	102 read (after skipping). If
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	103 negative truncates to
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	104 filesize - size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	105 (also after skipping).
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	106 `dtype` (dtype, optional) -- convert the data to this
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	107 dtype after reading.
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	108 `scale` (number, optional) -- scale (that is divide) the
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	109 data by this number (after
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	110 dtype conversion, if any).
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	111
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	112 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	113 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	114 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=1000)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	115 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	116 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=100, size=120)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	117 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	118 self.file = file
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	119 self.skip = skip
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	120 self.size = size
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	121 self.dtype = dtype
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	122 self.scale = scale
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	123
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	124 def open(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	125 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	126 Returns an FTFile that corresponds to this dataset.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	127
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	128 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	129 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	130 >>> f = s.open()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	131 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	132 >>> len(s.open().read(2))
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	133 1
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	134 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=57646)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	135 >>> s.open().size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	136 1000
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	137 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=57646, size=1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	138 >>> s.open().size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	139 1
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	140 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=-10)
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	141 >>> s.open().size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	142 58636
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	143 """
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	144 f = FTFile(self.file, scale=self.scale, dtype=self.dtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	145 if self.skip != 0:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	146 f.skip(self.skip)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	147 if self.size is not None and self.size < f.size:
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	148 if self.size < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	149 f.size += self.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	150 else:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	151 f.size = self.size
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	152 return f
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	153
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	154 class FTData(object):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	155 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	156 This is a list of FTSources.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	157 """
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	158 def __init__(self, datafiles, labelfiles, skip=0, size=None,
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	159 inscale=1, indtype=None, outscale=1, outdtype=None):
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	160 self.inputs = [FTSource(f, skip, size, scale=inscale, dtype=indtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	161 for f in datafiles]
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	162 self.outputs = [FTSource(f, skip, size, scale=outscale, dtype=outdtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	163 for f in labelfiles]
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	164
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	165 def open_inputs(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	166 return [f.open() for f in self.inputs]
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	167
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	168 def open_outputs(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	169 return [f.open() for f in self.outputs]
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	170
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	171
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	172 class FTDataSet(DataSet):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	173 def __init__(self, train_data, train_lbl, test_data, test_lbl, valid_data=None, valid_lbl=None, indtype=None, outdtype=None, inscale=1, outscale=1):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	174 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	175 Defines a DataSet from a bunch of files.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	176
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	177 Parameters:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	178 `train_data` -- list of train data files
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	179 `train_label` -- list of train label files (same length as `train_data`)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	180 `test_data`, `test_labels` -- same thing as train, but for
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	181 test. The number of files
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	182 can differ from train.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	183 `valid_data`, `valid_labels` -- same thing again for validation.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	184 (optional)
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	185 `indtype`, `outdtype`, -- see FTSource.__init__()
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	186 `inscale`, `outscale` (optional)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	187
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	188
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	189 If `valid_data` and `valid_labels` are not supplied then a sample
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	190 approximately equal in size to the test set is taken from the train
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	191 set.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	192 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	193 if valid_data is None:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	194 total_valid_size = sum(FTFile(td).size for td in test_data)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	195 valid_size = total_valid_size/len(train_data)
214 1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	196 self._train = FTData(train_data, train_lbl, size=-valid_size,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	197 inscale=inscale, outscale=outscale, indtype=indtype,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	198 outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	199 self._valid = FTData(train_data, train_lbl, skip=-valid_size,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	200 inscale=inscale, outscale=outscale, indtype=indtype,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	201 outdtype=outdtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	202 else:
214 1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	203 self._train = FTData(train_data, train_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	204 outscale=outscale, indtype=indtype, outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	205 self._valid = FTData(valid_data, valid_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	206 outscale=outscale, indtype=indtype, outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	207 self._test = FTData(test_data, test_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	208 outscale=outscale, indtype=indtype, outdtype=outdtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	209
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	210 def _return_it(self, batchsize, bufsize, ftdata):
177 be714ac9bcbd Use izip(), not zip() to return a lazy iterator. (datasets) Arnaud Bergeron <abergeron@gmail.com> parents: 173 diff changeset	211 return izip(DataIterator(ftdata.open_inputs(), batchsize, bufsize),
181 f0f47b045cbf Remove a stray cast in the FTDataSet code and export the ocr dataset. Arnaud Bergeron <abergeron@gmail.com> parents: 180 diff changeset	212 DataIterator(ftdata.open_outputs(), batchsize, bufsize))

Mercurial > ift6266

annotate datasets/ftfile.py @ 239:42005ec87747