ift6266: datasets/ftfile.py annotate

annotate datasets/ftfile.py @ 217:de3aef84714a

merge, second try

author	Dumitru Erhan <dumitru.erhan@gmail.com>
date	Wed, 10 Mar 2010 17:08:50 -0500
parents	1faae5079522
children	966272e7f14b

rev	line source
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	1 from pylearn.io.filetensor import _read_header, _prod
178 938bd350dbf0 Make the datasets iterators return theano shared slices with the appropriate types. Arnaud Bergeron <abergeron@gmail.com> parents: 177 diff changeset	2 import numpy, theano
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	3 from dataset import DataSet
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	4 from dsetiter import DataIterator
178 938bd350dbf0 Make the datasets iterators return theano shared slices with the appropriate types. Arnaud Bergeron <abergeron@gmail.com> parents: 177 diff changeset	5 from itertools import izip, imap
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	6
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	7 class FTFile(object):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	8 def __init__(self, fname, scale=1, dtype=None):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	9 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	10 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	11 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	12 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	13 self.file = open(fname, 'rb')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	14 self.magic_t, self.elsize, _, self.dim, _ = _read_header(self.file, False)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	15 self.size = self.dim[0]
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	16 self.scale = scale
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	17 self.dtype = dtype
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	18
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	19 def skip(self, num):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	20 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	21 Skips `num` items in the file.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	22
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	23 If `num` is negative, skips size-num.
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	24
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	25 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	26 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	27 >>> f.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	28 58646
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	29 >>> f.elsize
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	30 4
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	31 >>> f.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	32 20
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	33 >>> f.skip(1000)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	34 >>> f.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	35 4020
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	36 >>> f.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	37 57646
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	38 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	39 >>> f.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	40 58646
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	41 >>> f.file.tell()
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	42 20
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	43 >>> f.skip(-1000)
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	44 >>> f.file.tell()
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	45 230604
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	46 >>> f.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	47 1000
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	48 """
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	49 if num < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	50 num += self.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	51 if num < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	52 raise ValueError('Skipping past the start of the file')
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	53 if num >= self.size:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	54 self.size = 0
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	55 else:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	56 self.size -= num
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	57 f_start = self.file.tell()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	58 self.file.seek(f_start + (self.elsize * _prod(self.dim[1:]) * num))
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	59
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	60 def read(self, num):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	61 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	62 Reads `num` elements from the file and return the result as a
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	63 numpy matrix. Last read is truncated.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	64
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	65 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	66 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_labels.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	67 >>> f.read(1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	68 array([6], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	69 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	70 array([7, 4, 7, 5, 6, 4, 8, 0, 9, 6], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	71 >>> f.skip(58630)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	72 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	73 array([9, 2, 4, 2, 8], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	74 >>> f.read(10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	75 array([], dtype=int32)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	76 >>> f = FTFile('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	77 >>> f.read(1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	78 array([[0, 0, 0, ..., 0, 0, 0]], dtype=uint8)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	79 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	80 if num > self.size:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	81 num = self.size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	82 self.dim[0] = num
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	83 self.size -= num
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	84 res = numpy.fromfile(self.file, dtype=self.magic_t, count=_prod(self.dim)).reshape(self.dim)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	85 if self.dtype is not None:
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	86 res = res.astype(self.dtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	87 if self.scale != 1:
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	88 res /= self.scale
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	89 return res
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	90
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	91 class FTSource(object):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	92 def __init__(self, file, skip=0, size=None, dtype=None, scale=1):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	93 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	94 Create a data source from a possible subset of a .ft file.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	95
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	96 Parameters:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	97 `file` (string) -- the filename
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	98 `skip` (int, optional) -- amount of examples to skip from
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	99 the start of the file. If
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	100 negative, skips filesize - skip.
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	101 `size` (int, optional) -- truncates number of examples
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	102 read (after skipping). If
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	103 negative truncates to
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	104 filesize - size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	105 (also after skipping).
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	106 `dtype` (dtype, optional) -- convert the data to this
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	107 dtype after reading.
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	108 `scale` (number, optional) -- scale (that is divide) the
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	109 data by this number (after
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	110 dtype conversion, if any).
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	111
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	112 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	113 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	114 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=1000)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	115 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=10)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	116 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=100, size=120)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	117 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	118 self.file = file
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	119 self.skip = skip
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	120 self.size = size
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	121 self.dtype = dtype
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	122 self.scale = scale
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	123
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	124 def open(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	125 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	126 Returns an FTFile that corresponds to this dataset.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	127
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	128 Tests:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	129 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft')
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	130 >>> f = s.open()
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	131 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	132 >>> len(s.open().read(2))
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	133 1
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	134 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=57646)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	135 >>> s.open().size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	136 1000
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	137 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', skip=57646, size=1)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	138 >>> s.open().size
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	139 1
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	140 >>> s = FTSource('/data/lisa/data/nist/by_class/digits/digits_test_data.ft', size=-10)
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	141 >>> s.open().size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	142 58636
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	143 """
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	144 f = FTFile(self.file, scale=self.scale, dtype=self.dtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	145 if self.skip != 0:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	146 f.skip(self.skip)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	147 if self.size is not None and self.size < f.size:
173 954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	148 if self.size < 0:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	149 f.size += self.size
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	150 else:
954185d6002a Take the validation set at the end of the training set files rather than at the beginning. Arnaud Bergeron <abergeron@gmail.com> parents: 163 diff changeset	151 f.size = self.size
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	152 return f
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	153
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	154 class FTData(object):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	155 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	156 This is a list of FTSources.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	157 """
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	158 def __init__(self, datafiles, labelfiles, skip=0, size=None,
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	159 inscale=1, indtype=None, outscale=1, outdtype=None):
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	160 self.inputs = [FTSource(f, skip, size, scale=inscale, dtype=indtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	161 for f in datafiles]
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	162 self.outputs = [FTSource(f, skip, size, scale=outscale, dtype=outdtype)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	163 for f in labelfiles]
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	164
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	165 def open_inputs(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	166 return [f.open() for f in self.inputs]
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	167
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	168 def open_outputs(self):
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	169 return [f.open() for f in self.outputs]
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	170
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	171
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	172 class FTDataSet(DataSet):
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	173 def __init__(self, train_data, train_lbl, test_data, test_lbl, valid_data=None, valid_lbl=None, indtype=None, outdtype=None, inscale=1, outscale=1):
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	174 r"""
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	175 Defines a DataSet from a bunch of files.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	176
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	177 Parameters:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	178 `train_data` -- list of train data files
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	179 `train_label` -- list of train label files (same length as `train_data`)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	180 `test_data`, `test_labels` -- same thing as train, but for
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	181 test. The number of files
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	182 can differ from train.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	183 `valid_data`, `valid_labels` -- same thing again for validation.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	184 (optional)
180 76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	185 `indtype`, `outdtype`, -- see FTSource.__init__()
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	186 `inscale`, `outscale` (optional)
76bc047df5ee Add dtype conversion and rescaling to the read path. Arnaud Bergeron <abergeron@gmail.com> parents: 178 diff changeset	187
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	188
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	189 If `valid_data` and `valid_labels` are not supplied then a sample
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	190 approximately equal in size to the test set is taken from the train
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	191 set.
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	192 """
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	193 if valid_data is None:
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	194 total_valid_size = sum(FTFile(td).size for td in test_data)
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	195 valid_size = total_valid_size/len(train_data)
214 1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	196 self._train = FTData(train_data, train_lbl, size=-valid_size,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	197 inscale=inscale, outscale=outscale, indtype=indtype,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	198 outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	199 self._valid = FTData(train_data, train_lbl, skip=-valid_size,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	200 inscale=inscale, outscale=outscale, indtype=indtype,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	201 outdtype=outdtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	202 else:
214 1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	203 self._train = FTData(train_data, train_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	204 outscale=outscale, indtype=indtype, outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	205 self._valid = FTData(valid_data, valid_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	206 outscale=outscale, indtype=indtype, outdtype=outdtype)
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	207 self._test = FTData(test_data, test_lbl,inscale=inscale,
1faae5079522 The in/outscale parameters were not passed to FTData Dumitru Erhan <dumitru.erhan@gmail.com> parents: 181 diff changeset	208 outscale=outscale, indtype=indtype, outdtype=outdtype)
163 4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	209
4b28d7382dbf Add inital implementation of datasets. Arnaud Bergeron <abergeron@gmail.com> parents: diff changeset	210 def _return_it(self, batchsize, bufsize, ftdata):
177 be714ac9bcbd Use izip(), not zip() to return a lazy iterator. (datasets) Arnaud Bergeron <abergeron@gmail.com> parents: 173 diff changeset	211 return izip(DataIterator(ftdata.open_inputs(), batchsize, bufsize),
181 f0f47b045cbf Remove a stray cast in the FTDataSet code and export the ocr dataset. Arnaud Bergeron <abergeron@gmail.com> parents: 180 diff changeset	212 DataIterator(ftdata.open_outputs(), batchsize, bufsize))

Mercurial > ift6266

annotate datasets/ftfile.py @ 217:de3aef84714a