pylearn: dataset.py annotate

annotate dataset.py @ 40:88fd1cce08b9

replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets

author	bengioy@esprit.iro.umontreal.ca
date	Fri, 25 Apr 2008 10:41:19 -0400
parents	c682c6e9bf93
children	283e95c15b47

rev	line source
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	1
12 ff4e551490f1 Added LookupList type in lookup_list.py and used it to keep order bengioy@esprit.iro.umontreal.ca parents: 11 diff changeset	2 from lookup_list import LookupList
ff4e551490f1 Added LookupList type in lookup_list.py and used it to keep order bengioy@esprit.iro.umontreal.ca parents: 11 diff changeset	3 Example = LookupList
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	4 from misc import *
26 672fe4b23032 Fixed dataset errors so that _test_dataset.py works again. bengioy@grenat.iro.umontreal.ca parents: 23 diff changeset	5 import copy
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	6
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	7 class AbstractFunction (Exception): """Derived class must override this function"""
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	8 class NotImplementedYet (NotImplementedError): """Work in progress, this should eventually be implemented"""
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	9 class UnboundedDataSet (Exception): """Trying to obtain length of unbounded dataset (a stream)"""
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	10
1 2cd82666b9a7 Added statscollector and started writing dataset and learner. bengioy@esprit.iro.umontreal.ca parents: 0 diff changeset	11 class DataSet(object):
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	12 """A virtual base class for datasets.
813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	13
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	14 A DataSet can be seen as a generalization of a matrix, meant to be used in conjunction
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	15 with learning algorithms (for training and testing them): rows/records are called examples, and
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	16 columns/attributes are called fields. The field value for a particular example can be an arbitrary
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	17 python object, which depends on the particular dataset.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	18
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	19 We call a DataSet a 'stream' when its length is unbounded (otherwise its __len__ method
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	20 should raise an UnboundedDataSet exception).
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	21
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	22 A DataSet is a generator of iterators; these iterators can run through the
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	23 examples or the fields in a variety of ways. A DataSet need not necessarily have a finite
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	24 or known length, so this class can be used to interface to a 'stream' which
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	25 feeds on-line learning (however, as noted below, some operations are not
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	26 feasible or not recommanded on streams).
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	27
813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	28 To iterate over examples, there are several possibilities:
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	29 * for example in dataset([field1, field2,field3, ...]):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	30 * for val1,val2,val3 in dataset([field1, field2,field3]):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	31 * for minibatch in dataset.minibatches([field1, field2, ...],minibatch_size=N):
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	32 * for mini1,mini2,mini3 in dataset.minibatches([field1, field2, ...],minibatch_size=N):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	33 * for example in dataset:
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	34 Each of these is documented below. All of these iterators are expected
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	35 to provide, in addition to the usual 'next()' method, a 'next_index()' method
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	36 which returns a non-negative integer pointing to the position of the next
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	37 example that will be returned by 'next()' (or of the first example in the
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	38 next minibatch returned). This is important because these iterators
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	39 can wrap around the dataset in order to do multiple passes through it,
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	40 in possibly unregular ways if the minibatch size is not a divisor of the
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	41 dataset length.
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	42
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	43 To iterate over fields, one can do
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	44 * for fields in dataset.fields()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	45 * for fields in dataset(field1,field2,...).fields() to select a subset of fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	46 * for fields in dataset.fields(field1,field2,...) to select a subset of fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	47 and each of these fields is iterable over the examples:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	48 * for field_examples in dataset.fields():
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	49 for example_value in field_examples:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	50 ...
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	51 but when the dataset is a stream (unbounded length), it is not recommanded to do
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	52 such things because the underlying dataset may refuse to access the different fields in
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	53 an unsynchronized ways. Hence the fields() method is illegal for streams, by default.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	54 The result of fields() is a DataSetFields object, which iterates over fields,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	55 and whose elements are iterable over examples. A DataSetFields object can
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	56 be turned back into a DataSet with its examples() method:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	57 dataset2 = dataset1.fields().examples()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	58 and dataset2 should behave exactly like dataset1 (in fact by default dataset2==dataset1).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	59
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	60 Note: Fields are not mutually exclusive, i.e. two fields can overlap in their actual content.
813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	61
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	62 Note: The content of a field can be of any type. Field values can also be 'missing'
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	63 (e.g. to handle semi-supervised learning), and in the case of numeric (numpy array)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	64 fields (i.e. an ArrayFieldsDataSet), NaN plays the role of a missing value.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	65
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	66 Dataset elements can be indexed and sub-datasets (with a subset
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	67 of examples) can be extracted. These operations are not supported
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	68 by default in the case of streams.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	69
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	70 * dataset[:n] returns a dataset with the n first examples.
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	71
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	72 * dataset[i1:i2:s] returns a dataset with the examples i1,i1+s,...i2-s.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	73
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	74 * dataset[i] returns an Example.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	75
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	76 * dataset[[i1,i2,...in]] returns a dataset with examples i1,i2,...in.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	77
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	78 Datasets can be concatenated either vertically (increasing the length) or
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	79 horizontally (augmenting the set of fields), if they are compatible, using
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	80 the following operations (with the same basic semantics as numpy.hstack
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	81 and numpy.vstack):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	82
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	83 * dataset1 \| dataset2 \| dataset3 == dataset.hstack([dataset1,dataset2,dataset3])
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	84
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	85 creates a new dataset whose list of fields is the concatenation of the list of
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	86 fields of the argument datasets. This only works if they all have the same length.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	87
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	88 * dataset1 & dataset2 & dataset3 == dataset.vstack([dataset1,dataset2,dataset3])
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	89
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	90 creates a new dataset that concatenates the examples from the argument datasets
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	91 (and whose length is the sum of the length of the argument datasets). This only
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	92 works if they all have the same fields.
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	93
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	94 According to the same logic, and viewing a DataSetFields object associated to
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	95 a DataSet as a kind of transpose of it, fields1 + fields2 concatenates fields of
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	96 a DataSetFields fields1 and fields2, and fields1 \| fields2 concatenates their
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	97 examples.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	98
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	99 A DataSet sub-class should always redefine the following methods:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	100 * __len__ if it is not a stream
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	101 * fieldNames
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	102 * minibatches_nowrap (called by DataSet.minibatches())
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	103 * valuesHStack
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	104 * valuesVStack
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	105 For efficiency of implementation, a sub-class might also want to redefine
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	106 * hasFields
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	107 * __getitem__ may not be feasible with some streams
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	108 * __iter__
2 3fddb1c8f955 Rewrote DataSet interface and created FiniteDataSet interface. bengioy@bengiomac.local parents: 1 diff changeset	109 """
1 2cd82666b9a7 Added statscollector and started writing dataset and learner. bengioy@esprit.iro.umontreal.ca parents: 0 diff changeset	110
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	111 def __init__(self):
be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	112 pass
be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	113
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	114 class MinibatchToSingleExampleIterator(object):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	115 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	116 Converts the result of minibatch iterator with minibatch_size==1 into
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	117 single-example values in the result. Therefore the result of
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	118 iterating on the dataset itself gives a sequence of single examples
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	119 (whereas the result of iterating over minibatches gives in each
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	120 Example field an iterable object over the individual examples in
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	121 the minibatch).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	122 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	123 def __init__(self, minibatch_iterator):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	124 self.minibatch_iterator = minibatch_iterator
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	125 def __iter__(self): #makes for loop work
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	126 return self
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	127 def next(self):
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	128 size1_minibatch = self.minibatch_iterator.next()
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	129 return Example(size1_minibatch.keys,[value[0] for value in size1_minibatch.values()])
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	130
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	131 def next_index(self):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	132 return self.minibatch_iterator.next_index()
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	133
3 378b68d5c4ad Added first (untested) version of ArrayDataSet bengioy@bengiomac.local parents: 2 diff changeset	134 def __iter__(self):
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	135 """Supports the syntax "for i in dataset: ..."
1 2cd82666b9a7 Added statscollector and started writing dataset and learner. bengioy@esprit.iro.umontreal.ca parents: 0 diff changeset	136
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	137 Using this syntax, "i" will be an Example instance (or equivalent) with
813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	138 all the fields of DataSet self. Every field of "i" will give access to
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	139 a field of a single example. Fields should be accessible via
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	140 i["fielname"] or i[3] (in the order defined by the elements of the
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	141 Example returned by this iterator), but the derived class is free
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	142 to accept any type of identifier, and add extra functionality to the iterator.
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	143
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	144 The default implementation calls the minibatches iterator and extracts the first example of each field.
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	145 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	146 return DataSet.MinibatchToSingleExampleIterator(self.minibatches(None, minibatch_size = 1))
2 3fddb1c8f955 Rewrote DataSet interface and created FiniteDataSet interface. bengioy@bengiomac.local parents: 1 diff changeset	147
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	148
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	149 class MinibatchWrapAroundIterator(object):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	150 """
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	151 An iterator for minibatches that handles the case where we need to wrap around the
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	152 dataset because n_batches*minibatch_size > len(dataset). It is constructed from
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	153 a dataset that provides a minibatch iterator that does not need to handle that problem.
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	154 This class is a utility for dataset subclass writers, so that they do not have to handle
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	155 this issue multiple times, nor check that fieldnames are valid, nor handle the
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	156 empty fieldnames (meaning 'use all the fields').
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	157 """
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	158 def __init__(self,dataset,fieldnames,minibatch_size,n_batches,offset):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	159 self.dataset=dataset
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	160 self.fieldnames=fieldnames
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	161 self.minibatch_size=minibatch_size
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	162 self.n_batches=n_batches
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	163 self.n_batches_done=0
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	164 self.next_row=offset
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	165 self.L=len(dataset)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	166 assert offset+minibatch_size<=self.L
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	167 ds_nbatches = (self.L-offset)/minibatch_size
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	168 if n_batches is not None:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	169 ds_nbatches = max(n_batches,ds_nbatches)
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	170 if fieldnames:
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	171 assert dataset.hasFields(*fieldnames)
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	172 else:
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	173 fieldnames=dataset.fieldNames()
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	174 self.iterator = dataset.minibatches_nowrap(fieldnames,minibatch_size,ds_nbatches,offset)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	175
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	176 def __iter__(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	177 return self
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	178
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	179 def next_index(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	180 return self.next_row
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	181
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	182 def next(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	183 if self.n_batches and self.n_batches_done==self.n_batches:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	184 raise StopIteration
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	185 upper = self.next_row+minibatch_size
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	186 if upper <=self.L:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	187 minibatch = self.minibatch_iterator.next()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	188 else:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	189 if not self.n_batches:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	190 raise StopIteration
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	191 # we must concatenate (vstack) the bottom and top parts of our minibatch
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	192 # first get the beginning of our minibatch (top of dataset)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	193 first_part = self.dataset.minibatches_nowrap(fieldnames,self.L-self.next_row,1,self.next_row).next()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	194 second_part = self.dataset.minibatches_nowrap(fieldnames,upper-self.L,1,0).next()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	195 minibatch = Example(self.fieldnames,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	196 [self.dataset.valuesVStack(name,[first_part[name],second_part[name]])
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	197 for name in self.fieldnames])
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	198 self.next_row=upper
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	199 self.n_batches_done+=1
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	200 if upper >= L:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	201 self.next_row -= L
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	202 return minibatch
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	203
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	204
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	205 minibatches_fieldnames = None
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	206 minibatches_minibatch_size = 1
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	207 minibatches_n_batches = None
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	208 def minibatches(self,
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	209 fieldnames = minibatches_fieldnames,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	210 minibatch_size = minibatches_minibatch_size,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	211 n_batches = minibatches_n_batches,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	212 offset = 0):
6 d5738b79089a Removed MinibatchIterator and instead made minibatch_size a field of all DataSets, bengioy@bengiomac.local parents: 5 diff changeset	213 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	214 Return an iterator that supports three forms of syntax:
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	215
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	216 for i in dataset.minibatches(None,**kwargs): ...
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	217
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	218 for i in dataset.minibatches([f1, f2, f3],**kwargs): ...
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	219
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	220 for i1, i2, i3 in dataset.minibatches([f1, f2, f3],**kwargs): ...
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	221
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	222 Using the first two syntaxes, "i" will be an indexable object, such as a list,
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	223 tuple, or Example instance. In both cases, i[k] is a list-like container
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	224 of a batch of current examples. In the second case, i[0] is
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	225 list-like container of the f1 field of a batch current examples, i[1] is
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	226 a list-like container of the f2 field, etc.
2 3fddb1c8f955 Rewrote DataSet interface and created FiniteDataSet interface. bengioy@bengiomac.local parents: 1 diff changeset	227
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	228 Using the first syntax, all the fields will be returned in "i".
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	229 Using the third syntax, i1, i2, i3 will be list-like containers of the
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	230 f1, f2, and f3 fields of a batch of examples on each loop iteration.
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	231
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	232 The minibatches iterator is expected to return upon each call to next()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	233 a DataSetFields object, which is a LookupList (indexed by the field names) whose
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	234 elements are iterable over the minibatch examples, and which keeps a pointer to
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	235 a sub-dataset that can be used to iterate over the individual examples
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	236 in the minibatch. Hence a minibatch can be converted back to a regular
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	237 dataset or its fields can be looked at individually (and possibly iterated over).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	238
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	239 PARAMETERS
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	240 - fieldnames (list of any type, default None):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	241 The loop variables i1, i2, i3 (in the example above) should contain the
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	242 f1, f2, and f3 fields of the current batch of examples. If None, the
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	243 derived class can choose a default, e.g. all fields.
16 813723310d75 commenting bergstrj@iro.umontreal.ca parents: 15 11 diff changeset	244
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	245 - minibatch_size (integer, default 1)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	246 On every iteration, the variables i1, i2, i3 will have
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	247 exactly minibatch_size elements. e.g. len(i1) == minibatch_size
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	248
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	249 - n_batches (integer, default None)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	250 The iterator will loop exactly this many times, and then stop. If None,
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	251 the derived class can choose a default. If (-1), then the returned
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	252 iterator should support looping indefinitely.
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	253
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	254 - offset (integer, default 0)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	255 The iterator will start at example 'offset' in the dataset, rather than the default.
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	256
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	257 Note: A list-like container is something like a tuple, list, numpy.ndarray or
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	258 any other object that supports integer indexing and slicing.
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	259
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	260 """
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	261 return MinibatchWrapAroundIterator(self,fieldnames,minibatch_size,n_batches,offset)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	262
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	263 def minibatches_nowrap(self,fieldnames,minibatch_size,n_batches,offset):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	264 """
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	265 This is the minibatches iterator generator that sub-classes must define.
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	266 It does not need to worry about wrapping around multiple times across the dataset,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	267 as this is handled by MinibatchWrapAroundIterator when DataSet.minibatches() is called.
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	268 The next() method of the returned iterator does not even need to worry about
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	269 the termination condition (as StopIteration will be raised by DataSet.minibatches
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	270 before an improper call to minibatches_nowrap's next() is made).
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	271 That next() method can assert that its next row will always be within [0,len(dataset)).
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	272 The iterator returned by minibatches_nowrap does not need to implement
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	273 a next_index() method either, as this will be provided by MinibatchWrapAroundIterator.
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	274 """
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	275 raise AbstractFunction()
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	276
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	277 def __len__(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	278 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	279 len(dataset) returns the number of examples in the dataset.
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	280 By default, a DataSet is a 'stream', i.e. it has an unbounded length (raises UnboundedDataSet).
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	281 Sub-classes which implement finite-length datasets should redefine this method.
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	282 Some methods only make sense for finite-length datasets.
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	283 """
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	284 raise UnboundedDataSet()
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	285
26 672fe4b23032 Fixed dataset errors so that _test_dataset.py works again. bengioy@grenat.iro.umontreal.ca parents: 23 diff changeset	286 def hasFields(self,*fieldnames):
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	287 """
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	288 Return true if the given field name (or field names, if multiple arguments are
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	289 given) is recognized by the DataSet (i.e. can be used as a field name in one
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	290 of the iterators).
29 46c5c90019c2 Changed apply_function so that it propagates methods of the source. bengioy@grenat.iro.umontreal.ca parents: 28 diff changeset	291
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	292 The default implementation may be inefficient (O(# fields in dataset)), as it calls the fieldNames()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	293 method. Many datasets may store their field names in a dictionary, which would allow more efficiency.
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	294 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	295 return len(unique_elements_list_intersection(fieldnames,self.fieldNames()))>0
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	296
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	297 def fieldNames(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	298 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	299 Return the list of field names that are supported by the iterators,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	300 and for which hasFields(fieldname) would return True.
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	301 """
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	302 raise AbstractFunction()
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	303
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	304 def __call__(self,*fieldnames):
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	305 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	306 Return a dataset that sees only the fields whose name are specified.
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	307 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	308 assert self.hasFields(fieldnames)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	309 return self.fields(fieldnames).examples()
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	310
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	311 def fields(self,*fieldnames):
29 46c5c90019c2 Changed apply_function so that it propagates methods of the source. bengioy@grenat.iro.umontreal.ca parents: 28 diff changeset	312 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	313 Return a DataSetFields object associated with this dataset.
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	314 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	315 return DataSetFields(self,fieldnames)
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	316
2 3fddb1c8f955 Rewrote DataSet interface and created FiniteDataSet interface. bengioy@bengiomac.local parents: 1 diff changeset	317 def __getitem__(self,i):
28 541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	318 """
541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	319 dataset[i] returns the (i+1)-th example of the dataset.
541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	320 dataset[i:j] returns the subdataset with examples i,i+1,...,j-1.
541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	321 dataset[i:j:s] returns the subdataset with examples i,i+2,i+4...,j-2.
541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	322 dataset[[i1,i2,..,in]] returns the subdataset with examples i1,i2,...,in.
1 2cd82666b9a7 Added statscollector and started writing dataset and learner. bengioy@esprit.iro.umontreal.ca parents: 0 diff changeset	323
39 c682c6e9bf93 Minor edits bengioy@esprit.iro.umontreal.ca parents: 38 diff changeset	324 Note that some stream datasets may be unable to implement random access, i.e.
c682c6e9bf93 Minor edits bengioy@esprit.iro.umontreal.ca parents: 38 diff changeset	325 arbitrary slicing/indexing
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	326 because they can only iterate through examples one or a minibatch at a time
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	327 and do not actually store or keep past (or future) examples.
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	328
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	329 The default implementation of getitem uses the minibatches iterator
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	330 to obtain one example, one slice, or a list of examples. It may not
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	331 always be the most efficient way to obtain the result, especially if
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	332 the data are actually stored in a memory array.
28 541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	333 """
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	334 if type(i) is int:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	335 return DataSet.MinibatchToSingleExampleIterator(
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	336 self.minibatches(minibatch_size=1,n_batches=1,offset=i)).next()
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	337 if type(i) is slice:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	338 if not i.start: i.start=0
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	339 if not i.step: i.step=1
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	340 if i.step is 1:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	341 return self.minibatches(minibatch_size=i.stop-i.start,n_batches=1,offset=i.start).next().examples()
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	342 rows = range(i.start,i.stop,i.step)
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	343 else:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	344 assert type(i) is list
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	345 rows = i
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	346 fields_values = zip(*[self[row] for row in rows])
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	347 return MinibatchDataSet(
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	348 Example(self.fieldNames(),[ self.valuesVStack(fieldname,field_values)
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	349 for fieldname,field_values
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	350 in zip(self.fieldNames(),fields_values)]))
22 b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, bengioy@esprit.iro.umontreal.ca parents: 20 diff changeset	351
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	352 def valuesHStack(self,fieldnames,fieldvalues):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	353 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	354 Return a value that corresponds to concatenating (horizontally) several field values.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	355 This can be useful to merge some fields. The implementation of this operation is likely
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	356 to involve a copy of the original values. When the values are numpy arrays, the
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	357 result should be numpy.hstack(values). If it makes sense, this operation should
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	358 work as well when each value corresponds to multiple examples in a minibatch
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	359 e.g. if each value is a Ni-vector and a minibatch of length L is a LxNi matrix,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	360 then the result should be a Lx(N1+N2+..) matrix equal to numpy.hstack(values).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	361 The default is to use numpy.hstack for numpy.ndarray values, and a list
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	362 pointing to the original values for other data types.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	363 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	364 all_numpy=True
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	365 for value in fieldvalues:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	366 if not type(value) is numpy.ndarray:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	367 all_numpy=False
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	368 if all_numpy:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	369 return numpy.hstack(fieldvalues)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	370 # the default implementation of horizontal stacking is to put values in a list
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	371 return fieldvalues
26 672fe4b23032 Fixed dataset errors so that _test_dataset.py works again. bengioy@grenat.iro.umontreal.ca parents: 23 diff changeset	372
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again. bengioy@grenat.iro.umontreal.ca parents: 23 diff changeset	373
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	374 def valuesVStack(self,fieldname,values):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	375 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	376 Return a value that corresponds to concatenating (vertically) several values of the
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	377 same field. This can be important to build a minibatch out of individual examples. This
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	378 is likely to involve a copy of the original values. When the values are numpy arrays, the
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	379 result should be numpy.vstack(values).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	380 The default is to use numpy.vstack for numpy.ndarray values, and a list
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	381 pointing to the original values for other data types.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	382 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	383 all_numpy=True
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	384 for value in values:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	385 if not type(value) is numpy.ndarray:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	386 all_numpy=False
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	387 if all_numpy:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	388 return numpy.vstack(values)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	389 # the default implementation of vertical stacking is to put values in a list
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	390 return values
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	391
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	392 def __or__(self,other):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	393 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	394 dataset1 \| dataset2 returns a dataset whose list of fields is the concatenation of the list of
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	395 fields of the argument datasets. This only works if they all have the same length.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	396 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	397 return HStackedDataSet(self,other)
3 378b68d5c4ad Added first (untested) version of ArrayDataSet bengioy@bengiomac.local parents: 2 diff changeset	398
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	399 def __and__(self,other):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	400 """
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	401 dataset1 & dataset2 is a dataset that concatenates the examples from the argument datasets
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	402 (and whose length is the sum of the length of the argument datasets). This only
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	403 works if they all have the same fields.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	404 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	405 return VStackedDataSet(self,other)
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	406
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	407 def hstack(datasets):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	408 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	409 hstack(dataset1,dataset2,...) returns dataset1 \| datataset2 \| ...
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	410 which is a dataset whose fields list is the concatenation of the fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	411 of the individual datasets.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	412 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	413 assert len(datasets)>0
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	414 if len(datasets)==1:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	415 return datasets[0]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	416 return HStackedDataSet(datasets)
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	417
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	418 def vstack(datasets):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	419 """
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	420 vstack(dataset1,dataset2,...) returns dataset1 & datataset2 & ...
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	421 which is a dataset which iterates first over the examples of dataset1, then
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	422 over those of dataset2, etc.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	423 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	424 assert len(datasets)>0
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	425 if len(datasets)==1:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	426 return datasets[0]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	427 return VStackedDataSet(datasets)
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	428
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	429
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	430 class DataSetFields(LookupList):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	431 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	432 Although a DataSet iterates over examples (like rows of a matrix), an associated
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	433 DataSetFields iterates over fields (like columns of a matrix), and can be understood
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	434 as a transpose of the associated dataset.
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	435
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	436 To iterate over fields, one can do
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	437 * for fields in dataset.fields()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	438 * for fields in dataset(field1,field2,...).fields() to select a subset of fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	439 * for fields in dataset.fields(field1,field2,...) to select a subset of fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	440 and each of these fields is iterable over the examples:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	441 * for field_examples in dataset.fields():
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	442 for example_value in field_examples:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	443 ...
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	444 but when the dataset is a stream (unbounded length), it is not recommanded to do
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	445 such things because the underlying dataset may refuse to access the different fields in
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	446 an unsynchronized ways. Hence the fields() method is illegal for streams, by default.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	447 The result of fields() is a DataSetFields object, which iterates over fields,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	448 and whose elements are iterable over examples. A DataSetFields object can
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	449 be turned back into a DataSet with its examples() method:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	450 dataset2 = dataset1.fields().examples()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	451 and dataset2 should behave exactly like dataset1 (in fact by default dataset2==dataset1).
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	452
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	453 DataSetFields can be concatenated vertically or horizontally. To be consistent with
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	454 the syntax used for DataSets, the \| concatenates the fields and the & concatenates
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	455 the examples.
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	456 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	457 def __init__(self,dataset,*fieldnames):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	458 self.dataset=dataset
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	459 if not fieldnames:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	460 fieldnames=dataset.fieldNames()
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	461 assert dataset.hasFields(*fieldnames)
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	462 LookupList.__init__(self,dataset.fieldNames(),
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	463 dataset.minibatches(fieldnames if len(fieldnames)>0 else self.fieldNames(),
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	464 minibatch_size=len(dataset)).next()
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	465 def examples(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	466 return self.dataset
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	467
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	468 def __or__(self,other):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	469 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	470 fields1 \| fields2 is a DataSetFields that whose list of examples is the concatenation
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	471 of the list of examples of DataSetFields fields1 and fields2.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	472 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	473 return (self.examples() + other.examples()).fields()
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	474
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	475 def __and__(self,other):
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	476 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	477 fields1 + fields2 is a DataSetFields that whose list of fields is the concatenation
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	478 of the fields of DataSetFields fields1 and fields2.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	479 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	480 return (self.examples() \| other.examples()).fields()
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	481
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	482
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	483 class MinibatchDataSet(DataSet):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	484 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	485 Turn a LookupList of same-length fields into an example-iterable dataset.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	486 Each element of the lookup-list should be an iterable and sliceable, all of the same length.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	487 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	488 def __init__(self,fields_lookuplist,values_vstack=DataSet().valuesVStack,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	489 values_hstack=DataSet().valuesHStack):
17 759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests bergstrj@iro.umontreal.ca parents: 16 12 diff changeset	490 """
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	491 The user can (and generally should) also provide values_vstack(fieldname,fieldvalues)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	492 and a values_hstack(fieldnames,fieldvalues) functions behaving with the same
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	493 semantics as the DataSet methods of the same name (but without the self argument).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	494 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	495 self.fields=fields_lookuplist
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	496 assert len(fields_lookuplist)>0
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	497 self.length=len(fields_lookuplist[0])
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	498 for field in fields_lookuplist[1:]:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	499 assert self.length==len(field)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	500 self.values_vstack=values_vstack
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	501 self.values_hstack=values_hstack
3 378b68d5c4ad Added first (untested) version of ArrayDataSet bengioy@bengiomac.local parents: 2 diff changeset	502
378b68d5c4ad Added first (untested) version of ArrayDataSet bengioy@bengiomac.local parents: 2 diff changeset	503 def __len__(self):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	504 return self.length
28 541a273bc89f Removed __array__ method from dataset, whose bengioy@grenat.iro.umontreal.ca parents: 26 diff changeset	505
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	506 def __getitem__(self,i):
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	507 return DataSetFields(MinibatchDataSet(
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	508 Example(self.fields.keys(),[field[i] for field in self.fields])),self.fields)
11 be128b9127c8 Debugged (to the extent of my tests) the new version of dataset bengioy@esprit.iro.umontreal.ca parents: 9 diff changeset	509
29 46c5c90019c2 Changed apply_function so that it propagates methods of the source. bengioy@grenat.iro.umontreal.ca parents: 28 diff changeset	510 def fieldNames(self):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	511 return self.fields.keys()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	512
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	513 def hasFields(self,*fieldnames):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	514 for fieldname in fieldnames:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	515 if fieldname not in self.fields:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	516 return False
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	517 return True
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	518
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	519 def minibatches_nowrap(self,fieldnames,minibatch_size,n_batches,offset):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	520 class Iterator(object):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	521 def __init__(self,ds):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	522 self.ds=ds
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	523 self.next_example=offset
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	524 assert minibatch_size > 0
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	525 if offset+minibatch_size > ds.length
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	526 raise NotImplementedError()
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	527 def __iter__(self):
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	528 return self
266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	529 def next(self):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	530 upper = next_example+minibatch_size
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	531 assert upper<=self.ds.length
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	532 minibatch = Example(self.ds.fields.keys(),
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	533 [field[next_example:upper]
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	534 for field in self.ds.fields])
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	535 self.next_example+=minibatch_size
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	536 return DataSetFields(MinibatchDataSet(minibatch),fieldnames)
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	537
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	538 return Iterator(self)
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	539
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	540 def valuesVStack(self,fieldname,fieldvalues):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	541 return self.values_vstack(fieldname,fieldvalues)
20 266c68cb6136 Minor editions, plus adding untested ApplyFunctionDataset for GradientLearner in the works. bengioy@bengiomac.local parents: 19 diff changeset	542
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	543 def valuesHStack(self,fieldnames,fieldvalues):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	544 return self.values_hstack(fieldnames,fieldvalues)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	545
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	546 class HStackedDataSet(DataSet):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	547 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	548 A DataSet that wraps several datasets and shows a view that includes all their fields,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	549 i.e. whose list of fields is the concatenation of their lists of fields.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	550
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	551 If a field name is found in more than one of the datasets, then either an error is
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	552 raised or the fields are renamed (either by prefixing the __name__ attribute
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	553 of the dataset + ".", if it exists, or by suffixing the dataset index in the argument list).
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	554
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	555 TODO: automatically detect a chain of stacked datasets due to A \| B \| C \| D ...
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	556 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	557 def __init__(self,datasets,accept_nonunique_names=False):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	558 DataSet.__init__(self)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	559 self.datasets=datasets
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	560 self.accept_nonunique_names=accept_nonunique_names
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	561 self.fieldname2dataset={}
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	562
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	563 def rename_field(fieldname,dataset,i):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	564 if hasattr(dataset,"__name__"):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	565 return dataset.__name__ + "." + fieldname
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	566 return fieldname+"."+str(i)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	567
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	568 # make sure all datasets have the same length and unique field names
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	569 self.length=None
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	570 names_to_change=[]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	571 for i in xrange(len(datasets)):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	572 dataset = datasets[i]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	573 length=len(dataset)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	574 if self.length:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	575 assert self.length==length
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	576 else:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	577 self.length=length
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	578 for fieldname in dataset.fieldNames():
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	579 if fieldname in self.fieldname2dataset: # name conflict!
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	580 if accept_nonunique_names:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	581 fieldname=rename_field(fieldname,dataset,i)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	582 names2change.append((fieldname,i))
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	583 else:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	584 raise ValueError("Incompatible datasets: non-unique field name = "+fieldname)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	585 self.fieldname2dataset[fieldname]=i
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	586 for fieldname,i in names_to_change:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	587 del self.fieldname2dataset[fieldname]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	588 self.fieldname2dataset[rename_field(fieldname,self.datasets[i],i)]=i
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	589
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	590 def hasFields(self,*fieldnames):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	591 for fieldname in fieldnames:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	592 if not fieldname in self.fieldname2dataset:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	593 return False
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	594 return True
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	595
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	596 def fieldNames(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	597 return self.fieldname2dataset.keys()
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	598
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	599 def minibatches_nowrap(self,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	600 fieldnames = minibatches_fieldnames,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	601 minibatch_size = minibatches_minibatch_size,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	602 n_batches = minibatches_n_batches,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	603 offset = 0):
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	604
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	605 class Iterator(object):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	606 def __init__(self,hsds,iterators):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	607 self.hsds=hsds
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	608 self.iterators=iterators
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	609 def __iter__(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	610 return self
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	611 def next(self):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	612 # concatenate all the fields of the minibatches
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	613 minibatch = reduce(LookupList.__add__,[iterator.next() for iterator in self.iterators])
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	614 # and return a DataSetFields whose dataset is the transpose (=examples()) of this minibatch
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	615 return DataSetFields(MinibatchDataSet(minibatch,self.hsds.valuesVStack,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	616 self.hsds.valuesHStack),
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	617 fieldnames if fieldnames else hsds.fieldNames())
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	618
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	619 assert self.hasfields(fieldnames)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	620 # find out which underlying datasets are necessary to service the required fields
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	621 # and construct corresponding minibatch iterators
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	622 if fieldnames:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	623 datasets=set([])
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	624 fields_in_dataset=dict([(dataset,[]) for dataset in datasets])
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	625 for fieldname in fieldnames:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	626 dataset=self.datasets[self.fieldnames2dataset[fieldname]]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	627 datasets.add(dataset)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	628 fields_in_dataset[dataset].append(fieldname)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	629 datasets=list(datasets)
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	630 iterators=[dataset.minibatches(fields_in_dataset[dataset],minibatch_size,n_batches,offset)
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	631 for dataset in datasets]
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	632 else:
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	633 datasets=self.datasets
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	634 iterators=[dataset.minibatches(None,minibatch_size,n_batches,offset) for dataset in datasets]
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	635 return Iterator(self,iterators)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	636
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	637
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	638 def valuesVStack(self,fieldname,fieldvalues):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	639 return self.datasets[self.fieldname2dataset[fieldname]].valuesVStack(fieldname,fieldvalues)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	640
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	641 def valuesHStack(self,fieldnames,fieldvalues):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	642 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	643 We will use the sub-dataset associated with the first fieldname in the fieldnames list
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	644 to do the work, hoping that it can cope with the other values (i.e. won't care
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	645 about the incompatible fieldnames). Hence this heuristic will always work if
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	646 all the fieldnames are of the same sub-dataset.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	647 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	648 return self.datasets[self.fieldname2dataset[fieldnames[0]]].valuesHStack(fieldnames,fieldvalues)
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	649
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	650 class VStackedDataSet(DataSet):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	651 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	652 A DataSet that wraps several datasets and shows a view that includes all their examples,
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	653 in the order provided. This clearly assumes that they all have the same field names
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	654 and all (except possibly the last one) are of finite length.
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	655
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	656 TODO: automatically detect a chain of stacked datasets due to A + B + C + D ...
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	657 """
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	658 def __init__(self,datasets):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	659 self.datasets=datasets
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	660 self.length=0
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	661 self.index2dataset={}
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	662 assert len(datasets)>0
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	663 fieldnames = datasets[-1].fieldNames()
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	664 self.datasets_start_row=[]
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	665 # We use this map from row index to dataset index for constant-time random access of examples,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	666 # to avoid having to search for the appropriate dataset each time and slice is asked for.
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	667 for dataset,k in enumerate(datasets[0:-1]):
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	668 try:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	669 L=len(dataset)
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	670 except UnboundedDataSet:
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	671 print "All VStacked datasets (except possibly the last) must be bounded (have a length)."
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	672 assert False
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	673 for i in xrange(L):
438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	674 self.index2dataset[self.length+i]=k
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	675 self.datasets_start_row.append(self.length)
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	676 self.length+=L
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	677 assert dataset.fieldNames()==fieldnames
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	678 self.datasets_start_row.append(self.length)
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	679 self.length+=len(datasets[-1])
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	680 # If length is very large, we should use a more memory-efficient mechanism
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	681 # that does not store all indices
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	682 if self.length>1000000:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	683 # 1 million entries would require about 60 meg for the index2dataset map
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	684 # TODO
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	685 print "A more efficient mechanism for index2dataset should be implemented"
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	686
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	687 def __len__(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	688 return self.length
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	689
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	690 def fieldNames(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	691 return self.datasets[0].fieldNames()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	692
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	693 def hasFields(self,*fieldnames):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	694 return self.datasets[0].hasFields(*fieldnames)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	695
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	696 def locate_row(self,row):
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	697 """Return (dataset_index, row_within_dataset) for global row number"""
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	698 dataset_index = self.index2dataset[row]
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	699 row_within_dataset = self.datasets_start_row[dataset_index]
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	700 return dataset_index, row_within_dataset
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	701
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	702 def minibatches_nowrap(self,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	703 fieldnames = minibatches_fieldnames,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	704 minibatch_size = minibatches_minibatch_size,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	705 n_batches = minibatches_n_batches,
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	706 offset = 0):
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	707
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	708 class Iterator(object):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	709 def __init__(self,vsds):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	710 self.vsds=vsds
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	711 self.next_row=offset
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	712 self.next_dataset_index,self.next_dataset_row=self.vsds.locate_row(offset)
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	713 self.current_iterator,self.n_left_at_the_end_of_ds,self.n_left_in_mb= \
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	714 self.next_iterator(vsds.datasets[0],offset,n_batches)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	715
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	716 def next_iterator(self,dataset,starting_offset,batches_left):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	717 L=len(dataset)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	718 ds_nbatches = (L-starting_offset)/minibatch_size
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	719 if batches_left is not None:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	720 ds_nbatches = max(batches_left,ds_nbatches)
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	721 if minibatch_size>L:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	722 ds_minibatch_size=L
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	723 n_left_in_mb=minibatch_size-L
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	724 ds_nbatches=1
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	725 else:
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	726 n_left_in_mb=0
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	727 return dataset.minibatches(fieldnames,minibatch_size,ds_nbatches,starting_offset), \
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	728 L-(starting_offset+ds_nbatches*minibatch_size), n_left_in_mb
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	729
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	730 def move_to_next_dataset(self):
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	731 if self.n_left_at_the_end_of_ds>0:
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	732 self.current_iterator,self.n_left_at_the_end_of_ds,self.n_left_in_mb= \
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	733 self.next_iterator(vsds.datasets[self.next_dataset_index],
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	734 self.n_left_at_the_end_of_ds,1)
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	735 else:
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	736 self.next_dataset_index +=1
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	737 if self.next_dataset_index==len(self.vsds.datasets):
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	738 self.next_dataset_index = 0
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	739 self.current_iterator,self.n_left_at_the_end_of_ds,self.n_left_in_mb= \
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	740 self.next_iterator(vsds.datasets[self.next_dataset_index],starting_offset,n_batches)
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	741
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	742 def __iter__(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	743 return self
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	744
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	745 def next(self):
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	746 dataset=self.vsds.datasets[self.next_dataset_index]
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	747 mb = self.next_iterator.next()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	748 if self.n_left_in_mb:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	749 extra_mb = []
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	750 while self.n_left_in_mb>0:
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	751 self.move_to_next_dataset()
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	752 extra_mb.append(self.next_iterator.next())
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	753 examples = Example(names,
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	754 [dataset.valuesVStack(name,
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	755 [mb[name]]+[b[name] for b in extra_mb])
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	756 for name in fieldnames])
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	757 mb = DataSetFields(MinibatchDataSet(examples),fieldnames)
88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	758
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	759 self.next_row+=minibatch_size
38 d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	760 self.next_dataset_row+=minibatch_size
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	761 if self.next_row+minibatch_size>len(dataset):
d637ad8f7352 Finished first untested version of VStackedDataset bengioy@esprit.iro.umontreal.ca parents: 37 diff changeset	762 self.move_to_next_dataset()
40 88fd1cce08b9 replaced infinity for length by raise UnboundedDataSet and use & instead of + to concatenate datasets bengioy@esprit.iro.umontreal.ca parents: 39 diff changeset	763 return
37 73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	764
73c4212ba5b3 Factored the minibatch-writing code into an iterator class inside DataSet bengioy@esprit.iro.umontreal.ca parents: 36 diff changeset	765
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	766 def supervised_learning_dataset(src_dataset,input_fields,target_fields,weight_field=None):
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	767 """
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	768 Wraps an arbitrary DataSet into one for supervised learning tasks by forcing the
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	769 user to define a set of fields as the 'input' field and a set of fields
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	770 as the 'target' field. Optionally, a single weight_field can also be defined.
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	771 """
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	772 args = ((input_fields,'input'),(output_fields,'target'))
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	773 if weight_field: args+=(([weight_field],'weight'))
36 438440ba0627 Rewriting dataset.py completely bengioy@zircon.iro.umontreal.ca parents: 29 diff changeset	774 return src_dataset.merge_fields(*args)
23 526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	775
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	776
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	777
526e192b0699 Working on ApplyFunctionDataSet, added constraint that bengioy@esprit.iro.umontreal.ca parents: 22 diff changeset	778

Mercurial > pylearn

annotate dataset.py @ 40:88fd1cce08b9