Mercurial > pylearn

annotate _test_dataset.py @ 221:58e17421c69c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
author Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
date Fri, 23 May 2008 14:07:53 -0400
parents 1f527fe65e22
children 174374d59405
rev   line source
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
1 from dataset import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
2 from math import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
3 import unittest
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
4 import sys
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
5 import numpy as N
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
6
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
7 def _sum_all(a):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
8 s=a
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
9 while isinstance(s,numpy.ndarray):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
10 s=sum(s)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
11 return s
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
12
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
13 class T_arraydataset(unittest.TestCase):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
14 def setUp(self):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
15 numpy.random.seed(123456)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
16
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
18 def test_ctor_len(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
19 n = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
20 a=ArrayDataSet(n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
21 self.failUnless(a.data is n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
22 self.failUnless(a.fields is None)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
23
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
24 self.failUnless(len(a) == n.shape[0])
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
25 self.failUnless(a[0].shape == (n.shape[1],))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
26
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
27 def test_iter(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
28 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
29 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
30 for i, example in enumerate(a):
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
31 self.failUnless(numpy.all( example['x'] == arr[i,:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
32 self.failUnless(numpy.all( example['y'] == arr[i,1:3]))
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
33
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
34 def test_zip(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
35 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
36 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
37 for i, x in enumerate(a.zip("x")):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
38 self.failUnless(numpy.all( x == arr[i,:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
39
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
40 def test_minibatch_basic(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
41 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
42 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
43 for i, mb in enumerate(a.minibatches(minibatch_size=2)): #all fields
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
44 self.failUnless(numpy.all( mb['x'] == arr[i*2:i*2+2,0:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
45 self.failUnless(numpy.all( mb['y'] == arr[i*2:i*2+2,1:4]))
7
6f8f338686db Moved iterating counter into a FiniteDataSetIterator to allow embedded iterations and multiple threads iterating at the same time on a dataset.
bengioy@bengiomac.local
parents: 6
diff changeset
46
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
47 def test_getattr(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
48 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
49 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
50 a_y = a.y
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
51 self.failUnless(numpy.all( a_y == arr[:,1:4]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
52
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
53 def test_minibatch_wraparound_even(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
54 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
55 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
56
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
57 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
58
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
59 #print arr
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
60 for i, x in enumerate(a.minibatches(["x"], minibatch_size=2, n_batches=8)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
61 #print 'x' , x
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
62 self.failUnless(numpy.all( x == arr2[i*2:i*2+2,0:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
63
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
64 def test_minibatch_wraparound_odd(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
65 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
66 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
67
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
68 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
69
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
70 for i, x in enumerate(a.minibatches(["x"], minibatch_size=3, n_batches=6)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
71 self.failUnless(numpy.all( x == arr2[i*3:i*3+3,0:2]))
19
57f4015e2e09 Iterators extend LookupList
bergstrj@iro.umontreal.ca
parents: 17
diff changeset
72
26
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
73
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
74 class T_renamingdataset(unittest.TestCase):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
75 def setUp(self):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
76 numpy.random.seed(123456)
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
77
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
78
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
79 def test_hasfield(self):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
80 n = numpy.random.rand(3,8)
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
81 a=ArrayDataSet(data=n,fields={"x":slice(2),"y":slice(1,4),"z":slice(4,6)})
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
82 b=a.rename({'xx':'x','zz':'z'})
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
83 self.failUnless(b.hasFields('xx','zz') and not b.hasFields('x') and not b.hasFields('y'))
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
84
29
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
85 class T_applyfunctiondataset(unittest.TestCase):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
86 def setUp(self):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
87 numpy.random.seed(123456)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
88
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
89 def test_function(self):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
90 n = numpy.random.rand(3,8)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
91 a=ArrayDataSet(data=n,fields={"x":slice(2),"y":slice(1,4),"z":slice(4,6)})
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
92 b=a.apply_function(lambda x,y: x+y,x+1, ['x','y'], ['x+y','x+1'], False,False,False)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
93 print b.fieldNames()
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
94 print b('x+y')
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
95
26
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
96
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
97
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
98
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
99 # to be used with a any new dataset
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
100 class T_dataset_tester(object):
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
101 """
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
102 This class' goal is to test any new dataset that is created
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
103 Tests are (will be!) designed to check the normal behaviours
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
104 of a dataset, as defined in dataset.py
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
105 """
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
106
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
107
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
108 def __init__(self,ds,runall=True) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
109 """if interested in only a subset of test, init with runall=False"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
110 self.ds = ds
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
111
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
112 if runall :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
113 self.test1_basicstats(ds)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
114 self.test2_slicing(ds)
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
115 self.test3_fields_iterator_consistency(ds)
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
116
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
117 def test1_basicstats(self,ds) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
118 """print basics stats on a dataset, like length"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
119
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
120 print 'len(ds) = ',len(ds)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
121 print 'num fields = ', len(ds.fieldNames())
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
122 print 'types of field: ',
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
123 for k in ds.fieldNames() :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
124 print type(ds[0](k)[0]),
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
125 print ''
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
126
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
127 def test2_slicing(self,ds) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
128 """test if slicing works properly"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
129 print 'testing slicing...',
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
130 sys.stdout.flush()
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
131
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
132 middle = len(ds) / 2
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
133 tenpercent = int(len(ds) * .1)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
134 set1 = ds[:middle+tenpercent]
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
135 set2 = ds[middle-tenpercent:]
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
136 for k in range(tenpercent + tenpercent -1):
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
137 for k2 in ds.fieldNames() :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
138 if type(set1[middle-tenpercent+k](k2)[0]) == N.ndarray :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
139 for k3 in range(len(set1[middle-tenpercent+k](k2)[0])) :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
140 assert set1[middle-tenpercent+k](k2)[0][k3] == set2[k](k2)[0][k3]
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
141 else :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
142 assert set1[middle-tenpercent+k](k2)[0] == set2[k](k2)[0]
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
143 assert tenpercent > 1
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
144 set3 = ds[middle-tenpercent:middle+tenpercent:2]
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
145 for k2 in ds.fieldNames() :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
146 if type(set2[2](k2)[0]) == N.ndarray :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
147 for k3 in range(len(set2[2](k2)[0])) :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
148 assert set2[2](k2)[0][k3] == set3[1](k2)[0][k3]
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
149 else :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
150 assert set2[2](k2)[0] == set3[1](k2)[0]
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
151
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
152 print 'done'
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
153
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
154
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
155 def test3_fields_iterator_consistency(self,ds) :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
156 """ check if the number of iterator corresponds to the number of fields"""
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
157 print 'testing fields/iterator consistency...',
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
158 sys.stdout.flush()
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
159
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
160 # basic test
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
161 maxsize = min(len(ds)-1,100)
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
162 for iter in ds[:maxsize] :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
163 assert len(iter) == len(ds.fieldNames())
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
164 if len(ds.fieldNames()) == 1 :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
165 print 'done'
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
166 return
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
167
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
168 # with minibatches iterator
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
169 ds2 = ds.minibatches[:maxsize]([ds.fieldNames()[0],ds.fieldNames()[1]],minibatch_size=2)
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
170 for iter in ds2 :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
171 assert len(iter) == 2
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
172
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
173 print 'done'
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
174
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
175
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
176
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
177
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
178
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
179 ###################################################################
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
180 # main
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
181 if __name__ == '__main__':
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
182 unittest.main()
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
183