Mercurial > pylearn

annotate _test_dataset.py @ 259:621faba17c60

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
created 'dummytests', tests that checks consistency of new weird datasets, where we can't compare with actual values in a matrix, for instance. Useful as a first debugging when creating a dataset
author Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
date Tue, 03 Jun 2008 16:41:55 -0400
parents 58e17421c69c
children 174374d59405
rev   line source
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
1 from dataset import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
2 from math import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
3 import unittest
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
4 import sys
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
5 import numpy as N
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
6
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
7 def _sum_all(a):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
8 s=a
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
9 while isinstance(s,numpy.ndarray):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
10 s=sum(s)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
11 return s
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
12
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
13 class T_arraydataset(unittest.TestCase):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
14 def setUp(self):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
15 numpy.random.seed(123456)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
16
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
18 def test_ctor_len(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
19 n = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
20 a=ArrayDataSet(n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
21 self.failUnless(a.data is n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
22 self.failUnless(a.fields is None)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
23
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
24 self.failUnless(len(a) == n.shape[0])
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
25 self.failUnless(a[0].shape == (n.shape[1],))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
26
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
27 def test_iter(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
28 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
29 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
30 for i, example in enumerate(a):
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
31 self.failUnless(numpy.all( example['x'] == arr[i,:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
32 self.failUnless(numpy.all( example['y'] == arr[i,1:3]))
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
33
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
34 def test_zip(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
35 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
36 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
37 for i, x in enumerate(a.zip("x")):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
38 self.failUnless(numpy.all( x == arr[i,:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
39
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
40 def test_minibatch_basic(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
41 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
42 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
43 for i, mb in enumerate(a.minibatches(minibatch_size=2)): #all fields
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
44 self.failUnless(numpy.all( mb['x'] == arr[i*2:i*2+2,0:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
45 self.failUnless(numpy.all( mb['y'] == arr[i*2:i*2+2,1:4]))
7
6f8f338686db Moved iterating counter into a FiniteDataSetIterator to allow embedded iterations and multiple threads iterating at the same time on a dataset.
bengioy@bengiomac.local
parents: 6
diff changeset
46
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
47 def test_getattr(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
48 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
49 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
50 a_y = a.y
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
51 self.failUnless(numpy.all( a_y == arr[:,1:4]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
52
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
53 def test_minibatch_wraparound_even(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
54 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
55 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
56
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
57 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
58
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
59 #print arr
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
60 for i, x in enumerate(a.minibatches(["x"], minibatch_size=2, n_batches=8)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
61 #print 'x' , x
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
62 self.failUnless(numpy.all( x == arr2[i*2:i*2+2,0:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
63
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
64 def test_minibatch_wraparound_odd(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
65 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
66 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
67
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
68 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
69
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
70 for i, x in enumerate(a.minibatches(["x"], minibatch_size=3, n_batches=6)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
71 self.failUnless(numpy.all( x == arr2[i*3:i*3+3,0:2]))
19
57f4015e2e09 Iterators extend LookupList
bergstrj@iro.umontreal.ca
parents: 17
diff changeset
72
26
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
73
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
74 class T_renamingdataset(unittest.TestCase):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
75 def setUp(self):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
76 numpy.random.seed(123456)
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
77
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
78
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
79 def test_hasfield(self):
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
80 n = numpy.random.rand(3,8)
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
81 a=ArrayDataSet(data=n,fields={"x":slice(2),"y":slice(1,4),"z":slice(4,6)})
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
82 b=a.rename({'xx':'x','zz':'z'})
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
83 self.failUnless(b.hasFields('xx','zz') and not b.hasFields('x') and not b.hasFields('y'))
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
84
29
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
85 class T_applyfunctiondataset(unittest.TestCase):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
86 def setUp(self):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
87 numpy.random.seed(123456)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
88
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
89 def test_function(self):
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
90 n = numpy.random.rand(3,8)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
91 a=ArrayDataSet(data=n,fields={"x":slice(2),"y":slice(1,4),"z":slice(4,6)})
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
92 b=a.apply_function(lambda x,y: x+y,x+1, ['x','y'], ['x+y','x+1'], False,False,False)
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
93 print b.fieldNames()
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
94 print b('x+y')
46c5c90019c2 Changed apply_function so that it propagates methods of the source.
bengioy@grenat.iro.umontreal.ca
parents: 28
diff changeset
95
26
672fe4b23032 Fixed dataset errors so that _test_dataset.py works again.
bengioy@grenat.iro.umontreal.ca
parents: 22
diff changeset
96
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
97
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
98
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
99 # to be used with a any new dataset
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
100 class T_dataset_tester(object):
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
101 """
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
102 This class' goal is to test any new dataset that is created
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
103 Tests are (will be!) designed to check the normal behaviours
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
104 of a dataset, as defined in dataset.py
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
105 """
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
106
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
107
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
108 def __init__(self,ds,runall=True) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
109 """if interested in only a subset of test, init with runall=False"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
110 self.ds = ds
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
111
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
112 if runall :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
113 self.test1_basicstats(ds)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
114 self.test2_slicing(ds)
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
115 self.test3_fields_iterator_consistency(ds)
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
116
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
117 def test1_basicstats(self,ds) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
118 """print basics stats on a dataset, like length"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
119
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
120 print 'len(ds) = ',len(ds)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
121 print 'num fields = ', len(ds.fieldNames())
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
122 print 'types of field: ',
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
123 for k in ds.fieldNames() :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
124 print type(ds[0](k)[0]),
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
125 print ''
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
126
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
127 def test2_slicing(self,ds) :
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
128 """test if slicing works properly"""
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
129 print 'testing slicing...',
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
130 sys.stdout.flush()
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
131
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
132 middle = len(ds) / 2
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
133 tenpercent = int(len(ds) * .1)
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
134 set1 = ds[:middle+tenpercent]
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
135 set2 = ds[middle-tenpercent:]
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
136 for k in range(tenpercent + tenpercent -1):
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
137 for k2 in ds.fieldNames() :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
138 if type(set1[middle-tenpercent+k](k2)[0]) == N.ndarray :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
139 for k3 in range(len(set1[middle-tenpercent+k](k2)[0])) :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
140 assert set1[middle-tenpercent+k](k2)[0][k3] == set2[k](k2)[0][k3]
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
141 else :
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
142 assert set1[middle-tenpercent+k](k2)[0] == set2[k](k2)[0]
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
143 assert tenpercent > 1
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
144 set3 = ds[middle-tenpercent:middle+tenpercent:2]
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
145 for k2 in ds.fieldNames() :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
146 if type(set2[2](k2)[0]) == N.ndarray :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
147 for k3 in range(len(set2[2](k2)[0])) :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
148 assert set2[2](k2)[0][k3] == set3[1](k2)[0][k3]
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
149 else :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
150 assert set2[2](k2)[0] == set3[1](k2)[0]
220
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
151
1f527fe65e22 test on simple slicing works
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 219
diff changeset
152 print 'done'
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
153
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
154
221
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
155 def test3_fields_iterator_consistency(self,ds) :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
156 """ check if the number of iterator corresponds to the number of fields"""
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
157 print 'testing fields/iterator consistency...',
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
158 sys.stdout.flush()
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
159
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
160 # basic test
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
161 maxsize = min(len(ds)-1,100)
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
162 for iter in ds[:maxsize] :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
163 assert len(iter) == len(ds.fieldNames())
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
164 if len(ds.fieldNames()) == 1 :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
165 print 'done'
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
166 return
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
167
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
168 # with minibatches iterator
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
169 ds2 = ds.minibatches[:maxsize]([ds.fieldNames()[0],ds.fieldNames()[1]],minibatch_size=2)
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
170 for iter in ds2 :
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
171 assert len(iter) == 2
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
172
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
173 print 'done'
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
174
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
175
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
176
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
177
58e17421c69c tester on iterator consistency now triggers a bug in dataset, linked to the combination of minibatch and slicing
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 220
diff changeset
178
219
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
179 ###################################################################
5b3afda2f1ad added a class to test any new dataset
Thierry Bertin-Mahieux <bertinmt@iro.umontreal.ca>
parents: 29
diff changeset
180 # main
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
181 if __name__ == '__main__':
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
182 unittest.main()
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
183