Mercurial > pylearn

annotate _test_dataset.py @ 22:b6b36f65664f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet, removed .field ability from LookupList (because of setattr problems), removed fieldNames() from DataSet (but is in FiniteWidthDataSet, where it makes sense), and added hasFields() instead. Fixed problems in asarray, and tested previous functionality in _test_dataset.py, but not yet new functionality.
author bengioy@esprit.iro.umontreal.ca
date Mon, 07 Apr 2008 20:44:37 -0400
parents fdf0abc490f7
children 672fe4b23032
rev   line source
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
1 from dataset import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
2 from math import *
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
3 import unittest
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
5 def _sum_all(a):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
6 s=a
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
7 while isinstance(s,numpy.ndarray):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
8 s=sum(s)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
9 return s
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
10
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
11 class T_arraydataset(unittest.TestCase):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
12 def setUp(self):
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
13 numpy.random.seed(123456)
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
14
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
15
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
16 def test_ctor_len(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
17 n = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
18 a=ArrayDataSet(n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
19 self.failUnless(a.data is n)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
20 self.failUnless(a.fields is None)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
21
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
22 self.failUnless(len(a) == n.shape[0])
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
23 self.failUnless(a[0].shape == (n.shape[1],))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
24
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
25 def test_iter(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
26 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
27 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
28 for i, example in enumerate(a):
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
29 self.failUnless(numpy.all( example['x'] == arr[i,:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
30 self.failUnless(numpy.all( example['y'] == arr[i,1:3]))
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
31
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
32 def test_zip(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
33 arr = numpy.random.rand(8,3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
34 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,3)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
35 for i, x in enumerate(a.zip("x")):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
36 self.failUnless(numpy.all( x == arr[i,:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
37
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
38 def test_minibatch_basic(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
39 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
40 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
41 for i, mb in enumerate(a.minibatches(minibatch_size=2)): #all fields
21
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
42 self.failUnless(numpy.all( mb['x'] == arr[i*2:i*2+2,0:2]))
fdf0abc490f7 Adapted _test_dataset.py to changes in LookupList
bengioy@bengiomac.local
parents: 19
diff changeset
43 self.failUnless(numpy.all( mb['y'] == arr[i*2:i*2+2,1:4]))
7
6f8f338686db Moved iterating counter into a FiniteDataSetIterator to allow embedded iterations and multiple threads iterating at the same time on a dataset.
bengioy@bengiomac.local
parents: 6
diff changeset
44
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
45 def test_getattr(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
46 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
47 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
48 a_y = a.y
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
49 self.failUnless(numpy.all( a_y == arr[:,1:4]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
50
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
51 def test_asarray(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
52 arr = numpy.random.rand(3,4)
22
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet,
bengioy@esprit.iro.umontreal.ca
parents: 21
diff changeset
53 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(2,4)})
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet,
bengioy@esprit.iro.umontreal.ca
parents: 21
diff changeset
54 a_arr = numpy.asarray(a)
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet,
bengioy@esprit.iro.umontreal.ca
parents: 21
diff changeset
55 self.failUnless(a_arr.shape[1] == 2 + 2)
b6b36f65664f Created virtual sub-classes of DataSet: {Finite{Length,Width},Sliceable}DataSet,
bengioy@esprit.iro.umontreal.ca
parents: 21
diff changeset
56 self.failUnless(numpy.sum(numpy.square(a_arr-a.data))==0)
17
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
57 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
58 a_arr = numpy.asarray(a)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
59 self.failUnless(a_arr.shape[1] == 2 + 3)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
60
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
61 def test_minibatch_wraparound_even(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
62 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
63 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
64
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
65 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
66
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
67 #print arr
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
68 for i, x in enumerate(a.minibatches(["x"], minibatch_size=2, n_batches=8)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
69 #print 'x' , x
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
70 self.failUnless(numpy.all( x == arr2[i*2:i*2+2,0:2]))
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
71
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
72 def test_minibatch_wraparound_odd(self):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
73 arr = numpy.random.rand(10,4)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
74 arr2 = ArrayDataSet.Iterator.matcat(arr,arr)
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
75
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
76 a=ArrayDataSet(data=arr,fields={"x":slice(2),"y":slice(1,4)})
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
77
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
78 for i, x in enumerate(a.minibatches(["x"], minibatch_size=3, n_batches=6)):
759d17112b23 more comments, looping ArrayDataSet iterator, bugfixes to lookup_list, more tests
bergstrj@iro.umontreal.ca
parents: 11
diff changeset
79 self.failUnless(numpy.all( x == arr2[i*3:i*3+3,0:2]))
19
57f4015e2e09 Iterators extend LookupList
bergstrj@iro.umontreal.ca
parents: 17
diff changeset
80
4
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
81 if __name__ == '__main__':
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
82 unittest.main()
f7dcfb5f9d5b Added test for dataset.
bengioy@bengiomac.local
parents:
diff changeset
83