Mercurial > ift6266

annotate baseline/mlp/mlp_get_error_from_model.py @ 596:f6a3b28b002c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
nips2010_submission.pdf
author Yoshua Bengio <bengioy@iro.umontreal.ca>
date Thu, 14 Oct 2010 15:52:02 -0400
parents 9b6e0af062af
children
rev   line source
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
1 __docformat__ = 'restructedtext en'
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
2
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
3 import pdb
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
4 import numpy as np
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
5 import pylab
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
6 import time
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
7 import pylearn
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
8 from pylearn.io import filetensor as ft
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
9
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
10 data_path = '/data/lisa/data/nist/by_class/'
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
11 test_data = 'all/all_train_data.ft'
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
12 test_labels = 'all/all_train_labels.ft'
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
13
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
14 def read_test_data(mlp_model):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
15
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
16
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
17 #read the data
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
18 h = open(data_path+test_data)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
19 i= open(data_path+test_labels)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
20 raw_test_data = ft.read(h)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
21 raw_test_labels = ft.read(i)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
22 i.close()
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
23 h.close()
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
24
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
25 #read the model chosen
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
26 a=np.load(mlp_model)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
27 W1=a['W1']
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
28 W2=a['W2']
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
29 b1=a['b1']
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
30 b2=a['b2']
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
31
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
32 return (W1,b1,W2,b2,raw_test_data,raw_test_labels)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
33
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
34
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
35
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
36
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
37 def get_total_test_error(everything):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
38
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
39 W1=everything[0]
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
40 b1=everything[1]
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
41 W2=everything[2]
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
42 b2=everything[3]
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
43 test_data=everything[4]
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
44 test_labels=everything[5]
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
45 total_error_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
46 total_exemple_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
47
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
48 nb_error_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
49 nb_exemple_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
50
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
51 char_error_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
52 char_exemple_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
53
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
54 min_error_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
55 min_exemple_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
56
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
57 maj_error_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
58 maj_exemple_count=0
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
59
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
60 for i in range(test_labels.size):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
61 total_exemple_count = total_exemple_count +1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
62 #get activation for layer 1
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
63 a0=np.dot(np.transpose(W1),np.transpose(test_data[i]/255.0)) + b1
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
64 #add non linear function to layer 1 activation
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
65 a0_out=np.tanh(a0)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
66
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
67 #get activation for output layer
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
68 a1= np.dot(np.transpose(W2),a0_out) + b2
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
69 #add non linear function for output activation (softmax)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
70 a1_exp = np.exp(a1)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
71 sum_a1=np.sum(a1_exp)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
72 a1_out=a1_exp/sum_a1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
73
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
74 predicted_class=np.argmax(a1_out)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
75 wanted_class=test_labels[i]
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
76
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
77 if(predicted_class!=wanted_class):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
78 total_error_count = total_error_count +1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
79
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
80 #get grouped based error
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
81 #with a priori
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
82 # if(wanted_class>9 and wanted_class<35):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
83 # min_exemple_count=min_exemple_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
84 # predicted_class=np.argmax(a1_out[10:35])+10
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
85 # if(predicted_class!=wanted_class):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
86 # min_error_count=min_error_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
87 # if(wanted_class<10):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
88 # nb_exemple_count=nb_exemple_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
89 # predicted_class=np.argmax(a1_out[0:10])
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
90 # if(predicted_class!=wanted_class):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
91 # nb_error_count=nb_error_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
92 # if(wanted_class>34):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
93 # maj_exemple_count=maj_exemple_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
94 # predicted_class=np.argmax(a1_out[35:])+35
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
95 # if(predicted_class!=wanted_class):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
96 # maj_error_count=maj_error_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
97 #
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
98 # if(wanted_class>9):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
99 # char_exemple_count=char_exemple_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
100 # predicted_class=np.argmax(a1_out[10:])+10
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
101 # if(predicted_class!=wanted_class):
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
102 # char_error_count=char_error_count+1
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
103
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
104
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
105
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
106 #get grouped based error
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
107 #with no a priori
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
108 if(wanted_class>9 and wanted_class<35):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
109 min_exemple_count=min_exemple_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
110 predicted_class=np.argmax(a1_out)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
111 if(predicted_class!=wanted_class):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
112 min_error_count=min_error_count+1
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
113 if(wanted_class<10):
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
114 nb_exemple_count=nb_exemple_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
115 predicted_class=np.argmax(a1_out)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
116 if(predicted_class!=wanted_class):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
117 nb_error_count=nb_error_count+1
237
9b6e0af062af corrected a bug in jobman interface
xaviermuller
parents: 212
diff changeset
118 if(wanted_class>34):
212
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
119 maj_exemple_count=maj_exemple_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
120 predicted_class=np.argmax(a1_out)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
121 if(predicted_class!=wanted_class):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
122 maj_error_count=maj_error_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
123
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
124 if(wanted_class>9):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
125 char_exemple_count=char_exemple_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
126 predicted_class=np.argmax(a1_out)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
127 if(predicted_class!=wanted_class):
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
128 char_error_count=char_error_count+1
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
129
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
130
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
131 #convert to float
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
132 return ( total_exemple_count,nb_exemple_count,char_exemple_count,min_exemple_count,maj_exemple_count,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
133 total_error_count,nb_error_count,char_error_count,min_error_count,maj_error_count,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
134 total_error_count*100.0/total_exemple_count*1.0,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
135 nb_error_count*100.0/nb_exemple_count*1.0,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
136 char_error_count*100.0/char_exemple_count*1.0,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
137 min_error_count*100.0/min_exemple_count*1.0,\
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
138 maj_error_count*100.0/maj_exemple_count*1.0)
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
139
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
140
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
141
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
142
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
143
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
144
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
145
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
146
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
147
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
148
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
149
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
150
e390b0454515 added classic lr time decay and py code to calculate the error based on a saved model
xaviermuller
parents:
diff changeset
151