Mercurial > ift6266

annotate baseline/log_reg/log_reg.py @ 647:47af8a002530 tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
changed Theano to ift6266 and remove numpy as we do not use code from numpy in this repository
author Razvan Pascanu <r.pascanu@gmail.com>
date Wed, 17 Oct 2012 09:26:14 -0400
parents 5541056d3fb0
children
rev   line source
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
1 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
2 This tutorial introduces logistic regression using Theano and stochastic
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
3 gradient descent.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
4
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
5 Logistic regression is a probabilistic, linear classifier. It is parametrized
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
6 by a weight matrix :math:`W` and a bias vector :math:`b`. Classification is
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
7 done by projecting data points onto a set of hyperplanes, the distance to
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
8 which is used to determine a class membership probability.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
9
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
10 Mathematically, this can be written as:
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
11
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
12 .. math::
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
13 P(Y=i|x, W,b) &= softmax_i(W x + b) \\
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
14 &= \frac {e^{W_i x + b_i}} {\sum_j e^{W_j x + b_j}}
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
15
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
16
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
17 The output of the model or prediction is then done by taking the argmax of
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
18 the vector whose i'th element is P(Y=i|x).
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
19
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
20 .. math::
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
21
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
22 y_{pred} = argmax_i P(Y=i|x,W,b)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
23
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
24
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
25 This tutorial presents a stochastic gradient descent optimization method
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
26 suitable for large datasets, and a conjugate gradient optimization method
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
27 that is suitable for smaller datasets.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
28
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
29
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
30 References:
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
31
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
32 - textbooks: "Pattern Recognition and Machine Learning" -
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
33 Christopher M. Bishop, section 4.3.2
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
34
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
35 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
36 __docformat__ = 'restructedtext en'
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
37
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
38 import numpy, time
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
39
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
40 import theano
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
41 import theano.tensor as T
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
42 from ift6266 import datasets
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
43
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
44 class LogisticRegression(object):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
45 """Multi-class Logistic Regression Class
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
46
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
47 The logistic regression is fully described by a weight matrix :math:`W`
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
48 and bias vector :math:`b`. Classification is done by projecting data
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
49 points onto a set of hyperplanes, the distance to which is used to
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
50 determine a class membership probability.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
51 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
52
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
53
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
54 def __init__( self, input, n_in, n_out ):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
55 """ Initialize the parameters of the logistic regression
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
56
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
57 :type input: theano.tensor.TensorType
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
58 :param input: symbolic variable that describes the input of the
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
59 architecture (one minibatch)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
60
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
61 :type n_in: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
62 :param n_in: number of input units, the dimension of the space in
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
63 which the datapoints lie
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
64
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
65 :type n_out: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
66 :param n_out: number of output units, the dimension of the space in
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
67 which the labels lie
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
68
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
69 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
70
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
71 # initialize with 0 the weights W as a matrix of shape (n_in, n_out)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
72 self.W = theano.shared( value = numpy.zeros(( n_in, n_out ), dtype = theano.config.floatX ),
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
73 name =' W')
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
74 # initialize the baises b as a vector of n_out 0s
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
75 self.b = theano.shared( value = numpy.zeros(( n_out, ), dtype = theano.config.floatX ),
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
76 name = 'b')
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
77
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
78
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
79 # compute vector of class-membership probabilities in symbolic form
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
80 self.p_y_given_x = T.nnet.softmax( T.dot( input, self.W ) + self.b )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
81
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
82 # compute prediction as class whose probability is maximal in
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
83 # symbolic form
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
84 self.y_pred=T.argmax( self.p_y_given_x, axis =1 )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
85
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
86 # parameters of the model
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
87 self.params = [ self.W, self.b ]
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
88
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
89
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
90 def negative_log_likelihood( self, y ):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
91 """Return the mean of the negative log-likelihood of the prediction
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
92 of this model under a given target distribution.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
93
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
94 .. math::
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
95
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
96 \frac{1}{|\mathcal{D}|} \mathcal{L} (\theta=\{W,b\}, \mathcal{D}) =
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
97 \frac{1}{|\mathcal{D}|} \sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
98 \ell (\theta=\{W,b\}, \mathcal{D})
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
99
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
100 :type y: theano.tensor.TensorType
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
101 :param y: corresponds to a vector that gives for each example the
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
102 correct label
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
103
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
104 Note: we use the mean instead of the sum so that
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
105 the learning rate is less dependent on the batch size
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
106 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
107 # y.shape[0] is (symbolically) the number of rows in y, i.e., number of examples (call it n) in the minibatch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
108 # T.arange(y.shape[0]) is a symbolic vector which will contain [0,1,2,... n-1]
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
109 # T.log(self.p_y_given_x) is a matrix of Log-Probabilities (call it LP) with one row per example and one column per class
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
110 # LP[T.arange(y.shape[0]),y] is a vector v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
111 # and T.mean(LP[T.arange(y.shape[0]),y]) is the mean (across minibatch examples) of the elements in v,
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
112 # i.e., the mean log-likelihood across the minibatch.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
113 return -T.mean( T.log( self.p_y_given_x )[ T.arange( y.shape[0] ), y ] )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
114
199
777f48ba30df Add MSE cost to log_reg.py
Arnaud Bergeron <abergeron@gmail.com>
parents: 198
diff changeset
115 def MSE(self, y):
777f48ba30df Add MSE cost to log_reg.py
Arnaud Bergeron <abergeron@gmail.com>
parents: 198
diff changeset
116 return -T.mean(abs((self.p_t_given_x)[T.arange(y.shape[0]), y]-y)**2)
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
117
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
118 def errors( self, y ):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
119 """Return a float representing the number of errors in the minibatch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
120 over the total number of examples of the minibatch ; zero one
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
121 loss over the size of the minibatch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
122
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
123 :type y: theano.tensor.TensorType
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
124 :param y: corresponds to a vector that gives for each example the
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
125 correct label
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
126 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
127
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
128 # check if y has same dimension of y_pred
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
129 if y.ndim != self.y_pred.ndim:
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
130 raise TypeError( 'y should have the same shape as self.y_pred',
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
131 ( 'y', target.type, 'y_pred', self.y_pred.type ) )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
132 # check if y is of the correct datatype
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
133 if y.dtype.startswith('int'):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
134 # the T.neq operator returns a vector of 0s and 1s, where 1
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
135 # represents a mistake in prediction
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
136 return T.mean( T.neq( self.y_pred, y ) )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
137 else:
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
138 raise NotImplementedError()
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
139
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
140 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
141 # MAIN
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
142 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
143
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
144 def log_reg( learning_rate = 0.13, nb_max_examples =1000000, batch_size = 50, \
271
a92ec9939e4f fixed a problem with maxsize when not provided
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 241
diff changeset
145 dataset=datasets.nist_digits(), image_size = 32 * 32, nb_class = 10, \
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
146 patience = 5000, patience_increase = 2, improvement_threshold = 0.995):
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
147
236
7be1f086a89e added __init__.py to allow module loading of baseline
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 199
diff changeset
148 #28 * 28 = 784
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
149 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
150 Demonstrate stochastic gradient descent optimization of a log-linear
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
151 model
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
152
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
153 This is demonstrated on MNIST.
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
154
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
155 :type learning_rate: float
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
156 :param learning_rate: learning rate used (factor for the stochastic
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
157 gradient)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
159 :type nb_max_examples: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
160 :param nb_max_examples: maximal number of epochs to run the optimizer
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
161
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
162 :type batch_size: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
163 :param batch_size: size of the minibatch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
164
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
165 :type dataset: dataset
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
166 :param dataset: a dataset instance from ift6266.datasets
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
167
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
168 :type image_size: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
169 :param image_size: size of the input image in pixels (width * height)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
170
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
171 :type nb_class: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
172 :param nb_class: number of classes
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
173
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
174 :type patience: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
175 :param patience: look as this many examples regardless
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
176
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
177 :type patience_increase: int
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
178 :param patience_increase: wait this much longer when a new best is found
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
179
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
180 :type improvement_threshold: float
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
181 :param improvement_threshold: a relative improvement of this much is considered significant
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
182
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
183
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
184 """
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
185 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
186 # Build actual model
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
187 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
188
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
189 print '... building the model'
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
190
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
191 # allocate symbolic variables for the data
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
192 index = T.lscalar( ) # index to a [mini]batch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
193 x = T.matrix('x') # the data is presented as rasterized images
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
194 y = T.ivector('y') # the labels are presented as 1D vector of
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
195 # [int] labels
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
196
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
197 # construct the logistic regression class
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
198
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
199 classifier = LogisticRegression( input = x, n_in = image_size, n_out = nb_class )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
200
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
201 # the cost we minimize during training is the negative log likelihood of
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
202 # the model in symbolic format
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
203 cost = classifier.negative_log_likelihood( y )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
204
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
205 # compiling a Theano function that computes the mistakes that are made by
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
206 # the model on a minibatch
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
207 test_model = theano.function( inputs = [ x, y ],
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
208 outputs = classifier.errors( y ))
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
209
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
210 validate_model = theano.function( inputs = [ x, y ],
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
211 outputs = classifier.errors( y ))
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
212
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
213 # compute the gradient of cost with respect to theta = ( W, b )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
214 g_W = T.grad( cost = cost, wrt = classifier.W )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
215 g_b = T.grad( cost = cost, wrt = classifier.b )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
216
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
217 # specify how to update the parameters of the model as a dictionary
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
218 updates = { classifier.W: classifier.W - learning_rate * g_W,\
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
219 classifier.b: classifier.b - learning_rate * g_b}
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
220
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
221 # compiling a Theano function `train_model` that returns the cost, but in
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
222 # the same time updates the parameter of the model based on the rules
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
223 # defined in `updates`
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
224 train_model = theano.function( inputs = [ x, y ],
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
225 outputs = cost,
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
226 updates = updates)
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
227
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
228 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
229 # Train model
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
230 #--------------------------------------------------------------------------------------------------------------------
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
231
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
232 print '... training the model'
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
233 # early-stopping parameters
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
234 patience = 5000 # look as this many examples regardless
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
235 patience_increase = 2 # wait this much longer when a new best is
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
236 # found
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
237 improvement_threshold = 0.995 # a relative improvement of this much is
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
238 # considered significant
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
239 validation_frequency = patience * 0.5
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
240 # go through this many
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
241 # minibatche before checking the network
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
242 # on the validation set; in this case we
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
243 # check every epoch
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
244
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
245 best_params = None
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
246 best_validation_loss = float('inf')
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
247 test_score = 0.
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
248 start_time = time.clock()
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
249
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
250 done_looping = False
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
251 n_iters = nb_max_examples / batch_size
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
252 epoch = 0
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
253 iter = 0
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
254
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
255 while ( iter < n_iters ) and ( not done_looping ):
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
256
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
257 epoch = epoch + 1
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
258 for x, y in dataset.train(batch_size):
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
259
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
260 minibatch_avg_cost = train_model( x, y )
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
261 # iteration number
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
262 iter += 1
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
263
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
264 if iter % validation_frequency == 0:
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
265 # compute zero-one loss on validation set
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
266 validation_losses = [ validate_model( xv, yv ) for xv, yv in dataset.valid(batch_size) ]
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
267 this_validation_loss = numpy.mean( validation_losses )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
268
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
269 print('epoch %i, iter %i, validation error %f %%' % \
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
270 ( epoch, iter, this_validation_loss*100. ) )
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
271
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
272
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
273 # if we got the best validation score until now
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
274 if this_validation_loss < best_validation_loss:
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
275 #improve patience if loss improvement is good enough
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
276 if this_validation_loss < best_validation_loss * \
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
277 improvement_threshold :
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
278 patience = max( patience, iter * patience_increase )
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
279
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
280 best_validation_loss = this_validation_loss
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
281 # test it on the test set
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
282
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
283 test_losses = [test_model(xt, yt) for xt, yt in dataset.test(batch_size)]
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
284 test_score = numpy.mean(test_losses)
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
285
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
286 print((' epoch %i, iter %i, test error of best '
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
287 'model %f %%') % \
198
5d88ed99c0af Modify the log_reg.py tutorial code to use the datasets module.
Arnaud Bergeron <abergeron@gmail.com>
parents: 169
diff changeset
288 (epoch, iter, test_score*100.))
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
289
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
290 if patience <= iter :
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
291 done_looping = True
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
292 break
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
293
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
294 end_time = time.clock()
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
295 print(('Optimization complete with best validation score of %f %%,'
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
296 'with test performance %f %%') %
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
297 ( best_validation_loss * 100., test_score * 100.))
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
298 print ('The code ran for %f minutes' % ((end_time-start_time) / 60.))
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
299
236
7be1f086a89e added __init__.py to allow module loading of baseline
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 199
diff changeset
300 return best_validation_loss, test_score, iter*batch_size, (end_time-start_time) / 60.
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
301
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
302 if __name__ == '__main__':
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
303 log_reg()
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
304
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
305
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
306 def jobman_log_reg(state, channel):
236
7be1f086a89e added __init__.py to allow module loading of baseline
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 199
diff changeset
307 print state
7be1f086a89e added __init__.py to allow module loading of baseline
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 199
diff changeset
308 (validation_error, test_error, nb_exemples, time) = log_reg( learning_rate = state.learning_rate, \
632
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
309 nb_max_examples = state.nb_max_examples, \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
310 dataset=eval(state.dataset), \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
311 batch_size = state.batch_size,\
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
312 image_size = state.image_size, \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
313 nb_class = state.nb_class, \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
314 patience = state.patience, \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
315 patience_increase = state.patience_increase, \
Yoshua Bengio <bengioy@iro.umontreal.ca>
parents: 271
diff changeset
316 improvement_threshold = state.improvement_threshold )
241
c24020aa38ac fix call to log_reg
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 236
diff changeset
317
c24020aa38ac fix call to log_reg
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 236
diff changeset
318
236
7be1f086a89e added __init__.py to allow module loading of baseline
Myriam Cote <cotemyri@iro.umontreal.ca>
parents: 199
diff changeset
319 print state
158
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
320 state.validation_error = validation_error
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
321 state.test_error = test_error
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
322 state.nb_exemples = nb_exemples
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
323 state.time = time
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
324 return channel.COMPLETE
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
325
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
326
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
327
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
328
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
329
d1bb6e06497a nouveau répertoire régression logistique
Myriam Cote <cotemyri@iro.umontreal.ca>
parents:
diff changeset
330