Mercurial > pylearn

annotate pylearn/formulas/costs.py @ 1399:d4a35c1c0a23

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
adding MultiHingeLoss cost function
author James Bergstra <bergstrj@iro.umontreal.ca>
date Thu, 13 Jan 2011 17:52:49 -0500
parents 63fe96ede21d
children f9066ff6e198
rev   line source
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
1 """
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
2
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
3 TODO: make sur stabilization optimization are done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
4 TODO: make test
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
5 TODO: check that this work for nd tensor.
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
6 """
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
7
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
8 #"""
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
9 #Common training criteria.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
10 #"""
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
11 import theano
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
12 import theano.tensor as T
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
13
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
14 from tags import tags
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
15
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
16 __authors__ = "Frederic Bastien, Nicolas Boulanger-Lewandowski, .."
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
17 __copyright__ = "(c) 2010, Universite de Montreal"
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
18 __license__ = "3-clause BSD License"
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
19 __contact__ = "theano-user <theano-users@googlegroups.com>"
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
20
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
21 @tags('cost','binary','cross-entropy')
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
22 def binary_crossentropy(output, target):
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
23 """ Compute the crossentropy of binary output wrt binary target.
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
24
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
25 .. math::
1295
7af2a89bff3d Change tensor.log -> T.log since tensor was undefined symbol in this scope.
David Warde-Farley <wardefar@iro.umontreal.ca>
parents: 1122
diff changeset
26 L_{CE} \equiv t\log(o) + (1-t)\log(1-o)
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
27
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
28 :type output: Theano variable
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
29 :param output: Binary output or prediction :math:`\in[0,1]`
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
30 :type target: Theano variable
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
31 :param target: Binary target usually :math:`\in\{0,1\}`
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
32
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
33 :note: no stabilization optimization needed for a generic output variable
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
34 """
1295
7af2a89bff3d Change tensor.log -> T.log since tensor was undefined symbol in this scope.
David Warde-Farley <wardefar@iro.umontreal.ca>
parents: 1122
diff changeset
35 return -(target * T.log(output) + (1.0 - target) * T.log(1.0 - output))
1122
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
36
be53f56b37b8 Added example cost formula with tags and created a formulas page in the doc
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
37
1328
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
38 @tags('cost','binary','cross-entropy', 'sigmoid')
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
39 def sigmoid_crossentropy(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
40 """ crossentropy of a sigmoid activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
41
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
42 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
43 L_{CE} \equiv t\log(\sigma(a)) + (1-t)\log(1-\sigma(a))
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
44
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
45 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
46 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
47 :type target: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
48 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
49
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
50 :note: no stabilization done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
51 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
52 return target * (- T.log(1.0 + T.exp(-output))) + (1.0 - target) * (- T.log(1.0 + T.exp(output)))
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
53
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
54 @tags('cost','binary','cross-entropy', 'tanh')
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
55 def tanh_crossentropy(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
56 """ crossentropy of a tanh activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
57
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
58 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
59 L_{CE} \equiv t\log(\\frac{1+\\tanh(a)}2) + (1-t)\log(\\frac{1-\\tanh(a)}2)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
60
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
61 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
62 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
63 :type target: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
64 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
65
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
66 :note: no stabilization done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
67 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
68 return sigmoid_crossentropy(2.0*output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
69
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
70 @tags('cost','binary','cross-entropy', 'tanh', 'abs')
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
71 def abstanh_crossentropy(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
72 """ crossentropy of a absolute value tanh activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
73
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
74 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
75 L_{CE} \equiv t\log(\\frac{1+\\tanh(|a|)}2) + (1-t)\log(\\frac{1-\\tanh(|a|)}2)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
76
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
77 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
78 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
79 :type target: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
80 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
81
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
82 :note: no stabilization done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
83 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
84 return tanh_crossentropy(T.abs_(output), target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
85
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
86 @tags('cost','binary','cross-entropy', 'tanh', 'normalized')
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
87 def normtanh_crossentropy(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
88 """ crossentropy of a "normalized" tanh activation (LeCun)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
89
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
90 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
91 L_{CE} \equiv t\log(\\frac{1+\\tanh(0.6666a)}2) + (1-t)\log(\\frac{1-\\tanh(0.6666a)}2)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
92
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
93 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
94 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
95 :type target: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
96 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
97
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
98 :note: no stabilization done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
99 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
100 return tanh_crossentropy(0.6666*output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
101
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
102 @tags('cost','binary','cross-entropy', 'tanh', 'normalized', 'abs')
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
103 def absnormtanh_cross_entropy(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
104 """ crossentropy of a "absolute normalized" tanh activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
105
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
106 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
107 L_{CE} \equiv t\log(\\frac{1+\\tanh(0.6666*|a|)}2) + (1-t)\log(\\frac{1-\\tanh(0.6666*|a|)}2)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
108
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
109 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
110 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
111 :type target: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
112 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
113
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
114 :note: no stabilization done.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
115 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
116 return normtanh_crossentropy(T.abs_(output), target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
117
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
118 def cross_entropy(output_act, output, target, act=None):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
119 """ Execute the cross entropy with a sum on the last dimension and a mean on the first dimension.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
120
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
121 If act is in 'sigmoid', 'tanh', 'tanhnorm', 'abstanh', 'abstanhnorm' we
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
122 call the specialized version.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
123
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
124 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
125 mean(sum(sqr(output-target),axis=-1),axis=0)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
126
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
127 :type output_act: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
128 :param output_act: Output after activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
129 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
130 :param output: Output before activation
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
131 :type target:Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
132 :param target: Target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
133 :type act: str or None
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
134 :param act: The type of activation used
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
135 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
136 if act in ['sigmoid','tanh','tanhnorm','abstanh','abstanhnorm']:
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
137 if act == 'sigmoid':
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
138 return sigmoid_crossentropy(output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
139 if act == 'tanh':
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
140 return tanh_crossentropy(output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
141 if act == 'tanhnorm':
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
142 return normtanh_crossentropy(output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
143 if act == 'abstanh':
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
144 return abstanh_crossentropy(output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
145 if act == 'abstanhnorm':
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
146 return absnormtanh_cross_entropy(output, target)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
147 elif act is None:
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
148 XE = target * T.log(output_act) + (1 - target) * T.log(1 - output_act)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
149 return -T.mean(T.sum(XE, axis=-1),axis=0)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
150 else:
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
151 raise Exception("cross_entropy() Expected parameter act to be in ['sigmoid','tanh','tanhnorm','abstanh','abstanhnorm', None]")
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
152
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
153 def quadratic_cost(output, target):
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
154 """ The quadratic cost of output again target with a sum on the last dimension and a mean on the first dimension.
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
155
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
156 .. math::
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
157 mean(sum(sqr(output-target),axis=-1),axis=0)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
158
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
159 :type output: Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
160 :param output: The value that we want to compare again target
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
161 :type target:Theano variable
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
162 :param target: The value that we consider correct
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
163 """
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
164 return T.mean(T.sum(T.sqr(output - target), axis=-1),axis=0)
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
165
63fe96ede21d added function doc and made the interface more constant between fct.
Frederic Bastien <nouiz@nouiz.org>
parents: 1307
diff changeset
166
1297
24890ca1d96b small changes to formulas to bring them closer to style recommendation
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1295
diff changeset
167 # This file seems like it has some overlap with theano.tensor.nnet. Which functions should go
24890ca1d96b small changes to formulas to bring them closer to style recommendation
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1295
diff changeset
168 # in which file?
1307
bc41fd23db25 Ported 4 cost formulas from XG github repository
boulanni <nicolas_boulanger@hotmail.com>
parents: 1297
diff changeset
169
1399
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
170 from theano import gof
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
171 from theano.tensor.tsor_apply import Apply
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
172 from theano import tensor
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
173 import numpy as np
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
174
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
175 class MultiHingeMargin(gof.Op):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
176 """
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
177 This is a hinge loss function for multiclass predictions.
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
178
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
179 For each vector X[i] and label index yidx[i],
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
180 output z[i] = 1 - margin
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
181
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
182 where margin is the difference between X[i, yidx[i]] and the maximum other element of X[i].
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
183 """
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
184 default_output = 0
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
185 def __eq__(self, other):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
186 return type(self) == type(other)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
187 def __hash__(self):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
188 return tensor.hashtype(self)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
189 def __str__(self):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
190 return self.__class__.__name__
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
191 def make_node(self, X, yidx):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
192 X_ = tensor.as_tensor_variable(X)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
193 yidx_ = tensor.as_tensor_variable(yidx)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
194 if X_.type.ndim != 2:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
195 raise TypeError('X must be matrix')
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
196 if yidx.type.ndim != 1:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
197 raise TypeError('yidx must be vector')
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
198 if 'int' not in str(yidx.type.dtype):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
199 raise TypeError("yidx must be integers, it's a vector of class labels")
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
200 hinge_loss = tensor.vector(dtype=X.dtype)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
201 winners = X.type()
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
202 return Apply(self, [X_, yidx_], [hinge_loss, winners])
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
203 def perform(self, node, input_storage, out):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
204 X, yidx = input_storage
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
205 toplabel = X.shape[1]-1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
206 out[0][0] = z = np.zeros_like(X[:,0])
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
207 out[1][0] = w = np.zeros_like(X)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
208 for i,Xi in enumerate(X):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
209 yi = yidx[i]
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
210 if yi == 0:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
211 next_best = Xi[1:].argmax()+1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
212 elif yi==toplabel:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
213 next_best = Xi[:toplabel].argmax()
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
214 else:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
215 next_best0 = Xi[:yi].argmax()
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
216 next_best1 = Xi[yi+1:].argmax()+yi+1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
217 next_best = next_best0 if Xi[next_best0]>Xi[next_best1] else next_best1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
218 margin = Xi[yi] - Xi[next_best]
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
219 if margin < 1:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
220 z[i] = 1 - margin
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
221 w[i,yi] = -1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
222 w[i,next_best] = 1
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
223 def grad(self, inputs, g_outs):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
224 z = self(*inputs)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
225 w = z.owner.outputs[1]
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
226 gz, gw = g_outs
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
227 if gw is not None:
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
228 raise NotImplementedError()
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
229 gX = gz.dimshuffle(0,'x') * w
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
230 return [gX, None]
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
231 def c_code_cache_version(self):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
232 return (1,)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
233 def c_code(self, node, name, (X, y_idx), (z,w), sub):
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
234 return '''
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
235 if ((%(X)s->descr->type_num != PyArray_DOUBLE) && (%(X)s->descr->type_num != PyArray_FLOAT))
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
236 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
237 PyErr_SetString(PyExc_TypeError, "types should be float or float64");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
238 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
239 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
240 if ((%(y_idx)s->descr->type_num != PyArray_INT64)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
241 && (%(y_idx)s->descr->type_num != PyArray_INT32)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
242 && (%(y_idx)s->descr->type_num != PyArray_INT16)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
243 && (%(y_idx)s->descr->type_num != PyArray_INT8))
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
244 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
245 PyErr_SetString(PyExc_TypeError, "y_idx not int8, int16, int32, or int64");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
246 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
247 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
248 if ((%(X)s->nd != 2)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
249 || (%(y_idx)s->nd != 1))
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
250 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
251 PyErr_SetString(PyExc_ValueError, "rank error");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
252 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
253 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
254 if (%(X)s->dimensions[0] != %(y_idx)s->dimensions[0])
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
255 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
256 PyErr_SetString(PyExc_ValueError, "dy.shape[0] != sm.shape[0]");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
257 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
258 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
259 if ((NULL == %(z)s)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
260 || (%(z)s->dimensions[0] != %(X)s->dimensions[0]))
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
261 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
262 Py_XDECREF(%(z)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
263 %(z)s = (PyArrayObject*) PyArray_SimpleNew(1, PyArray_DIMS(%(X)s),
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
264 type_num_%(X)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
265 if (!%(z)s)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
266 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
267 PyErr_SetString(PyExc_MemoryError, "failed to alloc dx output");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
268 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
269 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
270 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
271 if ((NULL == %(w)s)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
272 || (%(w)s->dimensions[0] != %(X)s->dimensions[0])
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
273 || (%(w)s->dimensions[1] != %(X)s->dimensions[1]))
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
274 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
275 Py_XDECREF(%(w)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
276 %(w)s = (PyArrayObject*) PyArray_SimpleNew(2, PyArray_DIMS(%(X)s),
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
277 type_num_%(X)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
278 if (!%(w)s)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
279 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
280 PyErr_SetString(PyExc_MemoryError, "failed to alloc dx output");
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
281 %(fail)s;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
282 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
283 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
284
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
285 for (size_t i = 0; i < %(X)s->dimensions[0]; ++i)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
286 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
287 const dtype_%(X)s* __restrict__ X_i = (dtype_%(X)s*) (%(X)s->data + %(X)s->strides[0] * i);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
288 npy_intp SX = %(X)s->strides[1]/sizeof(dtype_%(X)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
289
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
290 dtype_%(w)s* __restrict__ w_i = (dtype_%(w)s*) (%(w)s->data + %(w)s->strides[0] * i);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
291 npy_intp Sw = %(w)s->strides[1]/sizeof(dtype_%(w)s);
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
292
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
293 const dtype_%(y_idx)s y_i = ((dtype_%(y_idx)s*)(%(y_idx)s->data + %(y_idx)s->strides[0] * i))[0];
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
294
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
295 dtype_%(X)s X_i_max = X_i[0];
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
296 dtype_%(X)s X_at_y_i = X_i[0];
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
297 size_t X_i_argmax = 0;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
298 size_t j = 1;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
299 w_i[0] = 0;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
300
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
301 if (y_i == 0)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
302 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
303 X_i_max = X_i[SX];
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
304 X_i_argmax = 1;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
305 w_i[Sw] = 0;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
306 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
307 for (; j < %(X)s->dimensions[1]; ++j)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
308 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
309 dtype_%(X)s X_ij = X_i[j*SX];
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
310 if (j == y_i)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
311 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
312 X_at_y_i = X_ij;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
313 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
314 else if (X_ij > X_i_max)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
315 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
316 X_i_max = X_ij;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
317 X_i_argmax = j;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
318 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
319 w_i[j*Sw] = 0;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
320 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
321 if (0 < 1 - X_at_y_i + X_i_max)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
322 {
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
323 ((dtype_%(z)s*)(%(z)s->data + %(z)s->strides[0] * i))[0]
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
324 = 1 - X_at_y_i + X_i_max;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
325 w_i[y_i*Sw] = -1;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
326 w_i[X_i_argmax*Sw] = 1;
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
327 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
328 }
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
329 ''' % dict(locals(), **sub)
d4a35c1c0a23 adding MultiHingeLoss cost function
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1328
diff changeset
330 multi_hinge_margin = MultiHingeMargin()