Mercurial > pylearn

annotate sparse_random_autoassociator/graph.py @ 381:e4473d9697d7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added xent loss
author Joseph Turian <turian@gmail.com>
date Tue, 08 Jul 2008 01:57:33 -0400
parents a1bbcde6b456
children b4efd192d880
rev   line source
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
1 """
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
2 Theano graph for an autoassociator for sparse inputs, which will be trained
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
3 using Ronan Collobert + Jason Weston's sampling trick (2008).
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
4 @todo: Make nearly everything private.
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
5 """
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
6
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
7 from globals import MARGIN
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
8
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
9 from pylearn.nnet_ops import sigmoid, crossentropy_softmax_1hot
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
10 from theano import tensor as t
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
11 from theano.tensor import dot
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
12 xnonzero = t.dvector()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
13 w1nonzero = t.dmatrix()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
14 b1 = t.dvector()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
15 w2nonzero = t.dmatrix()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
16 w2zero = t.dmatrix()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
17 b2nonzero = t.dvector()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
18 b2zero = t.dvector()
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
19 h = sigmoid(dot(xnonzero, w1nonzero) + b1)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
20 ynonzero = sigmoid(dot(h, w2nonzero) + b2nonzero)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
21 yzero = sigmoid(dot(h, w2zero) + b2zero)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
22
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
23 # May want to weight loss wrt nonzero value? e.g. MARGIN violation for
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
24 # 0.1 nonzero is not as bad as MARGIN violation for 0.2 nonzero.
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
25 def hingeloss(MARGIN):
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
26 return -MARGIN * (MARGIN < 0)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
27 nonzeroloss = hingeloss(ynonzero - t.max(yzero) - MARGIN)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
28 zeroloss = hingeloss(-t.max(-(ynonzero)) - yzero - MARGIN)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
29 # xnonzero sensitive loss:
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
30 #nonzeroloss = hingeloss(ynonzero - t.max(yzero) - MARGIN - xnonzero)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
31 #zeroloss = hingeloss(-t.max(-(ynonzero - xnonzero)) - yzero - MARGIN)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
32 loss = t.sum(nonzeroloss) + t.sum(zeroloss)
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
33
381
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
34 def binary_crossentropy(output, target):
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
35 """
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
36 Compute the crossentropy of binary output wrt binary target.
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
37 @note: We do not sum, crossentropy is computed by component.
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
38 @todo: Rewrite as a scalar, and then broadcast to tensor.
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
39 """
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
40 return -(target * t.log(output) + (1 - target) * t.log(1 - output))
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
41 #loss = t.sum(binary_crossentropy(ynonzero, xnonzero)) + t.sum(binary_crossentropy(yzero, t.constant(0)))
e4473d9697d7 Added xent loss
Joseph Turian <turian@gmail.com>
parents: 370
diff changeset
42
370
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
43 (gw1nonzero, gb1, gw2nonzero, gw2zero, gb2nonzero, gb2zero) = t.grad(loss, [w1nonzero, b1, w2nonzero, w2zero, b2nonzero, b2zero])
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
44
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
45 import theano.compile
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
46
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
47 inputs = [xnonzero, w1nonzero, b1, w2nonzero, w2zero, b2nonzero, b2zero]
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
48 outputs = [ynonzero, yzero, loss, gw1nonzero, gb1, gw2nonzero, gw2zero, gb2nonzero, gb2zero]
a1bbcde6b456 Moved sparse_random_autoassociator from my repository
Joseph Turian <turian@gmail.com>
parents:
diff changeset
49 trainfn = theano.compile.function(inputs, outputs)