pylearn: pylearn/algorithms/rnn.py annotate

annotate pylearn/algorithms/rnn.py @ 574:220044be9fd8

added test for a bug that James reported to me.

author	Frederic Bastien <bastienf@iro.umontreal.ca>
date	Thu, 04 Dec 2008 10:56:44 -0500
parents	96221aa02fcb
children	1972bc9bea6d

rev	line source
550 b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	1
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	2 import numpy as N
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	3 from theano import Op, Apply, tensor as T, Module, Member, Method, Mode, compile
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	4 from theano.gof import OpSub, TopoOptimizer
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	5
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	6 from minimizer import make_minimizer # minimizer
550 b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	7 from theano.printing import Print
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	8 import sgd #until Olivier's module-import thing works better
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	9
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	10 ####################
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	11 # Library-type stuff
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	12 ####################
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	13
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	14 class TanhRnn(Op):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	15 """
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	16 This class implements the recurrent part of a recurrent neural network.
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	17
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	18 There is not a neat way to include this in a more fine-grained way in Theano at the moment,
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	19 so to get something working, I'm implementing a relatively complicated Op that could be
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	20 broken down later into constituents.
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	21
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	22 Anyway, this Op implements recursive computation of the form:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	23
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	24 .. latex-eqn:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	25 z_t &= \tanh( z_{t-1} A + x_{t-1})
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	26
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	27 For z0 a vector, and x a TxM matrix, it returns a matrix z of shape (T+1, M),
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	28 in which z[0] = z0.
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	29
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	30 """
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	31
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	32 def make_node(self, x, z0, A):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	33 """
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	34 :type x: matrix (each row is an x_t) (shape: (T, M))
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	35 :type z0: vector (the first row of output) (shape: M)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	36 :type A: matrix (M by M)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	37
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	38 """
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	39 x = T.as_tensor(x)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	40 z0 = T.as_tensor(z0)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	41 A = T.as_tensor(A)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	42 z = x.type() #make a new symbolic result with the same type as x
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	43 return Apply(self, [x, z0, A], [z])
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	44
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	45 def perform(self, node, (x,z0,A), out):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	46 T,M = x.shape
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	47 z = N.zeros((T+1, M))
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	48 z[0] = z0
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	49 for i in xrange(T):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	50 z[i+1] = N.tanh(N.dot(z[i], A) + x[i])
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	51 out[0][0] = z
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	52
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	53 def grad(self, (x, z0, A), (gz,)):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	54 z = tanh_rnn(x, z0, A)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	55 gz_incl_rnn, gx = tanh_rnn_grad(A, z, gz)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	56 return [gx, gz_incl_rnn[0], (T.dot(z[:-1].T, gx))]
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	57 tanh_rnn = TanhRnn()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	58
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	59 class TanhRnnGrad(Op):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	60 """Gradient calculation for TanhRnn"""
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	61
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	62 def __init__(self, inplace):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	63 self.inplace = inplace
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	64
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	65 if self.inplace:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	66 self.destroy_map = {0: [2]}
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	67
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	68 def __eq__(self, other):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	69 return (type(self) == type(other)) and (self.inplace == other.inplace)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	70
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	71 def __hash__(self, other):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	72 return hash(type(self)) ^ hash(self.inplace)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	73
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	74 def make_node(self, A, z, gz):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	75 return Apply(self, [A,z,gz], (z.type(), gz.type()))
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	76
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	77 def perform(self, node, (A, z, gz), out):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	78 Tp1,M = z.shape
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	79 T = Tp1 - 1
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	80 gx = N.zeros((T, M))
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	81
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	82 if not self.inplace:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	83 gz = gz.copy()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	84
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	85 for i in xrange(T-1, -1, -1):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	86 #back through the tanh
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	87 gx[i] = gz[i+1] * (1.0 - z[i+1] * z[i+1])
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	88 gz[i] += N.dot(A, gx[i])
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	89
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	90 out[0][0] = gz
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	91 out[1][0] = gx
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	92
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	93 def __str__(self):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	94 if self.inplace:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	95 return 'Inplace' + super(TanhRnnGrad, self).__str__()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	96 else:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	97 return super(TanhRnnGrad, self).__str__()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	98
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	99 tanh_rnn_grad = TanhRnnGrad(inplace=False)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	100 tanh_rnn_grad_inplace = TanhRnnGrad(inplace=True)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	101
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	102 compile.optdb.register('inplace_rnngrad', TopoOptimizer(OpSub(tanh_rnn_grad, tanh_rnn_grad_inplace)), 60, 'fast_run', 'inplace')
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	103
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	104
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	105 #######################
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	106 # Experiment-type stuff
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	107 #######################
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	108
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	109
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	110
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	111 class ExampleRNN(Module):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	112
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	113 def __init__(self, n_vis, n_hid, n_out, minimizer):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	114 super(ExampleRNN, self).__init__()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	115
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	116 def affine(weight, bias):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	117 return (lambda a : T.dot(a, weight) + bias)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	118
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	119 self.n_vis = n_vis
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	120 self.n_hid = n_hid
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	121 self.n_out = n_out
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	122
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	123 #affine transformatoin x -> latent space
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	124 self.v, self.b = Member(T.dmatrix()), Member(T.dvector())
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	125 input_transform = affine(self.v, self.b)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	126
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	127 #recurrent weight matrix in latent space
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	128 self.z0 = Member(T.dvector())
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	129 self.w = Member(T.dmatrix())
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	130
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	131 #affine transformation latent -> output space
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	132 self.u, self.c = Member(T.dmatrix()), Member(T.dvector())
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	133 output_transform = affine(self.u, self.c)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	134
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	135 self.params = [self.v, self.b, self.w, self.u, self.c]
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	136
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	137 #input and target
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	138 x, y = T.dmatrix(), T.dmatrix()
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	139
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	140 z = tanh_rnn(input_transform(x), self.z0, self.w)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	141 yhat = output_transform(z[1:])
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	142 self.cost = T.sum((y - yhat)**2)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	143
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	144 self.blah = Method([x,y], self.cost)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	145
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	146 # using the make_minimizer protocol
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	147 self.minimizer = minimizer([x, y], self.cost, self.params)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	148
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	149 def _instance_initialize(self, obj):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	150 n_vis = self.n_vis
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	151 n_hid = self.n_hid
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	152 n_out = self.n_out
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	153
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	154 rng = N.random.RandomState(2342)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	155
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	156 obj.z0 = N.zeros(n_hid)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	157 obj.v = rng.randn(n_vis, n_hid) * 0.01
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	158 obj.b = N.zeros(n_hid)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	159 obj.w = rng.randn(n_hid, n_hid) * 0.01
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	160 obj.u = rng.randn(n_hid, n_out) * 0.01
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	161 obj.c = N.zeros(n_out)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	162 obj.minimizer.initialize()
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	163 def _instance__eq__(self, other):
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	164 if not isinstance(other.component, ExampleRNN):
83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	165 raise NotImplemented
83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	166 #we compare the member.
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	167 # if self.n_vis != other.n_vis or slef.n_hid != other.n_hid or self.n_out != other.n_out:
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	168 # return False
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	169 if (N.abs(self.z0-other.z0)<1e-8).all() and (N.abs(self.v-other.v)<1e-8).all() and (N.abs(self.b-other.b)<1e-8).all() and (N.abs(self.w-other.w)<1e-8).all() and (N.abs(self.u-other.u)<1e-8).all() and (N.abs(self.c-other.c)<1e-8).all() and (N.abs(self.z0-other.z0)<1e-8).all():
83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	170 return True
83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	171 return False
550 b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	172
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	173 def _instance__hash__(self):
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	174 raise NotImplemented
550 b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	175
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	176 def test_example_rnn():
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	177 minimizer_fn = make_minimizer('sgd', stepsize = 0.001)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	178
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	179 n_vis = 5
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	180 n_out = 3
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	181 n_hid = 4
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	182 rnn_module = ExampleRNN(n_vis, n_hid, n_out, minimizer_fn)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	183
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	184 rnn = rnn_module.make(mode='FAST_RUN')
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	185
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	186 rng = N.random.RandomState(7722342)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	187 x = rng.randn(10,n_vis)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	188 y = rng.randn(10,n_out)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	189
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	190 #set y to be like x with a lag of LAG
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	191 LAG = 4
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	192 y[LAG:] = x[:-LAG, 0:n_out]
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	193
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	194 if 1:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	195 for i, node in enumerate(rnn.minimizer.step_cost.maker.env.toposort()):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	196 print i, node
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	197
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	198 niter=1500
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	199 for i in xrange(niter):
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	200 if i % 100 == 0:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	201 print i, rnn.minimizer.step_cost(x, y), rnn.minimizer.stepsize
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	202 else:
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	203 rnn.minimizer.step_cost(x, y)
b52c1a8811a6 init rnn.py James Bergstra <bergstrj@iro.umontreal.ca> parents: diff changeset	204
560 96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	205 def test_WEIRD_STUFF():
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	206 n_vis = 5
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	207 n_out = 3
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	208 n_hid = 4
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	209 rng = N.random.RandomState(7722342)
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	210 x = rng.randn(10,n_vis)
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	211 y = rng.randn(10,n_out)
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	212
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	213 #set y to be like x with a lag of LAG
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	214 LAG = 4
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	215 y[LAG:] = x[:-LAG, 0:n_out]
96221aa02fcb put the new test in a different test fct. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 559 diff changeset	216
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	217 minimizer_fn1 = make_minimizer('sgd', stepsize = 0.001, WEIRD_STUFF = False)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	218 minimizer_fn2 = make_minimizer('sgd', stepsize = 0.001, WEIRD_STUFF = True)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	219 rnn_module1 = ExampleRNN(n_vis, n_hid, n_out, minimizer_fn1)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	220 rnn_module2 = ExampleRNN(n_vis, n_hid, n_out, minimizer_fn2)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	221 rnn1 = rnn_module2.make(mode='FAST_RUN')
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	222 rnn2 = rnn_module1.make(mode='FAST_COMPILE')
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	223 topo1=rnn1.minimizer.step_cost.maker.env.toposort()
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	224 topo2=rnn2.minimizer.step_cost.maker.env.toposort()
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	225 if 0:
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	226 for i in range(len(topo1)):
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	227 print '1',i, topo1[i]
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	228 print '2',i, topo2[i]
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	229
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	230
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	231
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	232 niter=3
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	233 for i in xrange(niter):
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	234 rnn1.minimizer.step(x, y)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	235 rnn2.minimizer.step(x, y)
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	236
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	237 # assert rnn1.n_vis != rnn2.n_vis or slef.n_hid != rnn2.n_hid or rnn1.n_out != rnn2.n_out
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	238 assert (N.abs(rnn1.z0-rnn2.z0)<1e-8).all()
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	239 assert (N.abs(rnn1.v-rnn2.v)<1e-8).all() and (N.abs(rnn1.b-rnn2.b)<1e-8).all() and (N.abs(rnn1.w-rnn2.w)<1e-8).all() and (N.abs(rnn1.u-rnn2.u)<1e-8).all() and (N.abs(rnn1.c-rnn2.c)<1e-8).all()
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	240
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	241 # assert b
559 83ebb313b2f1 added a test for the WEIRD_STUFF flag in theano ticket 239 Frederic Bastien <bastienf@iro.umontreal.ca> parents: 550 diff changeset	242
574 220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	243 if __name__ == '__main__':
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	244 # from theano.tests import main
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	245 # main(__file__)
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	246 # test_example_rnn()
220044be9fd8 added test for a bug that James reported to me. Frederic Bastien <bastienf@iro.umontreal.ca> parents: 560 diff changeset	247 test_WEIRD_STUFF()

Mercurial > pylearn

annotate pylearn/algorithms/rnn.py @ 574:220044be9fd8