# HG changeset patch # User Razvan Pascanu # Date 1285256420 14400 # Node ID 32fc5f442dde1eaf006ad22928d41c1f0a8b4fd5 # Parent 23f63ecf0a9ac59665bad94d42f3e4fa3bd62c6a LAYER: sligthly long but somewhat clearer rendering of what I have in mind diff -r 23f63ecf0a9a -r 32fc5f442dde doc/v2_planning/layer_RP.txt --- a/doc/v2_planning/layer_RP.txt Thu Sep 23 11:21:48 2010 -0400 +++ b/doc/v2_planning/layer_RP.txt Thu Sep 23 11:40:20 2010 -0400 @@ -8,9 +8,16 @@ ============= You construct your neural network by constructing a graph of connections - between layers starting from data. While you construct the graph, + between "layers" starting from data. While you construct the graph, different theano formulas are put together to construct your model. + The idea would be that you need to describe exactly what you would draw + on the board if you are asked to draw the architecture. This would be of + course optional ( you will get macros that will return this graph + automatically for a well defined case). Things that are not neural networks, + and you wouldn't have any structure to draw are just a box. For example a + SVM, or PCA. This in case you want to connect their output to your network. + Hard details are not set yet, but all members of the committee agreed that this sound as a good idea. @@ -23,6 +30,7 @@ h1 = sigmoid(dotW_b(train_x, n = 300)) rbm1 = CDk( h1, train_x, k=5, sampler = binomial, cost = pseudolikelihood) + h2 = sigmoid(dotW_b(h1, n = 300)) rbm2 = CDk( h2, h1, k=5, sampler = binomial, cost= pseudolikelihood) @@ -31,7 +39,7 @@ train_err = cross_entropy( out, train_y) grads = grad( train_err, err.parameters() ) - learner = SGD( err, grads) + learner = SGD( err, err.parameters(), grads) valid_err = train_err.replace({ train_x : valid_x, train_y : valid_y}) test_err = train_err.replace({ train_x : test_x , train_y : test_y}) @@ -42,17 +50,25 @@ --------------------- 1) Your graph can have multiple terminal nodes; in this case rbm1, - rbm2 and learner, valid_err, test_err are all end nodes of the graph; + rbm2 and learner, valid_err, test_err are all end nodes of the graph; - 2) Any node is an "iterator", when you would call out.next() you would get - the next prediction; when you call err.next() you will get next error + 2) Any node is an "iterator", when you would call out.next() you would get + the next prediction; when you call err.next() you will get next error ( on the batch given by the data.next() ). - 3) Replace can replace any subgraph + 3) Replace can replace any subgraph or subgraphs with other + subgraphs/subgraph as long as : there are the same number of input units + and output units ( there is a 1 to 1 maping from those). I see replacing + subgraphs as looping over the list of subgraphs to replace and call replace + on which nothing fancier. Since nodes in my view produce the same interface + (execpt parameter nodes and hyper-parameter nodes) this constraint is not + hard to respect, so is up to the user to do a replace that makes sense. 4) You can have MACROS or SUBROUTINE that already give you the graph for known components ( in my view the CDk is such a macro, but simpler - examples will be vanilla versions of MLP, DAA, DBN, LOGREG) + examples will be vanilla versions of MLP, DAA, DBN, LOGREG). After + Guillaume pointed out a real shortcomming of the approach I've modified + a bit what you get from a macro .. look below. 5) Any node has the entire graph ( though arguably you don't use that graph too much). Running such a node in general will be done by compiling @@ -72,8 +88,10 @@ 6) Registering parameters and hyper-parameters to the graph is the job of the transform and therefore of the user who implemented that transform; the same for initializing the parameters ( so if we have - different way to initialize the weight matrix that might be a - hyperparameter with a default value) + different ways to initialize the weight matrix that might be a + hyperparameter with a default value or different transforms; to ease + the number of such transforms you can define a transform on the fly for + simple theano expressions ) @@ -85,6 +103,9 @@ Delayed or feature values ------------------------- + +This is can be dropped if people think is not useful. + Sometimes you might want future values of some nodes. For example you might be interested in : @@ -159,26 +180,32 @@ y = loop( dotW_b(y_tm1,50) + x_t, steps = 20) This would basically give all the information you need to add a scan op -to your theano expression of the result op, it is just a different way +to your theano expression of the result node y, it is just a different way of writing things .. which I think is more intuitive. You create your primitives which are either a recurrent_layer that should -have a initial value, or a slice of some other node ( a time slice that is) -Then you call loop giving a expression that starts from those primitives. +have a initial value, or a slice of some other node ( a time slice that is). +A tims slice is a special kind of node, which we should try to force people +not to use outside of a loop. If you use it though you have some default +behaviour like for example it behaves exactly like a delayed node. +You call loop giving a expression that starts from those primitives and +ta da, you have your recurrent expression in the graph. Similarly you can have foldl or map or anything else. -You would use this instead of writing scan especially if the formula is +You would use this instead of writing scan especially if the formulas are more complicated and you want to automatically collect parameters, -hyper-parameters and so on. +hyper-parameters and so on. You could also just use the scan op and +using a general apply command if you like that more. Optimizer --------- Personally I would respect the findings of the optimization committee, and have the SGD to require a Node that produces some error ( which can - be omitted) and the gradients. For this I would also have the grad - function which would actually only call T.grad. + be omitted) and the parameter nodes and nodes that compute gradients for + those paramters. For this I would also have the grad function which would + actually only call T.grad. If you have non-theano thing in the middle? I don't have any smart solution besides ignoring any parameter that it is below the first @@ -190,7 +217,10 @@ In my case I would not have a predict() and eval() method of the learner, but just a eval(). If you want the predictions you should use the corresponding node ( before applying the error measure ). This was - for example **out** in my first example. + for example **out** in my first example. Note eval() in this case is + the same as next(). ( you might just have next for simplicity). The + only semantically important difference is that a call to next has now + side-effects in the sense that the parameters are updated. Of course we could require learners to be special nodes that also have a predict output. In that case I'm not sure what the iterating behaiour @@ -208,8 +238,10 @@ you would have transforms for the most popular option ( dotW_b) for example. If you need something else you can always decorate a function that takes -theano arguments and produces theano arguments. More then decoratting you -can have a general apply transform that does something like : +theano arguments and produces theano arguments. The formulas produced by +the formula committee might be a rich source of such function to decorate. +More then decoratting, you can have a general apply transform that does +something like : apply( lambda x,y,z: x*y+z, inputs = x, hyperparams = [(name,2)], @@ -217,47 +249,109 @@ The order of the arguments in lambda is nodes, params, hyper-params or so. This would apply the theano expression but it will also register the the parameters. It is like creating a transform on the fly. +You should, or could provide names for parameters, you might need them +later. I think you can do such that the result of the apply is -pickable, but not the apply operation. Meaning that in the graph, the op -doesn't actually store the lambda expression but a mini theano graph. - -Also names might be optional, so you can write hyperparam = [2,] +pickable, but not the general apply transform. What I mean is that +the output node does not store the lambda expression but some theano +graph (?) and it know which are the input ( and when you can replace +them so that you link this little graph to the rest of the +theano expression. Is just an ugly hack given that you can not save +lambda expressions, but I'm open to other alternatives .. What this way of doing things would buy you hopefully is that you do not -need to worry about most of your model ( would be just a few macros or -subrutines). -you would do something like : +need to worry about most of your model ( would be just a few macros) that +will get you to the point you want to change and then you do surgery on +that point. Compare this with hacking a class, it feels cleaner, because +you what is up to that point you want to change is sort of separated from +what you change. Plus you could do this in your script, and you don't need +to create your local branch of the library where you hack the class, or +duplicate the class file under a different name .. +Once what you are doing becomes stable it can be converted in either a +different macro or a parameter to the initial macro. -rbm1,hidden1 = rbm_layer(data,20) -rbm2,hidden2 = rbm_layer(data,20) +** New part ** -and then the part you care about : +If this is not convincing enough, there is another point that I want to +make. While creating the graph you can optionally create a model object. +I will encourage most people to do that ! This idea I had a long time ago, +but then I used a singleton class as the world which could potentially create +a lot of issues. This is a nicer version of that. -hidden3 = apply( lambda x,W: T.dot(x,W), inputs = hidden2, params = -theano.shared(scipy.sparse_CSR(..))) +This model class is optional but it can be extremely useful. What you do in +this model class is to store the graph, together with different annotations +on that graph. What I would do is identify different subgraphs in the model +and register them under different names. For example if err is the node that +points to the graph that represents a DBN, that graph will be registerd to +a model in which I have annotated which subgraphs represent the different +rbms, which represents the logistic regression and so on. The model will also +have a list of all the input nodes and all the output nodes of the graph. +We could potentially use this model class to control some global default +parameters initialization or hyper-parameters. This all might sound like +magic but is actually easy to implement. -and after that you pottentially still do what you did before : +If you have such a model, which is just some annotations on the graph, this +approach makes it easy to change components of the graph based on their names. +For example I can replace rbm1 with a daa, because based on these annotations +I know which part is rbm1. -err = cross_entropy(hidden3, target) -grads = grad(err, err.paramters()) -... +Why do I feel you need such a thing? It is just because you get the DBN by +calling a macro, and you don't have variables that point to different nodes +of your network so that you can define where a subgraph starts or not. But +if a graph returns such a model, you can introspect what annotations you have. +There should also be standard conventions, but you could also in the +interactive shell look at : + +model.annotations(depth = 2) + +This would print something like : -I do agree that some of the "transforms" that I have been writing here -and there are pretty low level, and maybe we don't need them. We might need -only somewhat higher level transforms. My hope is that for now people think -of the approach and not about all inner details ( like what transforms we -need and so on) and see if they are comfortable with it or not. + 'DBN' + 'rbm1' + 'hidden_layer1' + 'CDk_layer1' + 'rbm2' + 'hidden_layer2' + 'CDk_layer2' + 'logreg' + 'cross_entropy' + +And then you can say -Do we want to think in this terms? I think is a bit better do have -a normal python class, hacking it to change something and then either add -a parameter to init or create a new version. It seems a bit more natural. +daa1 = daa(..) +daa2 = daa(..) +new_model = model.replace('rbm1', daa1, new_name = 'daa1') +new_model = new_model.replace('rbm2', daa2, new_name = 'daa2') + +and you get a SDAA. +What is the hierarhical structure ? Well, in my view if some subgrah +(annotated as S1) is part of another subgraph (annotated as S2) then +S1 is a child of S2 in this hierarchy of annotations. If they share +just a few nodes, but have nodes that are not shared, then they are on +the same level. We might one a flat space for the annotations, but I think +this simple convention can get as a lot. - +So macros should in general return such models. It is up to you if you want to +ground the graph that you create in your script into a model or not. You do +so by manually adding nodes to the model. The annotations are also manually +done .. So this might be a bit annoying for a developer of a macro, but I +don't think is cognitively complicated, and it would help a lot when using +the macros. -Anyhow Guillaume I'm working on a better answer :) +You can see how this annotation system becomes easily interesting. You can +also annotate parameters ( and it is not too overwhelming to do so when +you create the graph as well) and you can use this to sort of collect all +parameters that you annotated in some way and then do something to them. + +The way I see it is just that a transform could have an optional annotations +argument and it will add that string to all parameters and hyper-parameters. +How much sense this makes is debatable, but I strongly believe that is not +complicated to implement ( I actually have something like this already +implemented, just that I use that single ton class, and I sort of made the +framework work mostly for DAA by making a few poor choices). Params and hyperparams @@ -267,40 +361,44 @@ around the theano expression. I haven't wrote down all the details of that class. I think there should be such a wrapper around parameters and hyper-parameters as well. By default those wrappers might not provide -any informtion. Later on, they can provide for hyper-params for example a -distribution. If when inserting your hyper-param in the graph ( i.e. when -you call a given transform) you provide the distribution then maybe a -hyperlearner could use it to sample from it. +any informtion. But you can potentially add interesting information for +"graph" aware transforms. For example you can add annotations for a find +or replace function that will collect you all parameters or hyper-parameter +so you do some common thing to all of them (when it makes sense). -For parameters you might define properties like freeze. It can be true or -false. Whenever it is set to true, the param is not adapted by the optimizer. -Changing this value like changing most of hyper-params implies recompilation -of the graph. +You could have a freeze property for parameters. If you change that property +the theano function (where needed) for all nodes that follow this one is +recomputed. This argument would be used by the collecting paramters function +used to compute the gradient. If parameters are frozen they are ignored, +if not they are updated. -I would have a special class of hyper-params which don't require -recompilation of the graph. Learning rate is an example. This info is also -given by the wrapper and by how the parameter is used. +For hyper-parameters you would also have a different wrapper that would +contain, possibly, the distribution of that hyper-parameters for a +hyper-learner. -It is up to the user and "transform" implementer to wrap params and -hyper-params correspondingly. But I don't think this is to complicated. -The apply function above has a default behaviour, maybe you would have -a forth type of argument which is hyper-param that doesn't require -compilation. We could find a nice name for it. - +I would also have the learning rate or noise_amounts as some strange +hyper-paramter. I would say by default, if any hyper-paramter changes its +value, then the theano expressions need to be recompiled. If you are dealing +with this strange types of hyper-parameters you don't need to do that. +This can be automatically for you and I guess it will all boil down to, +is you hyper-paramter a theano shared variable or theano tensor ? If so we +are dealing with the second type. So this kind of stuff can be detected +automatically. How does this work? ------------------- You always have a pointer to the entire graph. Whenever a hyper-param changes ( or a param freezes) all region of the graph affected get recompiled. -This is by traversing the graph from the bottom node and constructing the -theano expression. +This is by traversing the graph from the bottom node and re-constructing the +theano expression. Where needed this theano expression get compiled. This function that updates / re-constructs the graph is sligthly more complex -if you have non-theano functions in the graph .. +if you have non-theano functions in the middle of the graph .. but not too +much in my view. -replace -------- +replace & find +-------------- Replace, replaces a part of the graph. The way it works in my view is that if I write : @@ -311,61 +409,169 @@ You would first copy the graph that is represented by x ( the params or hyper-params are not copied) and then replace the subgraphs. I.e., x will still point to x1+x2+x3, y will point to x1+x5+x3. Replace is not done -inplace. +inplace ! + +I think these Node classes as something light-weighted, like theano variables +and creating copy is not harmful. Also params & shared variables are shared +between these graphs. If you want new params / shared variables we can offer +a copy / deepcopy command. + +Replace (given that it starts from a model) can take string(s) that indicate +specific annotations. + +Find does the same ( without the copying). + + + +If you have two things named the same in the graph you would return the first +one in a breadth search starting from the top node. The idea is that if you +have all the weight matrices annotated as 'W' and you look for 'W' starting +from node hiddens2, you want the W of the second layer, and not of the first. + +I wold support : +model.replace( look_at , search_for , replace_with, annotate_as) +replace(model , look_at , search_for , replace_with, annotate_as) +node.replace(model , look_at, replace_with, annotate_as) -I think these Node classes as something light-weighted, like theano variables. +look_at if it is a node it reffers to the subgraph that has as a final +node that node. I.e. all up to that point. If it is a string, you would look +at the subgraph annotated by that string. + +Of course we can optionally choose not to allow things to be annotate with +the same name, though I sort of liked it. It makes a lot of things easy. For +a DBN I would have the annotations : + +DBN + rbm1 + hidden + CDk + rbm2 + hidden + CDk + logreg + +If I want to change the first CDk with PCD I would do + +pcd1 = PCD (..) +model.replace(look_at='rbm1', search_for='CDk', replace_with=pcd1, + annotate_as='PCD1') + + +Bottom line is : + + I think having a graph and having a way to search in that graph and replace + parts is a very flexible and powerful way of doing things. reconstruct ----------- -This is something nice for DAA. It is definetely not useful for the rest. +This is something nice for DAA. It is definetely not useful for the rest. I think though that is a shame having that transformation graph and not being able to use it to do this. It will make life so much easier when you do deep auto-encoders. I wouldn't put it in the core library, but I would -have in the DAA module. The way I see it you can either have something like +have in the DAA module. For reconstruct to work you need to have inverse +transforms for the ones you use. + +The way I see it you can either have something like # generate your inversable transforms on the fly fn = create_transform(lambda : , params, hyper-params ) inv = create_transform(lambda : , params, hyper-params ) my_transform = couple_transforms( forward = fn, inv = inv) -# have some already widely used such transform in the daa submodule. +and generate special transforms on the fly that have some pseudo-inverses +when you construct the graph. Maybe you can also have spcific pre-defined +transforms for the most used cases, whith specific names. Even more I don't +see the harm of something as simple as dotW_b to have a inverse defined ( as +using tied weights) in all cases, but you would only use it for the DAA. +It just to reduce the number of names of transforms you have, is like a +feature that doesn't hurt or help in 95% of times but it helps in 5% of times. + + +But this is up to debate. The only reason I bring it up is to say that the +class that represents a transform should have a inverse method that by +default throws an exception. transforms ---------- -In my view there will be quite a few of such standard transforms. They -can be grouped by architecture, basic, sampler, optimizer and so on. - -We do not need to provide all of them, just the ones we need. Researching -on an architecture would actually lead in creating new such transforms in -the library. +In my view there will be quite a few of such standard transforms. +This can be annoying, but I think that if we group them by +architectures (MLP, DAA, RBM), sampler, optimizers it will be less of a mess. +This would be crucial for their documentation as well. This categories should +also come with macros. There will be though some basic transforms that +are available at the core ( like replace, find, things related to annotating +and creating a model, collecting parameters and hyper-paramters) -There will be definetely a list of basic such transforms in the begining, -like : - replace, - search, - get_param(name) - get_params(..) - -You can have and should have something like a switch ( that based on a -hyper parameter replaces a part of a graph with another or not). This is -done by re-compiling the graph. +I also think that we can start small by having just very few such transforms +and add them as the library grows. We don't need many of this, most are +nice to have .. Constraints ----------- -Nodes also can also keep track of constraints. +You can always add constraints. I think the easier to make this explicit is to +get a hand on the parameter or ndoe on which you want to add constraint and +do something like -When you write +add_constraint(on_what, what) -y = add_constraint(x, sum(x**2)) +on_what can be a node, a parameter node, a list of nodes, a list of parameter +nodes, an annotation string, given that you provided a model, and what is a +graph. In terms of the graph that you are creating what this does is to +create a dependency link from your main graph to that constraint graph. +This means that the grad function that computes the grad function that +computes the gradients with respect to parameters will also (if there are +such dependency links) add the gradient of those parameters with respect +to the output of that dependency graph. There are some constraints on +what a dependency graph can be, in the sense that it should start from only +one input ( the parameters / node) and it should end in only one node that +is a scalar. -y is the same node as x, just that it also links to this second graph that -computes constraints. Whenever you call grad, grad will also sum to the -cost all attached constraints to the graph. +From an implementation point of view, this can be done by just collecting a +list of constraints cost, that will be added to the cost before calling +T.grad. But I like to think about it in terms of graph linked through +dependency links. + + +Some general comments +--------------------- + + I think that what you get in the end is a very flexible framework, where + adding new things is just a matter of putting together a few transforms and + annotating the entire thing. Worst case scenario you would need to invent a + transform, which I do believe could be quite painless. + + The harder part to implement is the back-bone. It is not difficult in my + view, mostly sligthly tideous. I had something like this implemented in a + matter of a week, though it was a bit less restrictive. I do believe though + that we should not oversimplify the backbone of the library just to make it + easy to implement, but we should rather carefully consider what you get in + the end + + +Connection to the architecture committee +----------------------------------------- + + I think that if you get such iterator objects that can produce either + the error, or do an update step it is easy to wrap them in a plug-in, + or use it with the imperative language James proposed. + + I actually have ideas ( using non theano nodes) how to break the algo at + points such that you can have different parts run on remote machines .. + though we might not want to support that ( using the plug-in system .. + though it might work with other systems that support the same idea) + + I think it goes more natural with the imperative language that James + proposed, because that would create a graph as well. His graph is + in general simpler ( it always has only one termination node) where + the nodes have a different interpretation (?) so I would use a different + node class on those. But from writing the code, using some syntactic sugar + the difference can be blurred ( do we want this ?). I think that one + can come up with ways of making the approaches look alike and sligtly + homogeneous.