Mercurial > pylearn
comparison doc/v2_planning/API_formulas.txt @ 1171:fab72f424ee0
Merged
| author | Olivier Delalleau <delallea@iro> |
|---|---|
| date | Fri, 17 Sep 2010 14:37:08 -0400 |
| parents | 42ddbefd1e03 |
| children |
comparison
equal
deleted
inserted
replaced
| 1170:53340a8df1fa | 1171:fab72f424ee0 |
|---|---|
| 1 .. _v2planning_formulas: | |
| 2 | |
| 3 Math formulas API | |
| 4 ================= | |
| 5 | |
| 6 Why we need a formulas API | |
| 7 -------------------------- | |
| 8 | |
| 9 Their is a few reasons why having a library of mathematical formula for theano is a good reason: | |
| 10 | |
| 11 * Some formula have some special thing needed for the gpu. | |
| 12 * Sometimes we need to cast to floatX... | |
| 13 * Some formula have numerical stability problem. | |
| 14 * Some formula gradiant have numerical stability problem. (Happen more frequently then the previous ones) | |
| 15 * If theano don't always do some stability optimization, we could do it manually in the formulas | |
| 16 * Some formula as complex to implement and take many try to do correctly. | |
| 17 * Can mimic the hierarchy of other library to ease the migration to theano | |
| 18 | |
| 19 Having a library help in that we solve those problem only once. | |
| 20 | |
| 21 What is a formula | |
| 22 ----------------- | |
| 23 | |
| 24 We define formulas as something that don't have a state. They are implemented as | |
| 25 python function that take theano variable as input and they output theano | |
| 26 variable. If you want state, look at what the others commities will do. | |
| 27 | |
| 28 Formulas documentation | |
| 29 ---------------------- | |
| 30 | |
| 31 We must respect what the coding commitee have set for the docstring of the file and of the function. | |
| 32 | |
| 33 * A latex mathematical description of the formulas(for picture representation in generated documentation) | |
| 34 * Tags(for searching): | |
| 35 * a list of lower level fct used | |
| 36 * category(name of the submodule itself) | |
| 37 * Tell if we did some work to make it more numerical stable. Do theano do the optimization needed? | |
| 38 * Tell if the grad is numericaly stable? Do theano do the optimization needed? | |
| 39 * Tell if work/don't/unknow on gpu. | |
| 40 * Tell alternate name | |
| 41 * Tell the domaine, range of the input/output(range should use the english notation of including or excluding) | |
| 42 | |
| 43 Proposed hierarchy | |
| 44 ------------------ | |
| 45 | |
| 46 Here is the proposed hierarchy for formulas: | |
| 47 | |
| 48 * pylearn.formulas.costs: generic / common cost functions, e.g. various cross-entropies, squared error, | |
| 49 abs. error, various sparsity penalties (L1, Student) | |
| 50 * pylearn.formulas.regularization: formulas for regularization | |
| 51 * pylearn.formulas.linear: formulas for linear classifier, linear regression, factor analysis, PCA | |
| 52 * pylearn.formulas.nnet: formulas for building layers of various kinds, various activation functions, | |
| 53 layers which could be plugged with various costs & penalties, and stacked | |
| 54 * pylearn.formulas.ae: formulas for auto-encoders and denoising auto-encoder variants | |
| 55 * pylearn.formulas.noise: formulas for corruption processes | |
| 56 * pylearn.formulas.rbm: energies, free energies, conditional distributions, Gibbs sampling | |
| 57 * pylearn.formulas.trees: formulas for decision trees | |
| 58 * pylearn.formulas.boosting: formulas for boosting variants | |
| 59 * pylearn.formulas.maths for other math formulas | |
| 60 * pylearn.formulas.scipy.stats: example to implement the same interface as existing lib | |
| 61 | |
| 62 etc. | |
| 63 | |
| 64 Example | |
| 65 ------- | |
| 66 .. code-block:: python | |
| 67 | |
| 68 """ | |
| 69 This script defines a few often used cost functions. | |
| 70 """ | |
| 71 import theano | |
| 72 import theano.tensor as T | |
| 73 from tags import tags | |
| 74 | |
| 75 @tags('cost','binary','cross-entropy') | |
| 76 def binary_crossentropy(output, target): | |
| 77 """ Compute the crossentropy of binary output wrt binary target. | |
| 78 | |
| 79 .. math:: | |
| 80 L_{CE} \equiv t\log(o) + (1-t)\log(1-o) | |
| 81 | |
| 82 :type output: Theano variable | |
| 83 :param output: Binary output or prediction :math:`\in[0,1]` | |
| 84 :type target: Theano variable | |
| 85 :param target: Binary target usually :math:`\in\{0,1\}` | |
| 86 """ | |
| 87 return -(target * tensor.log(output) + (1.0 - target) * tensor.log(1.0 - output)) | |
| 88 | |
| 89 | |
| 90 TODO | |
| 91 ---- | |
| 92 * define a list of search tag to start with | |
| 93 * Add to the html page a list of the tag and a list of each fct associated to them. | |
| 94 * move existing formulas to pylearn as examples and add other basics ones. | |
| 95 * theano.tensor.nnet will probably be copied to pylearn.formulas.nnet and depricated. | |
| 96 |