Mercurial > pylearn

comparison doc/v2_planning/API_formulas.txt @ 1165:42ddbefd1e03

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
made the API_formulas.txt and removed duplicate stuff from the formulas.txt file
author Frederic Bastien <nouiz@nouiz.org>
date Fri, 17 Sep 2010 13:57:07 -0400
parents
children
comparison
equal deleted inserted replaced
1164:53d11eafdaa9 1165:42ddbefd1e03
1 .. _v2planning_formulas:
2
3 Math formulas API
4 =================
5
6 Why we need a formulas API
7 --------------------------
8
9 Their is a few reasons why having a library of mathematical formula for theano is a good reason:
10
11 * Some formula have some special thing needed for the gpu.
12 * Sometimes we need to cast to floatX...
13 * Some formula have numerical stability problem.
14 * Some formula gradiant have numerical stability problem. (Happen more frequently then the previous ones)
15 * If theano don't always do some stability optimization, we could do it manually in the formulas
16 * Some formula as complex to implement and take many try to do correctly.
17 * Can mimic the hierarchy of other library to ease the migration to theano
18
19 Having a library help in that we solve those problem only once.
20
21 What is a formula
22 -----------------
23
24 We define formulas as something that don't have a state. They are implemented as
25 python function that take theano variable as input and they output theano
26 variable. If you want state, look at what the others commities will do.
27
28 Formulas documentation
29 ----------------------
30
31 We must respect what the coding commitee have set for the docstring of the file and of the function.
32
33 * A latex mathematical description of the formulas(for picture representation in generated documentation)
34 * Tags(for searching):
35 * a list of lower level fct used
36 * category(name of the submodule itself)
37 * Tell if we did some work to make it more numerical stable. Do theano do the optimization needed?
38 * Tell if the grad is numericaly stable? Do theano do the optimization needed?
39 * Tell if work/don't/unknow on gpu.
40 * Tell alternate name
41 * Tell the domaine, range of the input/output(range should use the english notation of including or excluding)
42
43 Proposed hierarchy
44 ------------------
45
46 Here is the proposed hierarchy for formulas:
47
48 * pylearn.formulas.costs: generic / common cost functions, e.g. various cross-entropies, squared error,
49 abs. error, various sparsity penalties (L1, Student)
50 * pylearn.formulas.regularization: formulas for regularization
51 * pylearn.formulas.linear: formulas for linear classifier, linear regression, factor analysis, PCA
52 * pylearn.formulas.nnet: formulas for building layers of various kinds, various activation functions,
53 layers which could be plugged with various costs & penalties, and stacked
54 * pylearn.formulas.ae: formulas for auto-encoders and denoising auto-encoder variants
55 * pylearn.formulas.noise: formulas for corruption processes
56 * pylearn.formulas.rbm: energies, free energies, conditional distributions, Gibbs sampling
57 * pylearn.formulas.trees: formulas for decision trees
58 * pylearn.formulas.boosting: formulas for boosting variants
59 * pylearn.formulas.maths for other math formulas
60 * pylearn.formulas.scipy.stats: example to implement the same interface as existing lib
61
62 etc.
63
64 Example
65 -------
66 .. code-block:: python
67
68 """
69 This script defines a few often used cost functions.
70 """
71 import theano
72 import theano.tensor as T
73 from tags import tags
74
75 @tags('cost','binary','cross-entropy')
76 def binary_crossentropy(output, target):
77 """ Compute the crossentropy of binary output wrt binary target.
78
79 .. math::
80 L_{CE} \equiv t\log(o) + (1-t)\log(1-o)
81
82 :type output: Theano variable
83 :param output: Binary output or prediction :math:`\in[0,1]`
84 :type target: Theano variable
85 :param target: Binary target usually :math:`\in\{0,1\}`
86 """
87 return -(target * tensor.log(output) + (1.0 - target) * tensor.log(1.0 - output))
88
89
90 TODO
91 ----
92 * define a list of search tag to start with
93 * Add to the html page a list of the tag and a list of each fct associated to them.
94 * move existing formulas to pylearn as examples and add other basics ones.
95 * theano.tensor.nnet will probably be copied to pylearn.formulas.nnet and depricated.
96