Mercurial > pylearn

annotate doc/v2_planning/formulas.txt @ 1104:5e6d7d9e803a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
a comment on the GPU issue for datasets
author Razvan Pascanu <r.pascanu@gmail.com>
date Mon, 13 Sep 2010 20:21:23 -0400
parents bc246542d6ff
children 42ddbefd1e03
rev   line source
1051
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
1 Math formulas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
2 =============
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
3
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
4 Participants
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
5 ------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
6 - Fred*
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
7 - Razvan
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
8 - Aaron
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
9 - Olivier B.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
10 - Nicolas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
11
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
12 TODO
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
13 ----
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
14 * define a list of search tag to start with
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
15 * propose an interface(many inputs, outputs, doc style, hierrache, to search, html output?)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
16 * find existing repositories with files for formulas.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
17 * move existing formulas to pylearn as examples and add other basics ones.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
18 ** theano.tensor.nnet will probably be copied to pylearn.formulas.nnet and depricated.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
19
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
20 Why we need formulas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
21 --------------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
22
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
23 Their is a few reasons why having a library of mathematical formula for theano is a good reason:
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
24
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
25 * Some formula have some special thing needed for the gpu.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
26 * Sometimes we need to cast to floatX...
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
27 * Some formula have numerical stability problem.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
28 * Some formula gradiant have numerical stability problem. (Happen more frequently then the previous ones)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
29 * If theano don't always do some stability optimization, we could do it manually in the formulas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
30 * Some formula as complex to implement and take many try to do correctly.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
31
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
32 Having a library help in that we solve those problem only once.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
33
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
34 Formulas definition
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
35 -------------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
36
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
37 We define formulas as something that don't have a state. They are implemented as python function
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
38 that take theano variable as input and output theano variable. If you want state, look at what the
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
39 learner commity will do.
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
40
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
41 Formulas doc must have
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
42 ----------------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
43
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
44 * A latex mathematical description of the formulas(for picture representation in generated documentation)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
45 * Tags(for searching):
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
46 * a list of lower lovel fct used
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
47 * category(name of the submodule itself)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
48 * Tell if we did some work to make it more numerical stable. Do theano do the optimization needed?
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
49 * Tell if the grad is numericaly stable? Do theano do the optimization needed?
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
50 * Tell if work on gpu/not/unknow
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
51 * Tell alternate name
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
52 * Tell the domaine, range of the input/output(range should use the english notation of including or excluding)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
53
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
54 List of existing repos
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
55 ----------------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
56
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
57 Olivier B. ?
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
58 Xavier G.: git@github.com:glorotxa/DeepANN.git, see file deepANN/{Activations.py(to nnet),Noise.py,Reconstruction_cost.py(to costs),Regularization.py(to regularization}
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
59
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
60 Proposed hierarchy
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
61 ------------------
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
62
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
63 Here is the proposed hierarchy for formulas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
64
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
65 pylearn.formulas.costs: generic / common cost functions, e.g. various cross-entropies, squared error,
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
66 abs. error, various sparsity penalties (L1, Student)
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
67
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
68 pylearn.formulas.regularization: formulas for regularization
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
69
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
70 pylearn.formulas.linear: formulas for linear classifier, linear regression, factor analysis, PCA
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
71
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
72 pylearn.formulas.nnet: formulas for building layers of various kinds, various activation functions,
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
73 layers which could be plugged with various costs & penalties, and stacked
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
74
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
75 pylearn.formulas.ae: formulas for auto-encoders and denoising auto-encoder variants
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
76
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
77 pylearn.formulas.noise: formulas for corruption processes
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
78
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
79 pylearn.formulas.rbm: energies, free energies, conditional distributions, Gibbs sampling
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
80
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
81 pylearn.formulas.trees: formulas for decision trees
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
82
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
83 pylearn.formulas.boosting: formulas for boosting variants
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
84
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
85 pylearn.formulas.maths for other math formulas
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
86
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
87 pylearn.formulas.scipy.stats: example to implement the same interface as existing lib
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
88
bc246542d6ff added file for the formulas commitee.
Frederic Bastien <nouiz@nouiz.org>
parents:
diff changeset
89 etc.