Mercurial > pylearn

comparison doc/v2_planning/api_optimization.txt @ 1064:a41cc29cee26

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
v2planning optimization - API draft
author James Bergstra <bergstrj@iro.umontreal.ca>
date Thu, 09 Sep 2010 17:44:43 -0400
parents
children 2bbc464d6ed0
comparison
equal deleted inserted replaced
1063:074901ccf7b6 1064:a41cc29cee26
1 Optimization API
2 ================
3
4 Members: Bergstra, Lamblin, Dellaleau, Glorot, Breuleux, Bordes
5 Leader: Bergstra
6
7
8 Description
9 -----------
10
11 We provide an API for iterative optimization algorithms, such as:
12
13 - stochastic gradient descent (incl. momentum, annealing)
14 - delta bar delta
15 - conjugate methods
16 - L-BFGS
17 - "Hessian Free"
18 - SGD-QN
19 - TONGA
20
21 The API includes an iterative interface based on Theano, and a one-shot
22 interface similar to SciPy and MATLAB that is based on Python and Numpy, that
23 only uses Theano for the implementation.
24
25
26 Iterative Interface
27 -------------------
28
29 def iterative_optimizer(parameters,
30 cost=None,
31 grads=None,
32 stop=None,
33 updates=None,
34 **kwargs):
35 """
36 :param parameters: list or tuple of Theano variables (typically shared vars)
37 that we want to optimize iteratively. If we're minimizing f(x), then
38 together, these variables represent 'x'.
39
40 :param cost: scalar-valued Theano variable that computes an exact or noisy estimate of
41 cost (what are the conditions on the noise?). Some algorithms might
42 need an exact cost, some algorithms might ignore the cost if the grads
43 are given.
44
45 :param grads: list or tuple of Theano variables representing the gradients on
46 the corresponding parameters. These default to tensor.grad(cost,
47 parameters).
48
49 :param stop: a shared variable (scalar integer) that (if provided) will be
50 updated to say when the iterative minimization algorithm has finished
51 (1) or requires more iterations (0).
52
53 :param updates: a dictionary to update with the (var, new_value) items
54 associated with the iterative algorithm. The default is a new empty
55 dictionary. A KeyError is raised in case of key collisions.
56
57 :param kwargs: algorithm-dependent arguments
58
59 :returns: a dictionary mapping each parameter to an expression that it
60 should take in order to carry out the optimization procedure.
61
62 If all the parameters are shared variables, then this dictionary may be
63 passed as the ``updates`` argument to theano.function.
64
65 There may be more key,value pairs in the dictionary corresponding to
66 internal variables that are part of the optimization algorithm.
67
68 """
69
70
71 One-shot Interface
72 ------------------
73
74 def minimize(x0, f, df, opt_algo, **kwargs):
75 """
76 Return a point x_new that minimizes function `f` with derivative `df`.
77
78 This is supposed to provide an interface similar to scipy's minimize
79 routines, or MATLAB's.
80
81 :type x0: numpy ndarray
82 :param x0: starting point for minimization
83
84 :type f: python callable mapping something like x0 to a scalar
85 :param f: function to minimize
86
87 :type df: python callable mapping something like x0 to the derivative of f at that point
88 :param df: derivative of `f`
89
90 :param opt_algo: one of the functions that implements the
91 `iterative_optimizer` interface.
92
93 :param kwargs: passed through to `opt_algo`
94
95 """
96
97
98