Mercurial > pylearn

annotate doc/v2_planning/API_optimization.txt @ 1423:ea5d27727869

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added pickle disk usage inspection utility 'pkldu'
author Ian Goodfellow
date Mon, 07 Feb 2011 17:04:56 -0500
parents 4d7fdd04b66a
children
rev   line source
1185
4ea46ef9822a small fix to make the API_optimization show on the web.
Frederic Bastien <nouiz@nouiz.org>
parents: 1182
diff changeset
1 .. _v2planning_optimization:
4ea46ef9822a small fix to make the API_optimization show on the web.
Frederic Bastien <nouiz@nouiz.org>
parents: 1182
diff changeset
2
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
3 Optimization API
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
4 ================
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
5
1069
16ea3e5c5a7a api_optimization: Couple questions
Olivier Delalleau <delallea@iro>
parents: 1065
diff changeset
6 Members: Bergstra, Lamblin, Delalleau, Glorot, Breuleux, Bordes
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
7 Leader: Bergstra
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
8
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
9
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
10 Description
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
11 -----------
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
12
1065
2bbc464d6ed0 typo in doc
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1064
diff changeset
13 This API is for iterative optimization algorithms, such as:
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
14
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
15 - stochastic gradient descent (incl. momentum, annealing)
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
16 - delta bar delta
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
17 - conjugate methods
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
18 - L-BFGS
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
19 - "Hessian Free"
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
20 - SGD-QN
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
21 - TONGA
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
22
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
23 The API includes an iterative interface based on Theano, and a one-shot
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
24 interface similar to SciPy and MATLAB that is based on Python and Numpy, that
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
25 only uses Theano for the implementation.
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
26
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
27
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
28 Theano Interface
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
29 -----------------
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
30
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
31 The theano interface to optimization algorithms is to ask for a dictionary of
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
32 updates that can be used in theano.function. Implementations of iterative
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
33 optimization algorithms should be global functions with a signature like
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
34 'iterative_optimizer'.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
35
1188
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
36 .. code-block:: python
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
37
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
38 def iterative_optimizer(parameters,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
39 cost=None,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
40 gradients=None,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
41 stop=None,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
42 updates=None,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
43 **kwargs):
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
44 """
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
45 :param parameters: list or tuple of Theano variables
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
46 that we want to optimize iteratively. If we're minimizing f(x), then
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
47 together, these variables represent 'x'. Typically these are shared
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
48 variables and their values are the initial values for the minimization
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
49 algorithm.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
50
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
51 :param cost: scalar-valued Theano variable that computes an exact or noisy estimate of
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
52 cost (what are the conditions on the noise?). Some algorithms might
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
53 need an exact cost, some algorithms might ignore the cost if the
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
54 gradients are given.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
55
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
56 :param gradients: list or tuple of Theano variables representing the gradients on
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
57 the corresponding parameters. These default to tensor.grad(cost,
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
58 parameters).
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
59
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
60 :param stop: a shared variable (scalar integer) that (if provided) will be
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
61 updated to say when the iterative minimization algorithm has finished
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
62 (1) or requires more iterations (0).
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
63
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
64 :param updates: a dictionary to update with the (var, new_value) items
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
65 associated with the iterative algorithm. The default is a new empty
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
66 dictionary. A KeyError is raised in case of key collisions.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
67
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
68 :param kwargs: algorithm-dependent arguments
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
69
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
70 :returns: a dictionary mapping each parameter to an expression that it
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
71 should take in order to carry out the optimization procedure.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
72
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
73 If all the parameters are shared variables, then this dictionary may be
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
74 passed as the ``updates`` argument to theano.function.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
75
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
76 There may be more key,value pairs in the dictionary corresponding to
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
77 internal variables that are part of the optimization algorithm.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
78
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
79 """
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
80
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
81
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
82 Numpy Interface
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
83 ---------------
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
84
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
85 The numpy interface to optimization algorithms is supposed to mimick
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
86 scipy's. Its arguments are numpy arrays, and functions that manipulate numpy
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
87 arrays.
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
88
1188
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
89 .. code-block:: python
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
90
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
91 def minimize(x0, f, df, opt_algo, **kwargs):
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
92 """
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
93 Return a point x_new with the same type as x0 that minimizes function `f`
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
94 with derivative `df`.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
95
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
96 This is supposed to provide an interface similar to scipy's minimize
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
97 routines, or MATLAB's.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
98
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
99 :type x0: numpy ndarray or list of numpy ndarrays.
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
100 :param x0: starting point for minimization
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
101
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
102 :type f: python callable mapping something like x0 to a scalar
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
103 :param f: function to minimize
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
104
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
105 :type df: python callable mapping something like x0 to the derivative of f at that point
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
106 :param df: derivative of `f`
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
107
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
108 :param opt_algo: one of the functions that implements the
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
109 `iterative_optimizer` interface.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
110
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
111 :param kwargs: passed through to `opt_algo`
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
112
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
113 """
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
114
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
115
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
116 There is also a numpy-based wrapper to the iterative algorithms.
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
117 This can be more useful than minimize() because it doesn't hog program
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
118 control. Technically minimize() is probably implemented using this
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
119 minimize_iterator interface.
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
120
1188
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
121 .. code-block:: python
073c2fab7bcd fix rst syntax.
Frederic Bastien <nouiz@nouiz.org>
parents: 1185
diff changeset
122
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
123 class minimize_iterator(object):
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
124 """
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
125 Attributes
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
126 - x - the current best estimate of the minimum
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
127 - f - the function being minimized
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
128 - df - f's derivative function
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
129 - opt_algo - the optimization algorithm at work (a serializable, callable
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
130 object with the signature of iterative_optimizer above).
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
131
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
132 """
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
133 def __init__(self, x0, f, df, opt_algo, **kwargs):
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
134 """Initialize state (arguments as in minimize())
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
135 """
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
136 def __iter__(self):
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
137 return self
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
138 def next(self):
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
139 """Take a step of minimization and return self raises StopIteration when
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
140 the algorithm is finished with minimization
1064
a41cc29cee26 v2planning optimization - API draft
James Bergstra <bergstrj@iro.umontreal.ca>
parents:
diff changeset
141
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
142 """
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
143
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
144
1193
886db9dad2f9 api_optimization - added reply to OD
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1188
diff changeset
145 JB replies: I don't think so, but for the few lines of code required, I think it would be nice to
886db9dad2f9 api_optimization - added reply to OD
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1188
diff changeset
146 provide an function that matches scipy's API.
886db9dad2f9 api_optimization - added reply to OD
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1188
diff changeset
147
1206
203569655069 API_optimization: Added note about apparently wrongly placed reply to discussion
Olivier Delalleau <delallea@iro>
parents: 1193
diff changeset
148 OD: Looks like the reply above was pasted in the wrong place... where was it
203569655069 API_optimization: Added note about apparently wrongly placed reply to discussion
Olivier Delalleau <delallea@iro>
parents: 1193
diff changeset
149 supposed to go?
203569655069 API_optimization: Added note about apparently wrongly placed reply to discussion
Olivier Delalleau <delallea@iro>
parents: 1193
diff changeset
150
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
151 Examples
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
152 --------
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
153
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
154 Simple stochastic gradient descent could be called like this:
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
155
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
156 .. code-block:: python
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
157
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
158 sgd([p], gradients=[g], step_size=.1)
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
159
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
160 and this would return
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
161
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
162 .. code-block:: python
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
163
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
164 {p:p-.1*g}
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
165
1100
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
166
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
167 Simple stochastic gradient descent with extra updates:
153cf820a975 v2planning - updates to api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1069
diff changeset
168
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
169 .. code-block:: python
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
170
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
171 sgd([p], gradients=[g], updates={a:b}, step_size=.1)
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
172
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
173 will return
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
174
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
175 .. code-block:: python
1149
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
176
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
177 {a:b, p:p-.1*g}
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
178
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
179
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
180 If the parameters collide with keys in a given updates dictionary an exception
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
181 will be raised:
7c5dc11c850a cleaning up api_optimization
James Bergstra <bergstrj@iro.umontreal.ca>
parents: 1108
diff changeset
182
1262
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
183 .. code-block:: python
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
184
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
185 sgd([p], gradients=[g], updates={p:b}, step_size=.1)
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
186
4d7fdd04b66a More RSTification
Pascal Lamblin <lamblinp@iro.umontreal.ca>
parents: 1206
diff changeset
187 will raise a ``KeyError``.