changeset 1013:5e9a3d9bc0b4

optimization - added some text
author James Bergstra <bergstrj@iro.umontreal.ca>
date Fri, 03 Sep 2010 14:07:34 -0400
parents 5d7022325d8e
children 0feaaa3fc566
files doc/v2_planning/optimization.txt
diffstat 1 files changed, 34 insertions(+), 0 deletions(-) [+]
line wrap: on
line diff
--- a/doc/v2_planning/optimization.txt	Fri Sep 03 13:55:01 2010 -0400
+++ b/doc/v2_planning/optimization.txt	Fri Sep 03 14:07:34 2010 -0400
@@ -1,3 +1,37 @@
 Discussion of Optimization-Related Issues
 =========================================
 
+Members: JB, PL, OD
+
+Representative: JB
+
+
+Previous work - scikits, openopt, scipy  provide function optimization
+algorithms.  These are not currently GPU-enabled but may be in the future.
+
+
+IS PREVIOUS WORK SUFFICIENT?
+--------------------------------
+
+In many cases it is (I used it for sparse coding, and it was ok).
+
+These packages provide batch optimization, whereas we typically need online
+optimization.
+
+It can be faster (to run) and more convenient (to implement) to have
+optimization algorithms as as Theano update expressions.
+
+
+What optimization algorithms to we want/need?
+---------------------------------------------
+
+ - sgd 
+ - sgd + momentum
+ - sgd with annealing schedule
+ - TONGA
+ - James Marten's Hessian-free
+
+Do we need anything to make batch algos work better with Pylearn things?
+ - conjugate methods?
+ - L-BFGS?
+