pylearn: doc/v2_planning/datalearn.txt annotate

annotate doc/v2_planning/datalearn.txt @ 1491:8be8cdde97ee

tensor instead of T

author	Emmanuel Bengio <bengioe@iro.umontreal.ca>
date	Wed, 27 Jul 2011 13:27:00 -0400
parents	e3d02b0a05e3
children

rev	line source
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	1 DataLearn: How to plug Datasets & Learner together?
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	2 ===================================================
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	3
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	4
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	5 Participants
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	6 ------------
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	7 - Yoshua
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	8 - Razvan
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	9 - Olivier D [leader]
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	10 - James
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	11
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	12
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	13 High-Level Objectives
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	14 ---------------------
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	15
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	16 * Simple ML experiments should be simple to write
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	17 * More complex / advanced scenarios should be possible without being forced
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	18 to work "outside" of this framework
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	19 * Computations should be optimized whenever possible
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	20 * Existing code (in any language) should be "wrappable" within this
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	21 framework
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	22 * It should be possible to replace [parts of] this framework with C++ code
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	23
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	24
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	25 Theano-Like Data Flow
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	26 ---------------------
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	27
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	28 We want to rely on Theano to be able to take advantage of its efficient
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	29 computations. The general idea is that if we chain multiple processing
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	30 elements (think e.g. of a feature selection step followed by a PCA projection,
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	31 then a rescaling within a fixed bounded interval), the overall transformation
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	32 from input to output data can be represented by a Theano symbolic graph. When
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	33 one wants to access the actual numeric data, a function is compiled so as to
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	34 do these computations efficiently.
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	35
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	36 We discussed some specific API options for datasets and learners, which will
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	37 be added to this file in the future, but a core question that we feel should
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	38 be addressed first is how this Theano-based implementation could be achieved
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	39 exactly. For this purpose, in the following, let us assume that a dataset is
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	40 simply a matrix whose rows represent individual samples, and columns
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	41 individual features. How to handle field names, non-tensor-like data, etc. is
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	42 a very important topic that is not yet discussed in this file.
ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	43
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	44 The main idea in this proposal is to consider some Data object as a Theano
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	45 Variable (we call 'data' an object that is either a sample, or a collection of
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	46 samples i.e a dataset). Because the Data API (for the Machine Learning user)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	47 may conflict with the Variable API, in the following we take the approach that
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	48 a data object contains a Theano variable accessible through data.variable
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	49 (instead of Data being a subclass of Variable). For instance a basic way of
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	50 printing the content of a dataset could be:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	51
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	52 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	53
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	54 dataset = NumpyDataset(some_numpy_array) # View array as dataset.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	55 index = theano.tensor.lscalar()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	56 get_sample_value = theano.function([index], dataset[index].variable)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	57 for i in xrange(len(dataset)):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	58 print get_sample_value(i)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	59
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	60 There may also exist some helper function for the common task on iterating
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	61 over the numeric values found in a dataset, which would allow one to simply
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	62 write:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	63
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	64 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	65
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	66 for sample_value in theano_iterate(dataset):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	67 print sample_value
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	68
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	69 where the theano_iterate function would take care of the extra work:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	70
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	71 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	72
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	73 def theano_iterate(dataset, index=None, condition=None,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	74 stop_exceptions=(IndexError, )):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	75 if index is None:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	76 index = theano.tensor.lscalar()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	77 if condition is None:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	78 condition = index < len(dataset)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	79 get_value = theano.function([index],
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	80 [dataset[index].variable, condition])
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	81 i = 0
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	82 while True:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	83 try:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	84 output, cond = get_value(i)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	85 except stop_exceptions:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	86 break
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	87 i += 1
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	88 if cond:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	89 yield output
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	90 else:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	91 break
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	92
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	93 Now imagine a similar situation (willing to iterate on a dataset) where the
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	94 datsaet is the result of some transformation parameterized by another
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	95 Variable. For instance, let's say there exists a GetColumnDataset class such
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	96 that GetColumnDataset(dataset, index_variable) is a dataset whose associated
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	97 variable is dataset.variable[:, index_variable] (assuming here that
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	98 dataset.variable is a matrix variable). One would like to write:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	99
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	100 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	101
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	102 for j in xrange(dataset.n_columns()):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	103 print 'Printing column %s' % j
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	104 for sample_value in theano_iterate(GetColumnDataset(dataset, j)):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	105 print sample_value
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	106
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	107 Although this would work, note that it would compile a new Theano function
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	108 each time theano_iterate is called (one for each value of j), which may be a
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	109 performance bottleneck. One way to avoid this is to just ignore the helper
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	110 function and manually compile a function that also takes the column index as
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	111 input parameter:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	112
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	113 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	114
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	115 sample_idx = theano.tensor.lscalar()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	116 column_idx = theano.tensor.lscalar()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	117 get_value = theano.function(
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	118 [sample_idx, column_idx],
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	119 GetColumnDataset(dataset, column_idx)[sample_idx].variable)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	120 for j in xrange(dataset.n_columns()):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	121 print 'Printing column %s' % j
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	122 for i in xrange(len(dataset)):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	123 print get_value(i, j)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	124
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	125 It is however possible to use the helper function if it can accept an extra
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	126 argument ('givens') to be provided to the theano compilation step:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	127
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	128 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	129
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	130 def theano_iterate(dataset, index=None, condition=None,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	131 stop_exceptions=(IndexError, ),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	132 givens={}):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	133 (...)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	134 get_value = theano.function([index],
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	135 [dataset[index].variable, condition],
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	136 givens=givens)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	137 (...)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	138
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	139 column_idx = theano.tensor.lscalar()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	140 shared_column_idx = theano.shared(0)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	141 iterate = theano_iterate(GetColumnDataset(dataset, column_idx),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	142 givens={column_idx: shared_column_idx})
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	143 for j in xrange(dataset.n_columns()):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	144 print 'Printing column %s' % j
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	145 shared_column_idx.value = j
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	146 for sample_value in iterate:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	147 print sample_value
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	148
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	149 Note there are a couple oddities in the example above:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	150 1. The way theano_iterate was written, it is not possible to iterate on it
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	151 more than once. This is easily fixed by making it an iterable object.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	152 2. It would make more sense here to remove 'column_idx' and directly use
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	153 GetColumnDataset(dataset, shared_column_idx), in which case there is no
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	154 need to use the 'givens' keyword. But the goal here is to illustrate a
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	155 situation where one is given a dataset defined from a symbolic variable,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	156 and we want to compute it for different numeric values of this variable.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	157 This dataset may have been provided by code the user has no control on,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	158 thus the need for 'givens' to replace the variable with a shared one
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	159 whose value can be updated between successive calls to the same
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	160 function.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	161
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	162 In summary:
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	163 - Data (samples and datasets) are basically Theano Variables, and a data
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	164 transformation an Op.
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	165 - When writing code that requires some data numeric value, one has to compile
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	166 a Theano function to obtain it. This is done either manually or through some
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	167 helper Pylearn functions for common tasks. In both cases, the user should
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	168 have enough control to be able to obtain an efficient implementation.
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	169
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	170
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	171 What About Learners?
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	172 --------------------
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	173
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	174 The discussion above only mentioned datasets, but not learners. The learning
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	175 part of a learner is not a main concern (currently). What matters most w.r.t.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	176 what was discussed above is how a learner takes as input a dataset and outputs
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	177 another dataset that can be used with the dataset API.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	178
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	179 A Learner may be able to compute various things. For instance, a Neural
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	180 Network may output a ``prediction`` vector (whose elements correspond to
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	181 estimated probabilities of each class in a classification task), as well as a
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	182 ``cost`` vector (whose elements correspond to the penalized NLL, the NLL alone
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	183 and the classification error). We would want to be able to build a dataset
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	184 that contains some of these quantities computed on each sample in the input
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	185 dataset.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	186
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	187 The Neural Network code would then look something like this:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	188
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	189 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	190
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	191 class NeuralNetwork(Learner):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	192
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	193 # The decorator below is reponsible for turning a function that
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	194 # takes a symbolic sample as input, and outputs a Theano variable,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	195 # into a function that can also be applied on numeric sample data,
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	196 # or symbolic datasets.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	197 # Other approaches than a decorator are possible (e.g. using
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	198 # different function names).
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	199 def compute_prediction(self, sample):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	200 return softmax(theano.tensor.dot(self.weights, sample.input))
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	201
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	202 @datalearn
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	203 def compute_nll(self, sample):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	204 return - log(self.compute_prediction(sample)[sample.target])
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	205
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	206 @datalearn
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	207 def compute_penalized_nll(self, sample):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	208 return (self.compute_nll(self, sample) +
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	209 theano.tensor.sum(self.weights**2))
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	210
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	211 @datalearn
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	212 def compute_class_error(self, sample):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	213 probabilities = self.compute_prediction(sample)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	214 predicted_class = theano.tensor.argmax(probabilities)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	215 return predicted_class != sample.target
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	216
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	217 @datalearn
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	218 def compute_cost(self, sample):
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	219 return theano.tensor.concatenate([
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	220 self.compute_penalized_nll(sample),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	221 self.compute_nll(sample),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	222 self.compute_class_error(sample),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	223 ])
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	224
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	225 The ``@datalearn`` decorator would allow such a Learner to be used e.g. like
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	226 this:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	227
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	228 .. code-block:: python
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	229
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	230 nnet = NeuralNetwork()
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	231 # Symbolic dataset that represents the output on symbolic input data.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	232 predict_dataset = nnet.compute_prediction(dataset)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	233 for sample in dataset:
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	234 # Symbolic sample that represents the output on a single symbolic
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	235 # input sample.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	236 predict_sample = nnet.compute_prediction(sample)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	237 # Numeric prediction.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	238 predict_numeric = nnet.compute_prediction({'input': numpy.zeros(10)})
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	239 # Combining multiple symbolic outputs.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	240 multiple_fields_dataset = ConcatDataSet([
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	241 nnet.compute_prediction(dataset),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	242 nnet.compute_cost(dataset),
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	243 ])
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	244
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	245 In the code above, if one wants to obtain the numeric value of an element of
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	246 ``multiple_fields_dataset``, the Theano function being compiled should be able
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	247 to optimize computations so that the simultaneous computation of
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	248 ``prediction`` and ``cost`` is done efficiently.
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	249
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	250
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	251 Open Problems
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	252 -------------
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	253
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	254 The above is not yet a practical proposal. Investigation of the following
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	255 topics is still missing:
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	256
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	257 - Datasets whose variables are not matrices (e.g. large datasets that do not
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	258 fit in memory, non fixed-length vector samples, ...)
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	259 - Field names.
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	260 - Typical input / target / weight split.
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	261 - Learners whose output on a dataset cannot be obtained by computing outputs
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	262 on individual samples (e.g. a Learner that ranks samples based on pair-wise
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	263 comparisons).
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	264 - Code parallelization, stop & restart.
0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	265 - Modular C++ implementation without Theano.
1386 e3d02b0a05e3 Added another open question in datalearn Olivier Delalleau <delallea@iro> parents: 1377 diff changeset	266 - How do we take care of model learning within such a Theano graph?
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	267 - ...
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	268
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	269
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	270 Previous Introduction (deprecated)
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	271 ----------------------------------
e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	272
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	273 A question we did not discuss much is to which extent the architecture could
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	274 be "theanified", i.e. whether a whole experiment could be defined as a Theano
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	275 graph on which high level optimizations could be made possible, while also
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	276 relying on Theano to "run" the graph. The other option is to use a different
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	277 mechanism, with underlying Theano graphs being built wherever possible to link
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	278 the various components of an experiment together.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	279
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	280 For now, let us consider the latter option, where each dataset contains a
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	281 pointer to a Theano variable that represents the data stored in this dataset.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	282 One issue with this approach is illustrated by the following example. Imagine
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	283 we want to iterate on samples in a dataset and do something with their numeric
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	284 value. We would want the code to be as close as possible to:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	285
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	286 .. code-block:: python
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	287
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	288 for sample in dataset:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	289 do_something_with(sample.numeric_value())
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	290
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	291 A naive implementation of the sample API could be (assuming each sample also
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	292 contains a ``variable`` member which is the variable representing this
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	293 sample's data):
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	294
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	295 .. code-block:: python
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	296
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	297 def numeric_value(self):
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	298 if self.function is None:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	299 # Compile function to output the numeric value stored in this
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	300 # sample's variable.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	301 self.function = theano.function([], self.variable)
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	302 return self.function()
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	303
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	304 However, this is not a good idea, because it would trigger a new function
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	305 compilation for each sample. Instead, we would want something like this:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	306
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	307 .. code-block:: python
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	308
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	309 def numeric_value(self):
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	310 if self.function_storage[0] is None:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	311 # Compile function to output the numeric value stored in this
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	312 # sample's variable. This function takes as input the index of
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	313 # the sample in the dataset, and is shared among all samples.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	314 self.function_storage[0] = theano.function(
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	315 [self.symbolic_index], self.variable)
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	316 return self.function(self.numeric_index)
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	317
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	318 In the code above, we assume that all samples created by the action of
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	319 iterating over the dataset share the same ``function_storage``,
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	320 ``symbolic_index`` and ``variable``: the first time we try to access the numeric
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	321 value of some sample, a function is compiled, that takes as input the index,
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	322 and outputs the variable. The only difference between samples is thus that
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	323 they are given a different numeric value for the index (``numeric_index``).
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	324
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	325 Another way to obtain the same result is to actually let the user take care of
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	326 compiling the function. It would allow the user to really control what is
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	327 being compiled, at the cost of having to write more code:
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	328
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	329 .. code-block:: python
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	330
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	331 symbolic_index = dataset.get_index() # Or just theano.tensor.iscalar()
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	332 get_sample = theano.function([symbolic_index],
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	333 dataset[symbolic_index].variable)
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	334 for numeric_index in xrange(len(dataset))
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	335 do_something_with(get_sample(numeric_index))
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	336
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	337 James comments: this is how I have written the last couple of projects, it's
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	338 slightly verbose but it's clear and efficient.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	339
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	340 The code above may also be simplified by providing helper functions. In the
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	341 example above, such a function could allow us to iterate on the numeric values
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	342 of samples in a dataset while taking care of compiling the appropriate Theano
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	343 function. See Discussion: Helper Functions below.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	344
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	345 Note that although the above example focused on how to iterate over a dataset,
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	346 it can be cast into a more generic problem, where some data (either dataset or
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	347 sample) is the result of some transformation applied to other data, which is
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	348 parameterized by parameters p1, p2, ..., pN (in the above example, we were
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	349 considering a sample that was obtained by taking the p1-th element in a
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	350 dataset). If we use different values for a subset Q of the parameters but keep
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	351 other parameters fixed, we would probably want to compile a single function
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	352 that takes as input all parameters in Q, while other parameters are fixed. It
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	353 may be nice to try and get the best of both worlds, letting the user take
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	354 control on what is being compiled, while leaving the option of using a default
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	355 sensible behavior for those who do not want to worry about it. Whether this is
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	356 possible / desirable is still to-be-determined.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	357
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	358
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	359 Discussion: Are Datasets Variables / Ops?
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	360 -----------------------------------------
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	361
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	362 OD wonders: Should datasets directly be Theano Variables, or should they be a
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	363 different object subclass containing a Theano Variable? The advantage of the
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	364 former option would be that they would fit directly within the Theano
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	365 framework, which may allow high level optimizations on data transformations.
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	366 However, we would lose the ability to combine Theano expressions coded in
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	367 individual datasets into a single graph. Currently, I instead considered that
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	368 a dataset has a member that is a Theano variable, and this variable represents
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	369 the data stored in the dataset. The same is done for individual data samples.
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	370
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	371 James asks: Why would a Theano graph in which some nodes represent datasets give
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	372 up the ability to combine Theano expressions coded in individual datasets?
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	373 Firstly, if you want to use Theano expressions and compiled functions to
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	374 implement the perform() method of an Op, you can do that. Secondly, you can
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	375 just include those 'expressions coded in individual datasets' into the overall
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	376 graph.
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	377
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	378 OD replies to James: What I had in mind is you would be forced to compile your
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	379 own function inside the perform() method of an Op. This seemed like a
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	380 potential problem to me because it would prevent Theano from seeing the whole
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	381 fine-grained graph and do optimizations across multiple dataset
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	382 transformations (there may also be additional overhead from calling multiple
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	383 function). But if you are saying it is possible to include 'expressions coded
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	384 in individual datasets' into the overall graph, then I guess this point is
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	385 moot. Would this be achieved with an optimization that replaces the dataset
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	386 node with its internal graph?
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	387
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	388 Razvan comments: 1) Having Theano expressions inside the perform of a Theano
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	389 Op can lead to issues. I know I had to deal with a few when implementing
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	390 Scan which does exactly this. Well to be fair these issues mostly come into
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	391 play when the inner graph has to interact with the outer graph and most of
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	392 the time they can be solved. I guess all that I'm saying is going that way
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	393 might lead to some head-ache to developers, though I guess some head-ache
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	394 will be involved no matter what
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	395 2) In my view (I'm not sure this is what Olivier was saying) the idea of
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	396 not putting the Dataset into a Variable is to not put the logic related to
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	397 loading data, dividing it into slices when running it on the GPU and so on
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	398 into a theano variable. In my view this logic goes into a DataSet class
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	399 that gives you shared variables, symbolic indices into that shared
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	400 variables, and also numeric indices. When looping through those numeric
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	401 indices, the dataset class can reload parts of the data into the
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	402 shared variable and so on.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	403
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	404 OD replies to Razvan's point 2: I think what you are saying is another concern
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	405 I had, which was the fact it may be confusing to mix in the same class the
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	406 Variable/Op and DataSet interfaces. I would indeed prefer to keep them
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	407 separate. However, it may be possible to come up with a system that would get
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	408 the best of both worlds (maybe by having the Op/Variable as members of
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	409 Dataset, and just asking the user building a theano graph to use these instead
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	410 of the dataset directly). Note that I'm mixing up Op/Variable here, because
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	411 it's just not clear yet for me which would go where...
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	412
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	413
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	414 Discussion: Implicit / Explicit Function Compilation
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	415 ----------------------------------------------------
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	416
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	417 <Razvan comments>: I assume that ``do_something_with`` is suppose to be some
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	418 numeric function, and dataset in this case is the result of some
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	419 computations on a initial dataset.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	420 I would differentiate the two approaches (1) and (2) as :
1377 0665274b14af Minor fixes for better Sphinx doc output Olivier Delalleau <delallea@iro> parents: 1376 diff changeset	421
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	422 - first of all whatever you can do with (1) you can do with (2)
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	423 - approach (1) hides the fact that you are working with symbolic graphs.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	424 You apply functions to datasets, and when you want to see values a
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	425 function is compiled under the hood and those values are computed for
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	426 you. In approach (2) the fact that you deal with a symbolic graph is
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	427 explicit because you have to manually compile your functions.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	428 - approach (1) needs to use this function_storage trick shared between
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	429 certain nodes of the graph to reduce the number of compilation while in
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	430 approach (2) we don't need to deal with the complexity of lazy
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	431 compilation
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	432
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	433 OD comments: Well, to be fair, it means we put the burden of dealing with the
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	434 complexity of lazy compilation on the user (it's up to him to make sure he
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	435 compiles only one function).
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	436
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	437 - approach (1) needs a replace function if you want to change the dataset.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	438 What you would do, is once you have a "computational graph" or pipeline
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	439 or whatever you call it, say ``graph``, to change the input you would do
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	440 graph.replace({ init_data_X: new_data_X}), In approach (2) the init_data_X
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	441 and new_data_X is the ``dataset`` so you would compile two different
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	442 functions. Well I would re-write (2) -- to make the above more clear --
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	443 as :
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	444
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	445 .. code-block:: python
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	446
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	447 symbolic_index = theano.tensor.iscalar()
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	448 get_sample1 = theano.function( [symbolic_index],
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	449 graph( dataset[symbolic_index] ).variable)
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	450 for numeric_index in xrange(len(dataset)):
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	451 do_something_with(get_sample(numeric_index))
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	452
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	453 get_sample2 = theano.function( [symbolic_index],
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	454 graph( new_dataset[symbolic_index] ).variable)
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	455 ## Note: the dataset was replaced with new_dataset
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	456 for numeric_index in xrange(len(new_dataset)):
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	457 do_something_with(get_sample2(numeric_index))
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	458
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	459 ######### FOR (1) you write:
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	460
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	461 for datapoint in graph:
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	462 do_something_with( datapoint() )
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	463
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	464 new_graph = graph.replace({dataset:dataset2})
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	465
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	466 for datapoint in new_graph:
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	467 do_something_with(datapoint())
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	468
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	469 OD comments: I don't really understand what is 'graph' in this code (it
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	470 appears in both approaches but is used differently). What I have in mind would
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	471 be more with 'graph' removed in the first approach you describe (#2), and
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	472 graph / new_graph replaced by dataset / new_dataset in the second one (#1).
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	473 You wouldn't need to call some graph.replace method: the graphs compiled for
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	474 iterating on 'dataset' and 'new_dataset' would be entirely separate (using two
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	475 different compiled functions, pretty much like #2).
1363 18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	476
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	477 RP answers: Yes you are right. What I was trying to say is if you have two
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	478 different datasets on which you want to apply the same pre-processing you
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	479 can do that in both approaches. ``graph`` represents the pre-processing
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	480 steps in (2) and the end dataset (after preprocessing) in (1). So the idea
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	481 is that instead of making new_graph from scratch (re-applying all the
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	482 transforms on the original dataset) you can use replace. Or maybe the
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	483 __call__ (that compiles the function if needed) can get a givens dictionary
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	484 ( that replaces datasets or more ). I only gave this argument because I
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	485 thought this will be an issue people will raise. They will say, well in (2)
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	486 the pipeline logic is separated from the data, so you can use the same
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	487 transformation with different data easily, while in (1) you write the
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	488 transformation rooted in a dataset, and if you want same transformation
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	489 for a different dataset you have to re-write everything.
18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	490
1364 01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	491 OD replies: Still not sure I understand. If you have a "graph" function that
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	492 takes a dataset as input and outputs a new dataset, you can use this same
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	493 function with both (1) and (2). With (2) it is:
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	494 theano.function([index], graph(my_dataset)[index].variable)
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	495 while with (1) the same function is compiled implicitly with:
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	496 for sample in graph(my_dataset):
01157763c2d7 Reply to Razvan Olivier Delalleau <delallea@iro> parents: 1363 diff changeset	497 ...
1363 18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	498
1365 049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	499 RP answers: right. I was actually constructing this stupid example in my mind when
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	500 you would do like :
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	501 i1 = f1(data)
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	502 i2 = f2(i1)
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	503 i3 = f3(i2)
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	504 ...
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	505 iN = fN(iN-1)
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	506 and then you would say .. wait I want to do this on new_data as well. Oh no, I
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	507 have to copy the entire block or whatever. That is so annoying. But actually you
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	508 could just write:
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	509
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	510 def my_f(data):
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	511 i1 = f1(data)
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	512 ...
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	513 return iN
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	514
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	515 and then just use that function which is what you pointed out. I agree I'm
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	516 not sure anymore on the point that I was trying to make. Is like if you are
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	517 a lazy programmer, and you write everything without functions, you can
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	518 argue that you like more (2) because you only pass the dataset at the end
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	519 and not at the beginning. But if (1) would have the replace function this
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	520 argument will fail. Though this only stands if you like don't want to make
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	521 a function out of your pipeline that takes the dataset as input, which now
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	522 that I think about it is pretty stupid not to do. Sorry for that.
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	523
049b99f4b323 reply to OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1364 diff changeset	524
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	525 - in approach (1) the initial dataset object (the one that loads the data)
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	526 decides if you will use shared variables and indices to deal with the
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	527 dataset or if you will use ``theano.tensor.matrix`` and not the user( at
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	528 least not without hacking the code). Of course whoever writes that class
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	529 can add a flag to it to switch between behaviours that make sense.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	530 In approach (2) one is not forced to do this
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	531 inside that class by construction, though by convention I would do it.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	532 So if you consider the one who writes that class as a developer than
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	533 in (2) the user can decide/deal with this and not the developer.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	534 Though this is a fine-line -- I would say the user would actually
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	535 write that class as well using some template.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	536 That is to say (2) looks and feels more like working with Theano
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	537 directly,
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	538
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	539 Bottom line, I think (1) puts more stress on the development of the library,
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	540 and hides Theano and some of the complexity for day to day usage.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	541 In (2) everything is a bit more explicit, leaving the impression that you
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	542 have more control over the code, though I strongly feel that whatever can
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	543 be done in (2) can be done in (1). Traditionally I was more inclined
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	544 towards (1) but now I'm not that sure, I think both are equally interesting
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	545 and valid options.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	546 </Razvan comments>
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	547
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	548
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	549 Discussion: Fixed Parameters vs. Function Arguments
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	550 ---------------------------------------------------
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	551
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	552 Razvan Comment: I thought about this a bit at the Pylearn level. In my
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	553 original train of thought you would have the distinction between ``hand
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	554 picked parameters`` which I would call hyper-parameter and learned
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	555 parameters. A transformation in this framework (an op if you wish) could
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	556 take as inputs DataSet(s), DataField(s), Parameter(s) (which are the things
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	557 that the learner should adapt) and HyperParameter(s). All hyper-parameters
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	558 will turn into arguments of the compiled function (like the indices of each
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	559 of the dataset objects ) and therefore they can be changed without
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	560 re-compilation. Or in other words this can be easily done by having new
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	561 types of Variables that would represent Parameters and Hyper-parameters.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	562 And as an ending note I would say that there are
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	563 hyper-parameters for which you need to recompile the thenao function and
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	564 can not be just parameters ( so we would have yet another category ?).
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	565
1366 f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	566 Yoshua's comments on RP's comments: I don't understand why we would
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	567 need to create these types. Isn't it just a matter for the programmer
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	568 to decide what are the inputs of the compiled function, and which
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	569 are possibly constant (e.g. holding some hyper-parameters constant
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	570 for a while)?
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	571
1368 ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	572 RP answers: If we opt for this lazy compilation mechanism, the library needs
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	573 to know what to put into a shared, and what to expect as input. The
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	574 programmer should give hints to the library by saying this value will always
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	575 be constant, or this is a hyper-parameter that I might want to change, and
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	576 when I do that I don't want to recompile everything so put it as an
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	577 argument. Even when the compilation is done by the user, it would be helpful
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	578 to have some function that collects all the parameters for you. What I mean
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	579 is that it would be nice to write something like
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	580
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	581 corruption_layer_1 = Parameter ( value = 0.1, name = 'c1')
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	582 # Followed by (many) lines of code
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	583 f = function ( results.inputs()+ results.hyper-params(), result )
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	584
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	585
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	586 where results.hyper-params parses the graph, collects the hyper-parameter
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	587 and returns them as a list of theano.Variables wrappen in theano.In with
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	588 a default value and a name. You could call the function either as
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	589
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	590 f()
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	591 or
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	592 f(c1 = 0.2)
ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	593
1370 5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	594 OD comments: Here is a (hopefully simpler) suggestion to solve this problem.
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	595 Consider any data{set,point} obtained by a transformation of an existing
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	596 data{set,point} with parameters p1, p2, ..., pN. From the point of view of
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	597 theano variables, this is something like x2 = h(x1, p1=v1, ..., pn=vN) where
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	598 x1, x2 are variables and h is an Op. In addition v1 ... vN are also variables
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	599 since they are parameters of the transformation we may want to vary. This is
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	600 not, however, the way the user would build the graph, because being forced to
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	601 use variables for parameters is not user-friendly (IMO). Instead, someone
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	602 would write:
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	603 d2 = t(d1, p1=w1, ..., pn=wN)
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	604 where d1, d2 are data{set,point}s, t is the transformation, and w1 ... wN are
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	605 numeric values of the parameters. Then t would build the piece of graph above,
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	606 so that when you ask d2.numeric_value(), a function computing x2 would be
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	607 compiled, that would take as input variables v1, ... vN.
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	608 Now, the problem is that this may not be fully optimized, since parameters are
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	609 assumed to be varying (so as not to be forced to recompile a different
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	610 function when the user calls t with different parameter values). My suggestion
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	611 is to make this the default behavior, but add an extra argument to t:
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	612 d2 = t(d1, p1=w1, ..., pn=Wn, constants=['p3', 'p5'])
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	613 The line above would do the same, except that the function being compiled
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	614 would use the constant values w3 and w5 for p3 and p5.
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	615 Razvan's example above would be written in a different way as follows:
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	616 def f(c1=0.2):
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	617 return transformK(..(transform2(transform1(input_data,
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	618 corruption_layer_1=c1))))
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	619 With this code you could create various transformed datasets by callling f
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	620 with different values for c1. The first time you call f(c1=0).numeric_value()
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	621 a Theano function is compiled that takes a `corruption_layer_1` input variable
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	622 (whose value is 0 when the function is called by `numeric_value`). If you call
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	623 f().numeric_value(), the same function is re-used (no need to compile it) with
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	624 this input set to 0.2. If on another hand you want to compile a new function
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	625 for each new value of your `corruption_layer_1` parameter, you would instead
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	626 write:
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	627 def f(c1=0.2):
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	628 return transformK(..(transform2(transform1(input_data,
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	629 corruption_layer_1=c1,
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	630 constants=['corruption_layer_1']))))
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	631 This would be one way to have automatic lazy function cache / compilation
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	632 while still letting the user specify for which parameters a new function needs
5785cbac3361 Added a suggestion to solve the problem of Fixed vs. Varying parameters Olivier Delalleau <delallea@iro> parents: 1369 diff changeset	633 to be compiled when their value changes.
1368 ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	634
1371 98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	635 RP comment : What about the same trick that Theano uses, namely, if you want
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	636 a non "default" behaviour you wrap the input in a dictionary. You would
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	637 write tranform1( input_data,
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	638 corruption_layer_1= In(value = c1, fixed = True)) ?
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	639 I started to like this approach of passing extra info about an argument :).
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	640 Other that this it sounds good to me.
98d4232df1d8 comment on OD idea Razvan Pascanu <r.pascanu@gmail.com> parents: 1370 diff changeset	641
1372 decee534c78d Replied to RP Olivier Delalleau <delallea@iro> parents: 1371 diff changeset	642 OD replies: Yes, I guess it would make sense. The more I look at it, the more
decee534c78d Replied to RP Olivier Delalleau <delallea@iro> parents: 1371 diff changeset	643 it seems like it is very close to directly writing a Theano transform on some
decee534c78d Replied to RP Olivier Delalleau <delallea@iro> parents: 1371 diff changeset	644 variables.
decee534c78d Replied to RP Olivier Delalleau <delallea@iro> parents: 1371 diff changeset	645
1368 ad53f73020c2 Answered Yoshua's question Razvan Pascanu <r.pascanu@gmail.com> parents: 1367 diff changeset	646
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	647 Discussion: Helper Functions
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	648 ----------------------------
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	649
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	650 James: Another syntactic option for iterating over datasets is
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	651
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	652 .. code-block:: python
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	653
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	654 for sample in dataset.numeric_iterator(batchsize=10):
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	655 do_something_with(sample)
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	656
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	657 The numeric_iterator would create a symbolic batch index, and compile a single function
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	658 that extracts the corresponding minibatch. The arguments to the
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	659 numeric_iterator function can also specify what compile mode to use, any givens
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	660 you might want to apply, etc.
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	661
1366 f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	662 Yoshua's comment to James' comment: I like that approach.
f945ed016c68 comment by YB Yoshua Bengio <bengioy@iro.umontreal.ca> parents: 1365 diff changeset	663
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	664 OD comments: Would there also be some kind of function cache to avoid
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	665 compiling the same function again if we re-iterate on the same dataset with
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	666 the same arguments? Maybe a more generic issue is: would there be a way for
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	667 Theano to be more efficient when re-compiling the same function that was
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	668 already compiled in the same program? (note that I am assuming here it is not
1363 18b2ebec6bca Reply to a comment of OD Razvan Pascanu <r.pascanu@gmail.com> parents: 1362 diff changeset	669 efficient, but I may be wrong).
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	670
1369 f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	671 OD adds: After thinking more about it, this seems very close to my first
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	672 version where a function is automatically compiled "under the hood" when
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	673 iterating on a dataset and accessing the numeric value of a resulting
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	674 sample. The main differences are:
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	675 - In your version, the result is directly a numeric value, while in my version
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	676 one would obtain symbolic samples and would need to call some method to
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	677 obtain their numeric value. I think I like mine a bit better because it
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	678 means you can use the same syntax to e.g. iterate on a dataset, whether you
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	679 are interested in the symbolic representation of samples, or their numeric
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	680 values. On another hand, doing so could be less efficient since you create an
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	681 intermediate representation you may not use. The overhead does not seem much
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	682 to me but I am not sure about that.
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	683 - In your version, you can provide to the function e.g. compile modes /
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	684 givens. This could probably also be done in my version, although it makes it
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	685 more difficult if you want to cache the function to avoid compiling it more
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	686 than once (see next point).
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	687 - (Related to my first comment above) In your version it seems like a new
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	688 function would be compiled every time the user calls e.g.
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	689 'numeric_iterator', while in my version the function would be compiled only
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	690 once. Maybe this can be solved at the Theano level with an efficient
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	691 function cache?
f3a549bd8688 datalearn: Added another comment on James' numeric iterator function Olivier Delalleau <delallea@iro> parents: 1368 diff changeset	692
1376 e8fc563dad74 Rewrote the Theano-Like Data Flow section in datalearn.txt Olivier Delalleau <delallea@iro> parents: 1372 diff changeset	693
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	694 Discussion: Dataset as Learner Ouptut
9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	695 -------------------------------------
1357 ffa2932a8cba Added datalearn committee discussion file Olivier Delalleau <delallea@iro> parents: diff changeset	696
1359 5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	697 James asks:
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	698 What's wrong with simply passing the variables corresponding to the dataset to
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	699 the constructor of the learner?
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	700 That seems much more flexible, compact, and clear than the decorator.
5db730bb0e8e comments on datalearn James Bergstra <bergstrj@iro.umontreal.ca> parents: 1357 diff changeset	701
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	702 OD replies: Not sure I understand your idea here. We probably want a learner
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	703 to be able to compute its output on multiple datasets, without having to point
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	704 to these datasets within the learner itself (which seems cumbersome to me).
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	705 The point of the decorators is mostly to turn a single function (that outputs
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	706 a theano variable for the ouptut computed on a single sample) into a function
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	707 that can compute symbolic datasets as well as numeric sample outputs. Those
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	708 could also be instead different functions in the base Learner class if the
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	709 decorator approach is considered ugly / confusing.
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	710
1361 7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	711 Razvan asks: What is predict_sample for ? What is predict_dataset? What I
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	712 guess you mean is that the decorator is used to convert a function that
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	713 takes a theano variable and outputs a theano variable into a class/function
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	714 that takes a DataField/DataSet and outputs a DataField/DataSet. It could
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	715 also register all those different functions, so that the Dataset that
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	716 you get out of (not one of the function) the entire Learner (this Dataset
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	717 is returned by __call__) would contain all those as fields.
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	718 I would use it like this:
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	719
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	720 .. code-block:: python
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	721
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	722 nnet = NeuralNetwork()
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	723 results = nnet(dataset)
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	724 for datapoint in results:
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	725 print datapoint.prediction, datapoint.nll, ...
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	726
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	727 Is this close to what you are suggesting?
7548dc1b163c Some question/suggestions to datalearn Razvan Pascanu <r.pascanu@gmail.com> parents: 1359 diff changeset	728
1362 6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	729 OD: Yes, you guessed right, the decorator's role is to do something different
6b9673d72a41 Datalearn replies / comments Olivier Delalleau <delallea@iro> parents: 1361 diff changeset	730 depending on the input to the function (see my reply to James above).
1367 9474fb4ad109 Refactored datalearn committee file to be easier to read Olivier Delalleau <delallea@iro> parents: 1366 diff changeset	731

Mercurial > pylearn

annotate doc/v2_planning/datalearn.txt @ 1491:8be8cdde97ee