Mercurial > ift6266

annotate scripts/launch_generate100.py @ 349:22efb4968054

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added pnist support, will check in code for data set iterator later
author xaviermuller
date Mon, 19 Apr 2010 10:12:17 -0400
parents 44409b6652aa
children 543ae35e387e
rev   line source
136
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
1 #!/usr/bin/env python
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
2
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
3 import os
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
4 dir1 = "/data/lisa/data/ift6266h10/"
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
5
274
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
6 mach = ["maggie16.iro.umontreal.ca,zappa8@iro.umontreal.ca"]
203
fd1b5237e49e Actual script used to launch images generation on 4 machines
boulanni <nicolas_boulanger@hotmail.com>
parents: 157
diff changeset
7
274
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
8 #test and valid sets
136
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
9 for i,s in enumerate(['valid','test']):
157
221799d79188 Ajouté seed 100 à 207 aux ensembles de données générés
boulanni <nicolas_boulanger@hotmail.com>
parents: 136
diff changeset
10 for j,c in enumerate([0.3,0.5,0.7,1]):
136
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
11 l = str(c).replace('.','')
274
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
12 os.system("dbidispatch --condor --os=fc4,fc7,fc9 --machine=%s ./run_pipeline.sh -o %sdata/P%s_%s_data.ft -p %sdata/P%s_%s_params -x %sdata/P%s_%s_labels.ft -f %s%s_data.ft -l %s%s_labels.ft -c %socr_%s_data.ft -d %socr_%s_labels.ft -m %s -z 0.1 -a 0.1 -b 0.25 -g 0.25 -s %d -y %d" % (mach, dir1, l, s, dir1, l, s, dir1, l, s, dir1, s, dir1, s, dir1, s, dir1, s, c ,[20000,80000][i], 200+i*4+j))
136
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
13
255
17c72763d574 added PNIST generation script
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 203
diff changeset
14 #P07
136
4a78c9b83fee Added script to launch the ~108 data generation jobs with dbidispatch
boulanni <nicolas_boulanger@hotmail.com>
parents:
diff changeset
15 for i in range(100):
203
fd1b5237e49e Actual script used to launch images generation on 4 machines
boulanni <nicolas_boulanger@hotmail.com>
parents: 157
diff changeset
16 os.system("dbidispatch --condor --os=fc4,fc7,fc9 --machine=%s ./run_pipeline.sh -o %sdata/P07_train%d_data.ft -p %sdata/P07_train%d_params -x %sdata/P07_train%d_labels.ft -f %strain_data.ft -l %strain_labels.ft -c %socr_train_data.ft -d %socr_train_labels.ft -m 0.7 -z 0.1 -a 0.1 -b 0.25 -g 0.25 -s 819200 -y %d" % (mach, dir1, i, dir1, i, dir1, i, dir1, dir1, dir1, dir1, 100+i))
255
17c72763d574 added PNIST generation script
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 203
diff changeset
17
17c72763d574 added PNIST generation script
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 203
diff changeset
18 #PNIST07
17c72763d574 added PNIST generation script
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 203
diff changeset
19 for i in range(100):
17c72763d574 added PNIST generation script
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 203
diff changeset
20 os.system("dbidispatch --condor --os=fc4,fc7,fc9 --machine=%s ./run_pipeline.sh -o %sdata/PNIST07_train%d_data.ft -p %sdata/PNIST07_train%d_params -x %sdata/PNIST07_train%d_labels.ft -f %strain_data.ft -l %strain_labels.ft -c %socr_train_data.ft -d %socr_train_labels.ft -m 0.7 -z 0.1 -a 0.1 -b 0.25 -g 0.25 -s 819200 -y %d -t %d" % (mach, dir1, i, dir1, i, dir1, i, dir1, dir1, dir1, dir1, 100+i,1))
274
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
21
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
22
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
23
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
24 #P07
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
25 #for i in [90,94]:#[2,10,13,15,20,49,68,82,86,90,94]:
44409b6652aa small change in generating script (bug fix)
Xavier Glorot <glorotxa@iro.umontreal.ca>
parents: 255
diff changeset
26 #os.system("dbidispatch --condor --mem=3900 --os=fc4,fc7,fc9 --machine=maggie16.iro.umontreal.ca --machine=maggie15.iro.umontreal.ca --machine=zappa8@iro.umontreal.ca ./run_pipeline.sh -o %sdata2/P07_train%d_data.ft -p %sdata2/P07_train%d_params -x %sdata2/P07_train%d_labels.ft -f %strain_data.ft -l %strain_labels.ft -c %socr_train_data.ft -d %socr_train_labels.ft -m 0.7 -z 0.1 -a 0.1 -b 0.25 -g 0.25 -s 819200 -y %d" % (dir1, i, dir1, i, dir1, i, dir1, dir1, dir1, dir1,100+i))