Mercurial > eagle-eye

annotate ikweb/tools/lazy/www/__init__.py @ 366:2cfa03edc22e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
ignore island we don't know yet
author "Rex Tsai <chihchun@kalug.linux.org.tw>"
date Sun, 01 Mar 2009 11:10:52 +0800
parents 7747bbe5b68e
children
rev   line source
247
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
1 # -*- coding: utf-8 -*-
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
2 #
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
3 # Copyright 2008 Hsin Yi, Chen
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
4 """
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
5 [Note] the project is not available yet.
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
6
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
7 A web page fetcing tool chain that has a JQuery-like selector and supports chain working.
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
8
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
9 Here is an exmaple can show the the main idea, To restrive a content you want
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
10 in a div box in a web page, and then post and restrive next content in the other
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
11 web page with the param you just maked from the content in first restriving.
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
12 finally, storage the production.
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
13
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
14 def func(s):
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
15 return {'msg':s}
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
16
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
17 try:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
18 c("http://example.tw/").get().find("////ul/text()") \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
19 .build_param( func ).post_to("http://example2.com") \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
20 .save_as('hellow.html')
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
21 except:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
22 pass
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
23
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
24 more complex example
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
25
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
26 try:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
27 c("http://example.tw/").retry(4, '5m').get() \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
28 .find("#id > div"). \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
29 .build_param( func ).post_to("http://example2.com") \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
30 .save_as('hellow.html') \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
31 .end().find("#id2 > img").download('pretty-%s.jpg'). \
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
32 tar_and_zip("pretty_girl.tar.gz")
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
33 except NotFound:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
34 print "the web page is not found."
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
35 except NoPermissionTosave:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
36 print "the files can not be save with incorrect permission."
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
37 else:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
38 print "unknow error."
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
39 """
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
40 from lazy.www.work import WorkFlow
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
41 from lazy.www.work.fetch import Fetcher
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
42 from lazy.www.work.storage import FileStorager
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
43 from lazy.www.core import SemiProduct
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
44 import os
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
45 import sys
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
46 import re
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
47
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
48 def parse_scheme(scheme):
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
49 try:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
50 return re.findall("(\w+):\/\/(.*\/?)",scheme)[0]
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
51 except:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
52 sys.stdout.write("the scheme is not supported.")
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
53 sys.exit()
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
54
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
55 def c(scheme, worker=None):
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
56 """
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
57 connect to a web apge
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
58
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
59 >>> c('http://localhost:8080').get().worker.working_product.content
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
60 'It works!!\\n'
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
61
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
62 >>> c('http://localhost:8080').get().find('//text()')
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
63 'It works!!\\n'
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
64 """
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
65 target_type, path = parse_scheme(scheme)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
66
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
67 #@todo: SemiProduct Factory.
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
68 if worker is None:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
69 if 'file' == target_type:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
70 s= SemiProduct(source=path)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
71 worker = FileStorager(s)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
72 else:
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
73 s= SemiProduct(source=scheme)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
74 worker = Fetcher(s)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
75
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
76 return WorkFlow(worker)
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
77
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
78 if __name__ == '__main__':
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
79 import doctest
7747bbe5b68e start to develope Information Exchange Center of Ikariam Game. (prototpye)
"Hisn Yi, Chen <ossug.hychen@gmail.com>"
parents:
diff changeset
80 doctest.testmod()