Mercurial > lcfOS

comparison python/ppci/c3/lexer.py @ 306:b145f8e6050b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Start on c3 rewrite
author Windel Bouwman
date Mon, 09 Dec 2013 19:00:21 +0100
parents 0615b5308710
children 084cccaa5deb
comparison
equal deleted inserted replaced
305:0615b5308710 306:b145f8e6050b
16 16
17 class Lexer: 17 class Lexer:
18 """ Generates a sequence of token from an input stream """ 18 """ Generates a sequence of token from an input stream """
19 def __init__(self, diag): 19 def __init__(self, diag):
20 self.diag = diag 20 self.diag = diag
21
22 def lex(self, source):
23 return self.tokenize(source)
21 24
22 def tokenize(self, input_file): 25 def tokenize(self, input_file):
23 """ 26 """
24 Tokenizer, generates an iterator that 27 Tokenizer, generates an iterator that
25 returns tokens! 28 returns tokens!