menu

Handling Comments In Antlr

Posted by mb0 on November 24th, 2005

problem :

  1. comment tokens should be skipped - they can be anywhere
  2. i need comments and their regions for the codefolding

i read some articles about hidden token streams but this solution is some kind of lame i think. so i thought it over a while and my solution is quit easy and exactly what i need but a bit hackisch. i declare a list and two methods (void addComment(Token) and List getComments() ) in the lexer and before i set the token type to be skipped i simply add the comments (ML_COMMENT uses the same solution) 

SL_COMMENT 
: 	"//" ( ~('n'|'r'))* ('n'|'r'('n')? )?
	{ 	Token _mytoken = makeToken(_ttype);
		_mytoken.setText(
			new String(text.getBuffer(),
			_begin,text.length()-_begin));
		addComment(_mytoken);
		$setType(Token.SKIP);
		newline();
	}
;

now that i have a token list i can get all comment tokens after parsing the document. i build my custom ast"s with the tokens and add them to my custom model in which the whole ast is wrapped. 

AS2Lexer lexer = new AS2Lexer(inputStream);
AS2Parser parser = new AS2Parser(lexer);
ASSourceASTFactory factory = new ASSourceASTFactory();
ASSourceAST compilationUnit = (ASSourceAST) parser.getAST();
Iterator commentsIter = lexer.getComments().iterator();
while(commentsIter.hasNext()){
	Token t = (Token)commentsIter.next());
	ASSourceAST ast =(ASSourceAST)factory.create(t);
	//...
}

Comments