ANTLR Testing

For ANTLR v2

Development on this version has stopped, but it works acceptably to test ANTLR version 2.7 grammars.

Getting Started

Get antlrtesting-0.6.jar from the SourceForge download area.

You will probably need to use JUnit 3.

Testing a Lexer

The ANTLR-Testing library provides a test-case subclass called LexerTestCase. Use this as the superclass of ScannerTest instead of JUnit's TestCase. LexerTestCase is a subclass of JUnit's TestCase, so all of the methods you're used to are still available.

Your new test-case class must implement a method to return a new lexer; here's an example:

protected TokenStream makeLexer(String input) {
  return new TheLexer(new StringReader(input));
}

TheLexer is the name of the lexer you're writing in ANTLR.

To test a lexer production, use the assertToken() method. Here's an example for any standard INTEGER token:

public void testIntegerToken() throws TokenStreamException {
  assertToken(INTEGER, "123", "123");
}

• The first argument is the expected type of the token as specified in the lexer.
• The second argument is the expected text of the token.
• The third argument is the text to lex (lexify? scan!).

The second argument isn't always a complete waste:

public void testWhitespace() throws TokenStreamException {
  assertToken(INTEGER, "123", "  123");
  assertToken(INTEGER, "9", "  9  ");
}

Sometimes the output does differ from the input!

Now go to it!

Testing a Parser

The ANTLR-Testing library provides a test-case subclass called ParserTestCase. It's a subclass of JUnit's TestCase.

This superclass requires overriding two methods:

protected TokenStream makeLexer(String input) {
  return new TheLexer(new StringReader(input));
}
protected TestableParser makeParser(TokenStream lexer) {
  TheParser parser = new TheParser(lexer);
  parser.getASTFactory().setASTNodeClass("org.norecess.antlr.LineNumberAST");
  return parser;
}

The first method is the same as for the scanner's test-case class. Strictly speaking the AST factory isn't necessary in making the parser, but it's a special extra bonus for you.

You need to add some extra code to the parser in order for it to be properly testable. First, add this to the options of the parser:

classHeaderSuffix=org.norecess.antlr.TestableParser;

Second, add this block of code to the parser in your grammar file right after the options section:

{
  private boolean myFailure = false;
  private boolean myQuietErrors = false;
  public void reportError(RecognitionException ex) {
    if (shouldPrintErrors())
      super.reportError(ex);
    myFailure = true;
  }
  public boolean successfulParse() { return !myFailure; }
  public void atEOF() {
    try {
      match(EOF);
    } catch (Exception e) {
      myFailure = true;
    }
  }
  public void setQuietErrors(boolean quietErrors) {
    myQuietErrors = quietErrors;
  }
  private boolean shouldPrintErrors() {
    return !myQuietErrors;
  }
}

Technically, you only need to write a few of these methods, but using all of this code will satisfy all of the requirements of the methods.

(Incidentally, if anyone can figure out a better way to do this, I'd love to hear your ideas!)

Now you can start testing like this:

public void testExpression() {
  assertPreroder("parse 123", INTEGER, "(123)", produce("expression", "123"));
}

In general:

assertPreroder(message, expectedTokenType, expectedPreorderstring, producedAST);

• message is a standard JUnit message; describe the test.
• expectedTokenType is the expected token type of the root of the produced AST (tokens from the lexer and parser).
• expectedPreorderString is a String representation of a fully parenthesized preorder traversal of the expected AST.
• producedAST is the expression to be tested; helper methods make this easy to write.

To produce an AST from the grammar, use the produce(production, input) to try to produce an AST from input using production from the grammar. In the example above, the parser should have this production:

expression: INTEGER ;

Here's another example:

public void testDefinition() {
  assertPreorder("parse define x = 123",
    DECLARATION,
    "(define(x)(123))",
    produce("form", "define x = 123"));
}

assertPreorder does not test the token-types of the whole AST, only the root. So the AST here has a DEFINITION root. The children are seen in the preorder string.

Testing a Tree Builder

The ANTLR-Testing library provides a test-case subclass called TreeParserTestCase. It's a subclass of JUnit's TestCase class.

This superclass requires overriding three methods:

protected TokenStream makeLexer(String input) {
  return new TheScanner(new StringReader(input));
}

protected TestableParser makeParser(TokenStream lexer) {
  TheParser parser = new TheParser(lexer);
  parser.getASTFactory().setASTNodeClass("org.norecess.antlr.LineNumberAST");
  return parser;
}

protected TreeParser makeTreeParser() {
  return new TheTreeBuilder();
}

The first two methods should look familiar.

Assertions are now made with JUnit's standard assertEquals(); just make sure you override the equals(Object) methods of your tree-builder output objects!

Here's an example:

public void testInteger() {
  assertEquals(new IntegerTIR(123), build("expression", produce("expression", "123")));
}

IntegerTIR is an "integer tree-intermediate-representation" class of my own; build(treeProduction, ast) puts the specified AST through the specified tree-builder production. (produce() in this example is the same one used in the parser testing.)

Known Problems

Not all of the assertXXXX() methods I've written have messages. This should actually be easy to fix.

Some of the error reporting leaves a lot to be desired, especially when the reflection fails.