Farewell to Google blogger for now...

Too bad that I have to switch away from Google blogger and move to Tumblr.  Tumblr's customizability and better designed API are the two reasons for my migration.

A really good tutorial on Lex and yacc

I'm teaching compilers, and came across this excellent tutorial on Lex and Yacc online.

Unix in Windows - the native way

Compiler is an old technology - that means that it is not a native citizen of Microsoft Windows. UOIT is still pre-dominantly a Windows-based environment in which student's laptops are imaged with Windows 7. I actually began to like Windows since Windows 7... It's.... shall we say, tolerable to an UNIX fanboy like me. So, if you are less adventurous and want to work for Compilers in Windows, you will need roll-up your sleeves and go through some very simple setup procedures.

You will need setup MinGW (minimal Gnu for Windows) environment, and then install a few MinGW packages using the mingw-get tool.

Installing MinGW:

Go and download the MinGW installer from its sourceforge site. The file is called:

mingw-get-inst-YYYYMMDD.exe 

where the YYYYMMDD is the timestamp reflecting the version of the install. At time of writing, the version is 20101030.

Run it. Choose gcc and g++. This will create the MinGW program group under the start > all programs.
Try out the MinGW unix shell, running natively under windows.

Start > All Programs > MinGW > MinGW Shell

[Optional] Setup some environment variables. This is done by going to the properties of your computer.
Create an environment var. "HOME" to a directory of your choice.
Create an environment var. "SHELL" with the value "/bin/bash". This creates a better experience at the command line.

Install MinGW packages:

1. Run the mingw shell.
2. mingw-get install msys-flex
3. mingw-get install msys-bison

That's it (for now).

Tricky bug to find in Antlr

I am scratching my head for an hour before I realized that my bug was due to a spelling mismatch between a lexer rule in the lexer grammar and the parser grammar. Antlr never complained that the mis-spelled rule is missing.  Only at the run-time does it crash complaining of invalid input.

So, take care with rule names when working with multiple files.

Modifying token text in ANTLR

When parsing markup languages, often, we want to keep only the text and strip away the markup syntactic artifacts.  Using the rule driven code execution, we can do this right in the lexer:

TAG: '<' (~'>')+ '>' {state.text = $text.substring(1, $text.length()-1);};

All done.

Emitting multiple tokens from a single lexer rule in ANTLR

Sounds crazy, but it's actually extremely useful.

http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497

In summary:

Lexer.emit(Token t) constructs the token and saves it somewhere.

Lexer.nextToken() gets the next token in the stream.

Manually controlling the look-ahead in semantic predicate

Example:

HTML

    : {input.LA(1) == '<' && input.LA(2) == 'h' && input.LA(3) == 't'}?=> ''

    ;

This rule is activated only if the next three characters are '<', 'h' and 't'.