Ken Pu

Farewell to Google blogger for now...

2012-01-06T17:06:00.000-08:00

Too bad that I have to switch away from Google blogger and move to Tumblr. Tumblr's customizability and better designed API are the two reasons for my migration.

A really good tutorial on Lex and yacc

2011-01-18T01:52:00.000-08:00

I'm teaching compilers, and came across this excellent tutorial on Lex and Yacc online.

Unix in Windows - the native way

2011-01-04T09:21:00.001-08:00

Compiler is an old technology - that means that it is not a native citizen of Microsoft Windows. UOIT is still pre-dominantly a Windows-based environment in which student's laptops are imaged with Windows 7. I actually began to like Windows since Windows 7... It's.... shall we say, tolerable to an UNIX fanboy like me. So, if you are less adventurous and want to work for Compilers in Windows, you will need roll-up your sleeves and go through some very simple setup procedures.

You will need setup MinGW (minimal Gnu for Windows) environment, and then install a few MinGW packages using the mingw-get tool.

Installing MinGW:

Go and download the MinGW installer from its sourceforge site. The file is called:

mingw-get-inst-YYYYMMDD.exe

where the YYYYMMDD is the timestamp reflecting the version of the install. At time of writing, the version is 20101030.

Run it. Choose gcc and g++. This will create the MinGW program group under the start > all programs.
Try out the MinGW unix shell, running natively under windows.

Start > All Programs > MinGW > MinGW Shell

[Optional] Setup some environment variables. This is done by going to the properties of your computer.
Create an environment var. "HOME" to a directory of your choice.
Create an environment var. "SHELL" with the value "/bin/bash". This creates a better experience at the command line.

Install MinGW packages:

Run the mingw shell.
mingw-get install msys-flex
mingw-get install msys-bison

That's it (for now).

Tricky bug to find in Antlr

2011-01-04T09:20:00.003-08:00

I am scratching my head for an hour before I realized that my bug was due to a spelling mismatch between a lexer rule in the lexer grammar and the parser grammar. Antlr never complained that the mis-spelled rule is missing. Only at the run-time does it crash complaining of invalid input.

So, take care with rule names when working with multiple files.

Modifying token text in ANTLR

2011-01-04T09:20:00.001-08:00

When parsing markup languages, often, we want to keep only the text and strip away the markup syntactic artifacts. Using the rule driven code execution, we can do this right in the lexer:

TAG: '<' (~'>')+ '>' {state.text = $text.substring(1, $text.length()-1);};

All done.

Emitting multiple tokens from a single lexer rule in ANTLR

2011-01-04T09:19:00.001-08:00

Sounds crazy, but it's actually extremely useful.

http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497

In summary:

Lexer.emit(Token t) constructs the token and saves it somewhere.

Lexer.nextToken() gets the next token in the stream.

Manually controlling the look-ahead in semantic predicate

2011-01-04T09:18:00.000-08:00

Example:

HTML

: {input.LA(1) == '<' && input.LA(2) == 'h' && input.LA(3) == 't'}?=> ''

;

This rule is activated only if the next three characters are '<', 'h' and 't'.

Installing Eclipse Antlr Plugin

2011-01-04T09:16:00.000-08:00

Antlr has an Eclipse plugin. I've had a lot of issues with plugin for Eclipse 3.6, so I would recommend to stick to Eclipse 3.5, and use the Antlr IDE version 2.0.2. You will need to follow these steps:

Install the dynamic language toolkit first from Eclipse.
Install the Antlr IDE from: http://antlrv3ide.sourceforge.net/updates/2.0.2

Setting up github repo

2010-12-12T10:00:00.000-08:00

You really need a Linux environment because Git has not been made Windows friendly (to my knowledge).

Install git, apt-get install git-core.
Create a project on github, say called "proj1".
On the local machine create a Git repository:

mkdir my-proj1
cd my-proj1
git init
touch README
git add README
git commit -m 'Creating the project'

Now you have a master branch. Create a remote repository to mirror this master branch.

git remote add github-repo git@github.com:kenpu/proj1.git
git push github-repo master

This starts a project hosted by Github.

Installing Twisted 10 on Windows

2010-12-05T08:12:00.000-08:00

I have been on Twisted 8.0 for the longest time. After getting a new laptop, I decided to setup my dev. environment with all new and shiny versions. So, I went to Twisted 10.2.0, and to a bit of a surprise, it didn't come with zope.interface. Here is a summary of what worked for me from a lot of blogs and forum search.

Download Twisted-10.2.0 for Python for Windows MSI from the Twisted download section.
Run it. It will install the Twisted packages, but not the zope interfaces used by Twisted.
Download setuptools-0.6c11 for Windows from the Python Package Index download section.
Now, you have an easy_install.exe in your Python\Scripts directory. It's a good idea to add Python\Scripts into your PATH.
Run easy_install zope.interface, and that will locate the proper zope.interface package from the repository.

All done.

Dropbox, Windows 7 Hibernation, and Startup cleanup...

2010-11-13T09:39:00.000-08:00

Okay, so I was handed a new laptop (Lenovo X201 Tablet) by our IT support as part of the perks of working at my university. I really like the laptop - well built, good battery life, and GREAT processor (Intel Core i7). It has Windows 7 running. While I am a Linux native, Windows 7 has managed to charm me in its vast improvement from the previous versions of Windows.

Now, the trouble starts: I installed Dropbox client among lots and lots of other my favourites (Google essential pack, mu-torrient client, etc.). So, a ton of things were added to the Windows 7. Then end result is that my windows can no longer to into sleep mode. The symptom is that it just hangs if the sleep mode is trigged. I didn't know which program is causing the hanging. This is what I did in the process of debugging:

Enabled hibernation option. This was not necessary to the eventual discovery of the problem.
powercfg /hibernate on
Downloaded an autoruns analysis problem. It lists all the auto-runs. I suspected that one of the programs I installed were causing the hanging.
Started to disable one after another using autoruns.

Viola! Dropbox client prevents Windows 7 to sleep for my Lenovo X201 tablet. I don't know why, but now, I live with manually starting dropbox client when I need it.

Ken

Integrating apache and google site via proxy

2010-09-27T12:43:00.000-07:00

Google site is really quite simple to how, and it looks surprisingly good. I've started using Google sites to organize some of my course material, and it works really well. I even started to use Google site to organize my own research activities, and I find that the non-flat structure really helps me to organize my thoughts, which makes it a preferred choice over Google doc.

Now, the down side: the URL is plain awful. My home page is called at http://leda.science.uoit.ca/kenpu, but the teaching sites are at http://sites.google.com/site/kenputeaching. People, typically, pay a few bucks a year and buy a domain name, and create a CNAME record with the DNS provider to link the domain name to the Google site, but this does not work for me, because I want the Google site to be a URL under my main site: leda.science.uoit.ca.

The solution is to configure my main server (running Apache 2) to act as a proxy server to the Google site. You might think that this should be simple to do, but it ended up eating up a couple of hours of my time. But now, it works... Here is my story of getting it working for me:

The software I use are:

Ubuntu 10.04 LTS as the OS.
Apache 2.2, the default version from apt-get install apache2.
mod_proxy, this comes with Apache 2, so you don't need to install it explicitly.
mod_proxy_html, you need to install this separately, but it should be easy enough with apt-get install libapache2-mod-proxy-html. This will install proxy_html 3.0. The latest is 3.1 which comes with some minor differences. I assume that we use proxy_html 3.0.

The google site I wish to integrate is called:

http://sites.google.com/site/kenputeaching

I want to map http://leda.science.uoit.ca/teaching to the google site. I found many documentations and tutorials (here, here and here). While they cover the basics, there were many pitfalls, that took me awhile to solve. For the impatients, here are the steps that worked for me.

Everything is done on my main machine leda.science.uoit.ca.

Map the URL /teaching to the Google site. This uses the ProxyPass directive of Apache2. In /etc/apache2/sites-enable/000-default, add (inside the section):

ProxyRequests OffProxyPass /teaching/ http://sites.google.com/site/kenputeaching/

If one is to visit http://leda.science.uoit.ca/teaching, one sees the Google site instead. However, some links are broken or links to http://sites.google.com/...
Next, we configure proxy_html to rewrite all occurrences of sites.google.com/site/kenputeaching/ to leda.science.uoit.ca/teaching. This is done by inserting the following lines after the ProxyPass.

ProxyHTMLURLMap http://sites.google.com/site/kenputeaching /teaching
  <Location /teaching/>    ProxyPassReverse http://sites.google.com/
    SetOutputFilter proxy-html
    ProxyHTMLURLMap /site/kenputeaching /teaching
    RequestHeader unset Accept-Encoding
  </Location>

Now, everything should work. Make sure that you restart the Apache server when the conf file is updated, and that you clear the browser cache when reloading a page.

For the more patient ones, these were the problems I wasted a lot of time figuring out:

SetOutputFilter proxy-html. The online documentation of proxy-html didn't really emphasize this, so I never sent the output for proxy-html to perform the rewriting. Lots of time was wasted trying to answer the question "Why isn't my rewrite rule being applied?"
Google site sends the data in compressed form, so proxy-html cannot parse the compressed HTML (at least not for version 3.0). This is addressed by RequestHeader unset Accept-Encoding.

PyLucene, multiprocessing and all that...

2010-02-22T18:17:00.001-08:00

It's been awhile since my last post. Hopefully, I will be better at keeping up with the writing.

I've been working with PyLucene for the past while, and in attempt to speed up search queries, I discovered quite a few things I didn't know before. I thought that some are worth sharing.

Lucene is just as much CPU intensive as it is disk intensive. The latter is more or less as expected, but I had no idea that Lucene requires so much CPU cycles. So when planning to distribute your Lucene queries over multiple threads / processes, think in terms of the number of cores of your server. Two dual core CPU implies four cores. So having more than four Lucene threads won't really help anymore. This is especially true of your queries are complex.
I tried to use Python's multiprocessing package to perform distributed PyLucene search. PyLucene requies lucene.initVM() call which isn't multiprocess safe. It created a ton of headaches for me. In the end, I concluded that:
You can only include lucene package and call lucene.initVM(...) in your worker processes. So the main process cannot call lucene.initVM(...)
Python processes are heavy, so don't use them for just a single query. Instead, use one multiprocess.Queue to pass a stream of queries to the worker threads, and collect the search results using another multiprocessing.Queue.

That's it for today. Baby's getting fussy now...

Pylucene tips and tricks

2009-10-02T10:45:00.000-07:00

PyLucene is the Lucene interface to the Python language. Here are a few how-to's for PyLucene.

Load Lucene


import lucene
lucene.initVM(lucene.CLASSPATH)

Open an index for writing


writer = lucene.IndexWriter("/home/lucene/index", lucene.StandardAnalyzer())

Open an index for searching


searcher = lucene.IndexSearcher("/home/lucene/index");

Create a document


doc = lucene.Document()
doc.add(lucene.Field(
  "title", 
  "This is a long title for an essay",
  lucene.Field.Store.YES, 
  lucene.Field.Index.TOKENIZED))

Put the document into the index
Don't forget to optimize the index before closing. It merges multiple segment files created during the writer.addDocument() phase. This will significantly improve the search time during querying phase.


writer.addDocument(doc)
...
writer.optimize()
writer.close()

Making a query
When working in Python, it's particularly convenient to build queries from a string using the Lucene query syntax. However, you may look into building queries pragmatically using the more basic building like lucene.TermQuery, lucene.BooleanQuery etc.


parser = lucene.QueryParser("title", lucene.StandardAnalyzer())
query = parser.parse("+happy movie year:1990")

Searching the index


hits = searcher.search(query)
for h in hits:
  hit = lucene.Hit.cast_(h)
  id, doc = hit.getId(), hit.getDocument()
  return (id, doc)

Getting the fields in a document


// doc is a PyLucene document
for f in doc.getFields():
  field = lucene.Field.cast_(f)
  (k, v) = field.name(), field.stringValue()
  print "field name = %s, its value = %s" % (k,v)

2009-09-22T22:55:00.000-07:00

Where are my packages? If you are programming in Python and using a Debian distribution, then you may from time to time wonder where the hack the source for your favourite Python packages are in fact located. Many of us don't know off the top of our head because we are dumb'ed by the wonderful apt and easy_install tools for quick-n-dirty installation of Python packages.
Here is a fast and sure way of locating the source of your packages: use the __file__ attribute of all imported package. Consider the scenario that you need to look up django.contrib.auth source code to see how user password is encrypted by the Django Web application framework.


>>> import django.contrib.auth
>>> django.contrib.auth.__file__
'/usr/local/lib/python2.6/dist-packages/django/contrib/auth/__init__.pyc'

There, it is where it is. Now you can vim your way through the code...

Stealing data from the Web

2009-08-18T09:08:00.000-07:00

The Web hosts so much data (often referred to as the Deep Web) that if we can freely query it, one can't even begin to imagine the questions one can ask. Many research papers have been written on how to harness the Deep Web. See Communications of ACM, Volume 50, Issue 5, May 2007. But if you actually sit down and try to pull data off other sites, you very quickly hit many walls. These walls are setup to protect the data from data collecting crawlers, so that only users through Web browsers can get to them. I'll just talk about two of the walls: protection by POST and COOKIES. I will use wget, but you can use a number of alternatives: curl, httpclient for Java, urllib2 for Python.

1. Figure out what POST parameters they are expecting. Use a Firefox plugin, called Firebug to monitor the HTTP headers being sent to the Web page that you are trying to pull data off of.

2. Mimic browser with wget using the --user-agent parameter.

3. Send the expected POST parameter using --post-data 'name=value&name=value...'

4. Send the expected COOKIE parameters using --load-cookie file. But this is actually somewhat inconvenient because you have to prepare the cookie file. I prefer to use --no-cookies --header "Cookie: name=value" instead.

Many years ago, when I was looking to buy a house, I was crawling www.mls.ca site on a nightly basis to mine for a good deal. Well, nowadays, what I wrote using PERL + WGET has become a standard feature offered by any decent realstate agent.

My work station setup

2009-07-03T20:16:00.000-07:00

I recently purchased a new workstation with plenty of RAM and lots of hard-drive space. So, what should I put on it? I guess it depends on what one does for work. I do a few things:

Write lots of code as rapidly as I can manage. So I program a lot in Python and Java. Furthermore, the type of software I write deals with large volumes of data and tends to run in server mode. Ubuntu please.
Write articles using LaTeX. Again, I prefer to do it in a Linux environment.
Write presentation slides using PowerPoint. This is mainly for my courses. I actually enjoy making good slides for my lectures in Prosper. Unfortunately, time is not on my side, so I'm somewhat forced to use Office.
Filling up forms (endlessly) in Office 2007 format.
Save cost on software licenses.

So you can see, I'm sort of stuck between Linux and Windows. Mac OS is out of the question because my workstation is all PC. Below is a step-by-step description of how I setup my machine.

Download Ubuntu 9.04 Desktop ISO image, and burn it on to a CD/DVD.
Install Ubuntu. I allocated the entire drive to Ubuntu (ext3 and swap) partitions. Don't worry, we will install Windows still.
Install build-essential, sun-java6-sdk for the basic programming needs.
Install texlive-latex and all the fonts, extra and utils packages for my technical writing.
Install various scientific packages for my work.
Install samba, openssh-server for remotely accessing my workstation
Samba is important for Windows (to be installed later) to share files with Ubuntu.
Setup samba:
- edit /etc/samba/smb.conf to enable home directory sharing.
- create samba user using smbpasswd -a
- test samba accessibility with the command
smbclient //localhost/
Download vmware-server 2.0 from VMWARE. Here is a good guide for Ubuntu 8.10, but it applies to 9.04 as well. It's worth noting that vmware-server has changed completely - the system console is GONE. It's replaced with Web interface which requires Firefox plug-in to function.
Install Windows XP service pack 2 as a guest OS. I have chosen to use NAT for the network adaptor of the guest OS. When NAT is used, the host OS takes 192.168.187.1, while the guest OS's take on higher IP addresses 192.168.187.x.
You will first find that the arrow keys don't work. Here is a fix.
Now, you need to have XP and Ubuntu share files. It's easy with Samba.
In Windows, create a new network place, and use \\192.168.187.1\, and specify login as and the password.
I had to also install Office 2007 for sharing documents with co-workers.

There done: a workstation running Ubuntu and Windows happily.

Ubuntu 9.04 upgrade, MySQL and Appamor

2009-06-16T08:54:00.000-07:00

My Ubuntu server was getting old, so I decided to upgrade it to the latest version (9.04), only to find that MySQL no longer starts. Long story short: I had my datadir set to somewhere else, not registered with AppAmor which is now used by Ubuntu.

There are two solutions:

Configure appamor - (see here) so it allows MySQL server process to access the data directory.
Disable appamour altogether (see here) so the traditional UNIX file permission model prevails again:
```
sudo /etc/init.d/apparmor stop
sudo update-rc.d -f apparmor remove
```

Google Android dev phone with an existing Rogers plan

2009-05-27T12:37:00.000-07:00

I have had a cheap Rogers plan (125 min) for the past 5+ years. I always stayed away from the fancy data plans, Internet access, or even text messaging - call me old-fashioned. But the new Android platform and its open development environment forced me to see smart phones in a new light. So I went and bought a Google Android dev phone, (T-mobile G1) from the Android market. After hours of fiddling and following some well written blogs (this one and this one ) about activating the T-mobile G1 phone in Canada, I still get the "Unable to communicate with Google server" message. Finally, this is what solved my problem.

Put the SIM card back to my old Sony Ericsson phone.
Select SETTINGS > MASTER RESET
Take out the SIM card from the Sony phone, and put it back to the Google phone.
Delete all APN settings, and recreate the two APN's: Rogers and Rogers MMS using the APN settings. See
here for details.

Viola. It works.

Python Twisted: bring RFID, wireless sensors and Web data together

2009-05-23T14:20:00.000-07:00

I have a pretty cool job (Google me, and you will see). I get paid to play around with neat things to create even neater things - all in the name of Science and Research. I'll describe the neat things I have been using, and we can get into the neater things we are working on.

Neat thing #1: Data Streams

[RFID data streams]
At the lab, we have RFID equipment: tags and readers. Active tags are especially interesting because they can carry dynamic data such as environmental temperature reading and vibration detection.

[Wireless sensor data streams]
Equally interesting to work with are the various wireless sensors we purchased from Point Six (Google Point Six). One of the most useful sensors we deployed is the power sensors, allowing us to calculate the energy cost used by various electrical appliances, from computing servers to desk lamps, down to the cent.

[Webcam data streams]
We've been gathering real-time webcam feeds, especially traffic cameras around Southern Ontario. At 1 frame per 2 seconds, with a few hundred cameras, the volume of incoming data from the traffic Webcams is actually quite high.

Neat thing #2: Python Twisted

I had a student working with me for summers over the past two years. We tried a bunch of distributed infrastructure to manage the data streams to support:
- Live feed to Web portal
- Ad-hoc user queries
- Historic archival for user queries
We tried to develop our own Java RMI based solution which was than extended to XML-RPC Web services, but none offered the flexibility, scalability and robustness that I wanted. In the past year, I started looking at a Python Twisted framework, and sure enough, it is definitely the right thing for our problem. The biggest problem with our previous approaches is the treatment of concurrency using threads. Threads fundamentally limits the number of concurrent data streams that each process can handle. Furthermore, we constantly needed to worry about lockings when accessing shared databases. What we really needed was a turn-key event loop system that supports asynchronized I/O for sockets, files, HTTP, etc. And that's exactly what Twisted was built to do. So, thanks Twisted Lab.

The event neater thing we are working on:

We try to build distributed archival service that acts as the storage the heterogeneous data streams and a query processor for ad-hoc user queries. Here is the summary of our solution implemented using Python Twisted.

For image streams from Web cameras, we use
```
 twisted.web.client 
```
to query the remote Web pages for image download. It nicely integrates with the main event loop using
```
 twisted.reactor.callLater 
```
Thanks to this blog for the example of continuously polling a data source.

For the wireless sensors, the receiver communicates using the RS-232 serial port. I very thankful that the Python community has not forgotten the good-old COM ports. Python supports serial port for Linux and Windows with the package pyserial . Twisted provides the special event loop:
```
 twisted.internet.serialport 
```
to interface with Win32 or Linux posix serial port communication.

The RFID readers communicate via TCP/IP in either the push model (readers are clients) or the pull model (readers are servers). Either way, Twisted can handle it. In the push mode, we have a Protocol to handle the data coming in. Conversely, in the pull model, we implemented a TCP clien twisted.internet.tcp.Client to probe the RFID readers.

The event-based framework of Twisted offered several advantages over the thread-based Java solution. It's proven to be more scalable: we can handle about 30 connections using just one Twisted instance. We are also able to fire up multiple instances to load balance.

As an added advantage, we get for free the different modes for user to interact with the archived data: Web front end (twisted.web2), Web services (twisted.web.xmlrpc), SSH (twisted.conch), and even using Chat clients (twisted.words.protocols).

Baby cries

2009-02-01T18:56:00.001-08:00

My baby cries endlessly.