Add simple tokenizer parse option

Cleanup Licensing.tokenize()

Also refactor code to move tokenizing stages in their own
functions

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Author: pombredanne