Skip to the content ForoHackers
Log inJoin

Tokenization is a grammar problem: why "it's" is not always one token

CompilersInterpreters and virtual machinesNatural language processing
#1074 · polyglot_parser · · Report

Splitting text into words sounds trivial until you try. Spaces are not enough: punctuation sticks to words, numbers contain dots, and it's is a word with an apostrophe inside it.

A small regular-expression tokenizer (a lexer, in compiler terms) for English-ish text:

import re

TOKEN = re.compile(r"[A-Za-z]+(?:'[A-Za-z]+)?|\d+(?:\.\d+)?|[^\sA-Za-z\d]")

print(TOKEN.findall("It's 3.14 o'clock, isn't it?"))
# ["It's", '3.14', "o'clock", ',', "isn't", 'it', '?']

Decisions hidden in that one regex:

  • Is isn't one token, or is + n't? Some conventions split it, because n't carries the negation. Neither is "correct"; it depends on what comes next in the pipeline.
  • 3.14 is one number, but 3. at the end of a sentence is a number and a full stop.
  • It handles only ASCII letters. For other alphabets you need Unicode categories, and for languages written without spaces between words (Chinese, Japanese) whitespace tokenization is useless and the "tokenizer" is a real segmentation model.

The lesson I keep repeating to programmers: a lexer for a programming language gets to define its own rules. A tokenizer for natural language has to discover them, and the edge cases are the language.

#1650 · regex_regret · · Report

Approved use of regex: the lexer. It is a regular language, the tool fits, and nobody is trying to parse HTML. I will allow it.

#1651 · lambda_llama · · Report

The [^\sA-Za-z\d] catch-all at the end is my favourite kind of rule: "anything else is a one-character token". It means the lexer never fails, which is the other half of good error handling: fail later, with a better message.

#1652 · rubber_duck_supervisor · · Report

Question back: what does your tokenizer do with e-mail, state-of-the-art and well-known? Is a hyphen inside a word the same thing as a minus sign? How would you tell?

#1653 · packet_loss_poet · · Report

Token or two words? The apostrophe asks again. The lexer shrugs, splits.

Log in or join to reply.