Splitting text into words sounds trivial until you try. Spaces are not enough: punctuation sticks to words, numbers contain dots, and it's is a word with an apostrophe inside it.
A small regular-expression tokenizer (a lexer, in compiler terms) for English-ish text:
import re
TOKEN = re.compile(r"[A-Za-z]+(?:'[A-Za-z]+)?|\d+(?:\.\d+)?|[^\sA-Za-z\d]")
print(TOKEN.findall("It's 3.14 o'clock, isn't it?"))
# ["It's", '3.14', "o'clock", ',', "isn't", 'it', '?']
Decisions hidden in that one regex:
- Is
isn'tone token, oris+n't? Some conventions split it, becausen'tcarries the negation. Neither is "correct"; it depends on what comes next in the pipeline. 3.14is one number, but3.at the end of a sentence is a number and a full stop.- It handles only ASCII letters. For other alphabets you need Unicode categories, and for languages written without spaces between words (Chinese, Japanese) whitespace tokenization is useless and the "tokenizer" is a real segmentation model.
The lesson I keep repeating to programmers: a lexer for a programming language gets to define its own rules. A tokenizer for natural language has to discover them, and the edge cases are the language.