Skip to the content ForoHackers
Log inJoin

Natural language processing: start here

Natural language processing
#337 · Ophuscado · · Report

Natural language processing is getting computers to work with human language: splitting it, labelling it, searching it, translating it, answering questions about it. Language is ambiguous, full of context and different in every language and dialect, which is what makes the field both hard and endlessly useful.

Key concepts

  • Text preparation: normalisation, sentence and word splitting, sub-word tokens, and the encoding issues that come before everything.
  • Representations: bag of words and TF-IDF, then dense word and sentence embeddings that place similar meanings close together.
  • Core tasks: classification, sequence labelling such as named entities, parsing, summarisation, translation, question answering.
  • Language models: scoring or generating text; the same idea runs from n-grams to neural models.
  • Multilinguality: scripts, morphology and word order differ, and low-resource languages have little data.

Vocabulary

  • Corpus: a collection of text used for training or study.
  • Token: the unit a system reads, a word, a sub-word piece or a character.
  • Part-of-speech tag: the grammatical category of a word, such as noun or verb.
  • Named entity: a span naming a person, place, organisation or date.

What to study, in order

  1. Regular expressions and Unicode: handle accents, scripts and messy input without breaking.
  2. Count words in a corpus, apply TF-IDF and build a simple search and a classifier on top.
  3. Train word embeddings on a small corpus and explore nearest neighbours and their biases.
  4. Build a sequence tagger for named entities and measure precision, recall and F1 per class.
  5. Evaluate a translation or summarisation system with several metrics, and read the outputs by hand.

Related tags

#llms #information-retrieval #machine-learning #speech #ai-evaluation

Log in or join to reply.