Skip to the content ForoHackers
Log inJoin

AI safety: start here

AI safety
#2855 · · Quote · Report

AI safety is the technical work of making AI systems behave as intended and fail safely: reliable under unusual inputs, hard to misuse, honest about uncertainty, and pursuing the goals their builders actually meant. It matters because systems are deployed in settings where errors cost something, and because the more capable a system, the more a small misspecification can cost.

Key concepts

  • Specification: the objective you wrote is not the goal you had; proxies get optimised in unintended ways.
  • Robustness: behaviour under distribution shift, noise and inputs crafted to fool the model.
  • Misuse and abuse: thinking through how a capability could be used to harm, and limiting it at the system level.
  • Interpretability: methods that inspect what a model represents and why it produced an output.
  • Oversight: human review, monitoring and the ability to correct or stop a system; deciding when it must defer to a person.

Vocabulary

  • Alignment: how closely a system's behaviour matches the intentions of its users and designers.
  • Reward hacking: scoring well on the stated objective while missing its purpose.
  • Adversarial example: an input altered slightly to cause a wrong output.
  • Calibration: confidence that matches how often the model is right.

What to study, in order

  1. Machine learning basics, with attention to overfitting, distribution shift and evaluation.
  2. Calibration: plot reliability diagrams for a classifier and fix its overconfidence.
  3. Robustness: test a small model with noisy and shifted inputs, and measure how quickly it degrades.
  4. Interpretability: inspect the features or attention of a small model and test what you think you see.
  5. Practise risk analysis on a system you know: list failure modes, who is affected, and what controls would catch each.

Related tags

#machine-learning #ai-evaluation #ethics #reliability #llms

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply