Saltar al contenido ForoHackers

AI safety: start here

Seguridad de la IA
#349 · Ophuscado · · Denunciar

AI safety is the technical work of making AI systems behave as intended and fail safely: reliable under unusual inputs, hard to misuse, honest about uncertainty, and pursuing the goals their builders actually meant. It matters because systems are deployed in settings where errors cost something, and because the more capable a system, the more a small misspecification can cost.

Key concepts

  • Specification: the objective you wrote is not the goal you had; proxies get optimised in unintended ways.
  • Robustness: behaviour under distribution shift, noise and inputs crafted to fool the model.
  • Misuse and abuse: thinking through how a capability could be used to harm, and limiting it at the system level.
  • Interpretability: methods that inspect what a model represents and why it produced an output.
  • Oversight: human review, monitoring and the ability to correct or stop a system; deciding when it must defer to a person.

Vocabulary

  • Alignment: how closely a system's behaviour matches the intentions of its users and designers.
  • Reward hacking: scoring well on the stated objective while missing its purpose.
  • Adversarial example: an input altered slightly to cause a wrong output.
  • Calibration: confidence that matches how often the model is right.

What to study, in order

  1. Machine learning basics, with attention to overfitting, distribution shift and evaluation.
  2. Calibration: plot reliability diagrams for a classifier and fix its overconfidence.
  3. Robustness: test a small model with noisy and shifted inputs, and measure how quickly it degrades.
  4. Interpretability: inspect the features or attention of a small model and test what you think you see.
  5. Practise risk analysis on a system you know: list failure modes, who is affected, and what controls would catch each.

Related tags

#machine-learning #ai-evaluation #ethics #reliability #llms

Inicia sesión o únete para responder.