Skip to the content ForoHackers
Log inJoin

High-performance computing: start here

High-performance computing
#2933 · · Quote · Report

High-performance computing uses many processors working together to solve problems too big or too slow for one machine: weather, molecules, structures, large simulations. It is parallel programming where performance is the goal, on clusters whose network, memory and storage matter as much as the cores.

Key concepts

  • Parallelism: data, task and pipeline parallelism, across cores, accelerators and nodes.
  • Message passing and shared memory: MPI between nodes, threads or OpenMP within one.
  • Scaling: strong scaling fixes the problem and adds processors; weak scaling grows both; Amdahl's law bounds the first.
  • Memory and communication: bandwidth, cache use and network latency usually limit speed more than arithmetic does.
  • Job scheduling: users submit jobs to a queue, and a scheduler allocates nodes and time.
  • Accelerators: GPUs handle massively parallel kernels and need data moved carefully.

Vocabulary

  • Node: one computer in a cluster.
  • Rank: one process in an MPI program.
  • FLOPS: floating-point operations per second, a peak figure rarely reached in practice.
  • Interconnect: the fast network joining nodes.
  • Roofline model: a chart relating a program's speed to its arithmetic intensity and the machine's limits.

What to study, in order

  1. C, C++ or Fortran, plus the basics of caches and vectorisation.
  2. Parallelise a loop with OpenMP and measure speedup against thread count.
  3. Write a small MPI program, such as a stencil or a matrix multiply, and measure strong and weak scaling.
  4. Profile it and find whether it is limited by compute, memory or communication.
  5. Run it through a job scheduler on a shared cluster, then port a kernel to a GPU.

Related tags

#concurrency #gpu #performance #computational-science #numerical-analysis

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply