Skip to the content ForoHackers
Log inJoin

High-performance computing: start here

High-performance computing
#427 · Ophuscado · · Report

High-performance computing uses many processors working together to solve problems too big or too slow for one machine: weather, molecules, structures, large simulations. It is parallel programming where performance is the goal, on clusters whose network, memory and storage matter as much as the cores.

Key concepts

  • Parallelism: data, task and pipeline parallelism, across cores, accelerators and nodes.
  • Message passing and shared memory: MPI between nodes, threads or OpenMP within one.
  • Scaling: strong scaling fixes the problem and adds processors; weak scaling grows both; Amdahl's law bounds the first.
  • Memory and communication: bandwidth, cache use and network latency usually limit speed more than arithmetic does.
  • Job scheduling: users submit jobs to a queue, and a scheduler allocates nodes and time.
  • Accelerators: GPUs handle massively parallel kernels and need data moved carefully.

Vocabulary

  • Node: one computer in a cluster.
  • Rank: one process in an MPI program.
  • FLOPS: floating-point operations per second, a peak figure rarely reached in practice.
  • Interconnect: the fast network joining nodes.
  • Roofline model: a chart relating a program's speed to its arithmetic intensity and the machine's limits.

What to study, in order

  1. C, C++ or Fortran, plus the basics of caches and vectorisation.
  2. Parallelise a loop with OpenMP and measure speedup against thread count.
  3. Write a small MPI program, such as a stencil or a matrix multiply, and measure strong and weak scaling.
  4. Profile it and find whether it is limited by compute, memory or communication.
  5. Run it through a job scheduler on a shared cluster, then port a kernel to a GPU.

Related tags

#concurrency #gpu #performance #computational-science #numerical-analysis

Log in or join to reply.