High-performance computing uses many processors working together to solve problems too big or too slow for one machine: weather, molecules, structures, large simulations. It is parallel programming where performance is the goal, on clusters whose network, memory and storage matter as much as the cores.
Key concepts
- Parallelism: data, task and pipeline parallelism, across cores, accelerators and nodes.
- Message passing and shared memory: MPI between nodes, threads or OpenMP within one.
- Scaling: strong scaling fixes the problem and adds processors; weak scaling grows both; Amdahl's law bounds the first.
- Memory and communication: bandwidth, cache use and network latency usually limit speed more than arithmetic does.
- Job scheduling: users submit jobs to a queue, and a scheduler allocates nodes and time.
- Accelerators: GPUs handle massively parallel kernels and need data moved carefully.
Vocabulary
- Node: one computer in a cluster.
- Rank: one process in an MPI program.
- FLOPS: floating-point operations per second, a peak figure rarely reached in practice.
- Interconnect: the fast network joining nodes.
- Roofline model: a chart relating a program's speed to its arithmetic intensity and the machine's limits.
What to study, in order
- C, C++ or Fortran, plus the basics of caches and vectorisation.
- Parallelise a loop with OpenMP and measure speedup against thread count.
- Write a small MPI program, such as a stencil or a matrix multiply, and measure strong and weak scaling.
- Profile it and find whether it is limited by compute, memory or communication.
- Run it through a job scheduler on a shared cluster, then port a kernel to a GPU.
Related tags
#concurrency #gpu #performance #computational-science #numerical-analysis