Multimedia systems capture, compress, store, send and play sound, images and video. Raw media is enormous, so almost everything rests on codecs that throw away what people barely notice, and on pipelines that keep audio and video in step. Streaming adds the problem of delivering it smoothly over an unreliable network.
Key concepts
- Sampling and quantisation: turning continuous sound and light into numbers; sample rate and bit depth set the quality.
- Compression: lossless methods keep every bit; lossy codecs use transforms and perception to shrink data a great deal.
- Containers and codecs: the codec encodes the stream; the container packs audio, video and subtitles together with timestamps.
- Synchronisation: clocks and timestamps keep sound and picture aligned.
- Streaming: buffering, adaptive bit rate and loss handling for live and on-demand playback.
Vocabulary
- Bit rate: how many bits per second the stream uses.
- Frame types: key frames stand alone; other frames store only the changes from their neighbours.
- Latency and jitter: delay, and variation in delay, between sending and playing.
- Transcoding: converting from one format or size to another.
What to study, in order
- Sampling theory: why twice the highest frequency, and what aliasing sounds like.
- Write a WAV file by hand from a generated tone, and play it.
- Implement a simple lossy image compressor with a DCT and quantisation, and view the artefacts.
- Use a standard command-line media tool to inspect streams, convert formats and change bit rates.
- Build a small streaming setup and watch what loss and jitter do to the playback.
Related tags
#signal-processing #information-theory #computer-graphics #speech