Residual frame

Definition
A residual frame is a data structure used in digital video compression and image coding that contains the difference, or residual, between a predicted frame (derived from previously decoded frames or intra‑frame prediction) and the original source frame. Instead of storing the full pixel values of each frame, many video codecs encode only the residual information for frames that can be efficiently predicted, thereby reducing the amount of data required for transmission or storage.

Function in Video Coding

1. Prediction – Video codecs generate a predicted version of the current frame using motion compensation (inter‑prediction) or spatial prediction (intra‑prediction).
2. Residual Calculation – The residual frame is obtained by subtracting the predicted frame from the original frame on a per‑pixel basis.
3. Transformation and Quantization – The residual data are typically transformed (e.g., using discrete cosine transform or integer transforms) and quantized to further reduce redundancy.
4. Entropy Coding – The quantized coefficients are then entropy‑coded (e.g., CABAC, CAVLC) for inclusion in the bitstream.

During decoding, the residual frame is added back to the reconstructed prediction to recreate the original frame with acceptable fidelity.

Types of Frames Associated with Residuals

• P‑frames (Predictive frames): Carry residuals based on forward prediction from earlier reference frames.
• B‑frames (Bidirectional frames): Carry residuals derived from both past and future reference frames.
• I‑frames (Intra‑coded frames): May also include residuals when intra‑prediction is employed, though the term “residual frame” is most commonly associated with inter‑predicted frames.

Usage in Standards
Residual frames are integral to most modern video compression standards, including:

• H.264/AVC (Advanced Video Coding)
• H.265/HEVC (High Efficiency Video Coding)
• AV1 (AOMedia Video 1)
• VP9 and VP10

These standards define specific block sizes, transform types, and quantization parameters for handling residual data.

Historical Context
The concept of coding residual information emerged with the development of predictive video coding in the late 1970s and early 1980s. Early standards such as MPEG‑1 and MPEG‑2 incorporated residual coding, and subsequent generations refined the process to achieve higher compression efficiency.

Related Concepts

• Prediction error: Another term for the residual signal before transformation.
• Transform coding: The process applied to residuals to decorrelate spatial data.
• Rate‑distortion optimization: A technique that balances the amount of residual data transmitted against the resulting visual quality.

References

• ITU‑T Recommendation H.264 (02/2019), “Advanced Video Coding for Generic Audiovisual Services.”
• ISO/IEC 23008‑2:2019, “Information technology — High efficiency video coding.”
• A. M. B. et al., Video Codec Design, Springer, 2021.

The above description reflects the widely accepted technical understanding of residual frames in video compression literature.