Frame Re-ordering Support in iOS Video Encoding

Last year, I published a sample that showed how to use hardware accelerated video encoding on the iPhone by extracting the encoded data from the file during encoding. I’ve now updated the sample here to support frame re-ordering.

Frame re-ordering can improve compression ratios by allowing bi-directional prediction, and was enabled by Apple in iOS 7 for some devices. However, it means that the timestamps (received from the AVCaptureSession) are not in the same order as the output frames. I’ve previously published a fix for my demo that simply disabled frame re-ordering to ensure that the demo continued to work. I’ve now had the time to fix the problem properly.

The frames in the file are in decoding order. Each frame has a picture order count (POC) which indicates the presentation order — this is the original capture order and thus corresponds to the timestamp order. There are several ways to encode the POC in a slice header; my sample now correctly decodes type 0, which is the type used by the Apple encoder. The iPhone encoder creates POC values that increase by 2 for each frame. Rather than hardwire this knowledge into the code, I use the order of the POC values, not their absolute values. This means delaying output by up to 2 frames, increasing the latency by around 10%.

The updated sample is available here. Please get in touch if you have any comments or queries.

22nd April 2014