Fixes decoding of 9/10 bit per pixel h264 videos as reported in
https://bugzilla.libav.org/show_bug.cgi?id=404 .

This function used to be shared with the xxan decoder, but that is not
true anymore.

The background changes from 128 (used by the default/cmdutils
get_buffer()) to 0. This looks more correct.

Do not overwrite linesize set by get_buffer().

The last frame in the FATE test is not decoded anymore, since the file
is cut and a part of it is missing.

The description as updated in 6e5c44b0f made it sound like it is
handled exactly like the pts, which it is not. Clarify that it
is the packet that triggered returning the frame instead of just
the last avpacket (in case of frame threading, it won't be the
dts of the last packet).

Signed-off-by: Martin Storsjö <martin@martin.st>

Remove some pointless comments and empty lines.

Using a frame count, as is done currently, does not work at all with
seeking. Instead, when the number of samples in the final frame has
been decoded, we check if only the 32-bit CRC is remaining. If so, we
assume that it is the final frame.

There is no longer a need to keep total_frames in TTAContext.

Sometimes the extradata has duplicate atoms, but that shouldn't prevent
decoding. Just ensure that it is at least 36 bytes as a sanity check.

CC: libav-stable@libav.org

Fixes Bug 402

Across the many retouches those did not make the main commit.

Move some functions from dsputil. The idea is that videodsp contains
functions that are useful for a large and varied set of video decoders.
Currently, it contains emulated_edge_mc() and prefetch().

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Fixes CVE-2012-2803.

CC: libav-stable@libav.org

Fixes bug 396.

CC: libav-stable@libav.org

The VC1 decoder uses edge_emu_buffer simultaneously for luma and chroma
and needs more space. That was not a problem before f1d8763a02
since the size for edge_emu_buffer was always calculated with 2 byte per
pixel since the linesize was not known.

Fixes occasionally fate errors in vc1_sa10143.

Also adds forgotten Changelog entry.

Fixes CVE-2012-2782.

Since we can't know which stride a custom get_buffer() implementation is
going to use we have to allocate this scratch buffers after the linesize
is known. It was pretty safe for 8 bit per pixel pixel formats since we
always allocated memory for up to 16 bits per pixel. It broke hoever
with cmdutis.c's alloc_buffer() and high pixel bit depth since it
allocated larger edges than mpegvideo expected.

Fixes fuzzed sample nasa-8s2.ts_s244342.

Fixes null pointer dereference.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

Compute dist_scale_factor_field only for MBAFF since that is the only
case in which it is used.

Prevents writing beyond array bounds.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>

It is not posible to call get_buffer during frame-mt codec
initialization. Libavformat might pass huge amounts of data as
extradata after parsing broken files. The 'extradata' for the fuzzed
sample sample_varPAR_s5374_r001-02.avi is 2.8M large and contains
multiple slices.

Introduced in d7d6efe42b.

Since a NAL_DPA can start a new frame it has to be handled before
ff_thread_finish_setup is called.

Fixes CVE-2012-2783

CC: libav-stable@libav.org

Returning 0 for failure is misleading.

CC: libav-stable@libav.org

Fixes CVE-2012-2791.

CC: libav-stable@libav.org

This requires to move the avcodec_default_free_buffers() call to
ff_MPV_common_end() since otherwise delayed pictures would get freed
during a size change.

Direct rendering capable decoders call get_buffer() which will set the
frame parameters.

Prevents frames with wrong parameters when a decoder outputs delayed
frames after a resolution or pixel format change.

Fixes a crash in the fuzzed sample sample_varPAR.avi_s26638 with
alternating bit depths.