Signed-off-by: Paul B Mahol <onemda@gmail.com>

This fixes the colour of the transparent background (as seen with ffplay),
and makes the background of some non-keyframes transparent that was
incorrectly shown as opaque for some samples.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Across the many retouches those did not make the main commit.

Move some functions from dsputil. The idea is that videodsp contains
functions that are useful for a large and varied set of video decoders.
Currently, it contains emulated_edge_mc() and prefetch().

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

These where reduced in f1d8763a02
and this change can cause out of array accesses.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2012-2803.

CC: libav-stable@libav.org

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Based on code by Janne Grunau

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Based on a patch by Janne Grunau
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Based on patch by Janne Grunau

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes bug 396.

CC: libav-stable@libav.org

The VC1 decoder uses edge_emu_buffer simultaneously for luma and chroma
and needs more space. That was not a problem before f1d8763a02
since the size for edge_emu_buffer was always calculated with 2 byte per
pixel since the linesize was not known.

Fixes occasionally fate errors in vc1_sa10143.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Also adds forgotten Changelog entry.

Fixes CVE-2012-2782.

Since we can't know which stride a custom get_buffer() implementation is
going to use we have to allocate this scratch buffers after the linesize
is known. It was pretty safe for 8 bit per pixel pixel formats since we
always allocated memory for up to 16 bits per pixel. It broke hoever
with cmdutis.c's alloc_buffer() and high pixel bit depth since it
allocated larger edges than mpegvideo expected.

Fixes fuzzed sample nasa-8s2.ts_s244342.

Fixes null pointer dereference.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

Compute dist_scale_factor_field only for MBAFF since that is the only
case in which it is used.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Prevents writing beyond array bounds.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>

It is not posible to call get_buffer during frame-mt codec
initialization. Libavformat might pass huge amounts of data as
extradata after parsing broken files. The 'extradata' for the fuzzed
sample sample_varPAR_s5374_r001-02.avi is 2.8M large and contains
multiple slices.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes Ticket2040

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Peter Ross <pross@xvid.org>

Signed-off-by: Peter Ross <pross@xvid.org>

Fixes decoding regression

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

This prevents a null pointer dereference

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes Ticket1990

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Fixes assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

RealPlayer does not play rv20 files when sizes are not multiple of 4

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Introduced in d7d6efe42b.

Since a NAL_DPA can start a new frame it has to be handled before
ff_thread_finish_setup is called.

Fixes out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>