s->ref_pics[i] is later used as ref argument of interpolate_refplane,
where it is dereferenced.

If it is NULL, it causes a segmentation fault.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d93181ef3e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

If bytes is large enough, bytes*8 can overflow and become negative.

In that case 'bufsize -= bytes*8' causes bufsize to increase instead of
decrease.

This leads to a segmentation fault.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9e66b39aa8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

buf_idx + data_unit_size can overflow, causing the '> buf_size' check to
wrongly fail.

This causes a segmentation fault.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 984f50deb2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

In init_planes p->xblen and p->yblen are set to:
            p->xblen = s->plane[0].xblen >> s->chroma_x_shift;
            p->yblen = s->plane[0].yblen >> s->chroma_y_shift;

These are later used as block_w and block_h arguments of
s->vdsp.emulated_edge_mc. If one of them is 0 it triggers an av_assert2
in emulated_edge_mc:
    av_assert2(start_x < end_x && block_w > 0);
    av_assert2(start_y < end_y && block_h > 0);

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 75fc81c831)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes very long but finite loop
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5145d22b88)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes integer overflow
Fixes out of array read
Fixes: asan_heap-oob_107866c_42_041.drc
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 526886e606)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: CSA
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Use the videodsp function pointer instead.

Signed-off-by: James Almer <jamrial@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: James Almer <jamrial@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Lou Logan <lou@lrcd.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes out of array accesses for non default buffers with large strides

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes part of Ticket3466
Found-by: Andrey_Karpov / PVS-Studio
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

See b2bed9325.

This allows supporting files for which the image stride is smaller than
the max. block size + number of subpel mc taps, e.g. a 64x64 VP9 file
or a 16x16 VP8 file with -fflags +emu_edge.

This prevents emulated_edge_mc from not undoing mvy*stride-related
integer overflows.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Coccinelle profile used:

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_get_buffer(ctx, f, flags)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str;
  @@

  -if ((r = ff_reget_buffer(ctx, f)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_reget_buffer(ctx, f)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0)
  +    return r;

...along with some manual patches for the remaining ones.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

There is only about 4 lines of common code, so it alot cleaner
when seperated.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes null pointer dereference

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Prevents null ptr dereference

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Also update some common misspelled words in patcheck

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes division by 0

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes division by zero

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes Ticket1156

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

This fixes some arith decoder overreads and a potential infinite loop.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes: CVE-2011-3950

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2011-3949

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Also remove some pointless NULL/0 assigments.

C++ code must be left as it is because named struct
initializers are not supported by C++ standard.

Fixed Ticket636

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

(cherry picked from commit 8db434f8a6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>