Fixes: signed integer overflow: 2147480546 + 4096 cannot be represented in type 'int'
Fixes: 16280/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5123442566758400

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Tomas Härdin <tjoppen@acc.umu.se>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 9d3ddef519)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: left shift of 1073741824 by 1 places cannot be represented in type 'int'
Fixes: signed integer overflow: 2049431315 + 262759074 cannot be represented in type 'int'
Fixes: 16012/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5719016003338240

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 392c028cd2)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: Assertion failure and memleak
Fixes: 15709/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5182435093905408

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8e4b522c91)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: 755176387 + 1515360583 cannot be represented in type 'int'
Fixes: 15506/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5706859232624640

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit bf778af149)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: left shift of negative value -4
Fixes: signed integer overflow: -15091694 * 167 cannot be represented in type 'int'
Fixes: signed integer overflow: 1898547155 + 453967445 cannot be represented in type 'int'
Fixes: 15258/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5759095564402688
Fixes: signed integer overflow: 962196438 * 31 cannot be represented in type 'int'
Fixes: 15364/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5718799845687296

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 267eb2ab7f)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: -538976267 * 31 cannot be represented in type 'int'
Fixes: left shift of 65312 by 16 places cannot be represented in type 'int'
Fixes: 15255/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5718831688843264
Fixes: 15547/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5691384901664768

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 240bf0e596)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signed integer overflow: -829262115 + -1410750414 cannot be represented in type 'int'
Fixes: 15251/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5651742252859392

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0af08cb803)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: 15255/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5718831688843264
Fixes: left shift of 1 by 31 places cannot be represented in type 'int'

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3d4f4f4a15)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: out of array access
Fixes: PoC.ape and others

Found-by: Bingchang, Liu@VARAS of IIE
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ba4beaf614)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit 9149e9c0ba)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes out of array read
Fixes: 0a7ff0c1d93da9cef28a315ec91b692a/asan_heap-oob_4a52e5_3604_9c56dbb20e308f4faeef7b35f688521a.ape

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit cd7524fdd1)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Express bitfields more simply.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

The decode_array_0000 assumed that 64 is the minimal block size
while it is not.

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

s->decoded_buffer is allocated with a min_size of:
    2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer)

Then it is assigned to s->decoded[0] (and s->decoded_buffer + FFALIGN(blockstodecode, 8)
to s->decoded[1]) and passed as out buffer to decode_array_0000.

In this function 64 elements of the out buffer are written
unconditionally and outside the array if blockstodecode is too small.

This causes memory corruption, leading to segmentation faults or other
crashes.

Thus change decode_array_0000 to write at most blockstodecode elements
of the out buffer.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Otherwise range_start_decoding is not necessarily run and thus
ctx->rc.range still 0 in range_dec_normalize leading to an infinite
loop.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Zeng Zhaoxiu <zhaoxiu.zeng@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Zeng Zhaoxiu <zhaoxiu.zeng@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

48078630 -> 32533850 dezi cycles

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

APE is not the sole codec using scalarproduct_and_madd_int16.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CID1206639
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes use of uninitialized memory
Fixes part of msan_uninit-mem_7fa0d8c8bd58_8417_sh3.ape
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes assertion failure
Fixes part of msan_uninit-mem_7fa0d8c8bd58_8417_sh3.ape
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Only consume an AVPacket when all the samples have been read.

When the rate of samples output is limited (by the default value
of max_samples), consuming the first packet immediately will cause
timing problems:

- The first packet with PTS 0 will output 4608 samples and be
consumed entirely
- The second packet with PTS 64 will output the remaining samples
(typically, a lot, that's why max_samples exist) until the decoded
samples of the first packet have been exhausted, at which point the
samples of the second packet will be decoded and output when
av_decode_frame is called with the next packet).

That means there's a PTS jump since the first packet is 'decoded'
immediately, which can be seen with avplay or mplayer: the timing
jumps immediately to 6.2s (which is the size of a packet).

Sample: http://streams.videolan.org/issues/6348/Goldwave-MAClib.ape
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes division by 0

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Range coder in 3.90-3.95 overread two bytes in the final normalize.

This will help in supporting old versions, e.g. version 3.93 uses the same
range coder but different predictor and version 3.82 uses different range
coder and predictor. Also this should not make decoding newer versions slower
by introducing additional checks on versions.

Coccinelle profile used:

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_get_buffer(ctx, f, flags)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str;
  @@

  -if ((r = ff_reget_buffer(ctx, f)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_reget_buffer(ctx, f)) < 0)
  +    return r;

  @@
  expression r, ctx, f, loglevel, str, flags;
  @@

  -if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0) {
  -    av_log(ctx, loglevel, str);
  -    return r;
  -}
  +if ((r = ff_thread_get_buffer(ctx, f, flags)) < 0)
  +    return r;

...along with some manual patches for the remaining ones.

It will be useful in the upcoming transition to refcounted AVFrames.

Also reorder some other #include when applicable.

Timestamp handling of decoded frames are much simpler now.

Fixes #1675.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Martin Storsjö <martin@martin.st>

Signed-off-by: Martin Storsjö <martin@martin.st>

This involves a division that should be a shift.

Signed-off-by: Diego Biurrun <diego@biurrun.de>

Also break some long lines, remove codec function placeholder comments
and add spaces in sample/pixel format lists.

Signed-off-by: Martin Storsjö <martin@martin.st>

Fixes a floating-point exception further down.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>