Personally, I need the decoder to back out if get_format() returns no
usable pixel format. This didn't work because the error code was not
propagated down the call chain. This in turn happened because the
variable declaration removed in this patch shadowed the variable, whose
value is returned at the end of the function. Consequently, failures of
decode_nal_unit() were ignored in this place.

Reviewed-by:  Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CID1239099 part 2

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CID1239099 part 1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

The variable should not be needed anymore

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: James Almer <jamrial@gmail.com>

This allows the callers to have a hint of the probable stream parameters
without actually decoding anything.

Additionally always set the software pixel format, so it's available
even if ff_get_format() is not called later. This will be useful for
exporting stream parameters from init().

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Clang -fsanitize=shift
Reported-by: Thierry Foucu <tfoucu@google.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

hevc seems to be the only place where the C implementation
of the av_clip function is explicitly selected, precluding
platform-specific optimizations

Signed-off-by: Peter Meerwald <pmeerw@pmeerw.net>
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Found-by: Clément Bœsch <u@pkh.me>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

The buffer pointers would be otherwise overwritten, causing a
leak on e.g. PERSIST_RPARAM_A_RExt_Sony_1.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Use edge emu buffers
And enable the code unconditionally

Speed difference without USE_SAO_SMALL_BUFFER and with the new code:
Decicycles: 26772->26220 (BO32),  83803->80942 (BO64)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

cherry picked from commit 5d9f79edef

SAO edge filter uses pre-SAO pixel data on the left and top of the ctb, so
this data must be kept available. This was done previously by having 2
copies of the frame, one before and one after SAO.

This commit reduces the storage to just that, instead of the previous whole
frame.

Commit message taken from patch by Christophe Gisquet <christophe.gisquet@gmail.com>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

cherry picked from commit 8e50557707

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Usefull for at least band filter, for which:
- Band filter call only:
           32      64
Before:  16556    54015
After:   16497    52355
- Whole case:
           32      64
Before:  37031   103008
After:   32045    93952

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

For future use by hardware accelerators.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Hardware Accelerators require access to the escaped bitstream.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Anton Khirnov <anton@khirnov.net>

Signed-off-by: Anton Khirnov <anton@khirnov.net>

Signed-off-by: Anton Khirnov <anton@khirnov.net>

For future use by hardware accelerators.

Signed-off-by: Anton Khirnov <anton@khirnov.net>

Hardware Accelerators require access to the escaped bitstream.

Signed-off-by: Anton Khirnov <anton@khirnov.net>

This enables decode_nal_unit to access additional fields added in
subsequent commits.

Signed-off-by: Anton Khirnov <anton@khirnov.net>

Prevents DPB from filling up with damaged input.

This enables decode_nal_unit to access additional fields added in
subsequent commits.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes CID1260704

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

This avoids use of uninitialized memory
Fixes: asan_static-oob_17aa046_582_cov_212287884_DBLK_G_VIXS_1.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Its value shall be between 0 and 7 according to the specifications.

CC: libav-stable@libav.org
Bug-Id: CID 1257502

Also make code more robust by initializing the pointers to NULL

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Also, collapse the array into a scalar, since only one value is needed
at a time.

pred_mode is overwritten a few lines immediately below.