Since the default in the libav fork is to only allow known layouts, making
unknown layouts allowed by default here can be a security risk for filters
directly merged from libav. However, usually it is simple to detect such cases,
use of av_get_channel_layout_nb_channels is a good indicator, so I suggest we
change this regardless.

See http://ffmpeg.org/pipermail/ffmpeg-devel/2016-November/203204.html.

This patch indirectly adds unknown channel layout support for filters where
query_formats is not specified:

abench
afifo
ainterleave
anullsink
apad
aperms
arealtime
aselect
asendcmd
asetnsamples
asetpts
asettb
ashowinfo
azmq

It introduces a query_formats callback for the asyncts filter, which only
supports known channel layouts since it is using libavresample.

And it removes .query_formats callback from filters where it was only there to
support unknown layouts, as this is now the default:

aloop
ametadata
anull
asidedata
asplit
atrim

Acked-by: Nicolas George <george@nsup.org>
Signed-off-by: Marton Balint <cus@passwd.hu>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This is a somewhat subtle failure that can occur when the realloc_array
fails in FORMATS_REF.

Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

Many of the functions from avfilter/formats can return errors, usually AVERROR(ENOMEM).
This propagates the return values.

All of these were found by using av_warn_unused_result, demonstrating its utility.

Tested with FATE. I am least sure of the changes to avfilter/filtergraph,
since I don't know what/how reduce_format is intended to behave and how it should
react to errors.

Fixes: CID 1325680, 1325679, 1325678.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Previous version Reviewed-by: Nicolas George <george@nsup.org>
Previous version Reviewed-by: Clément Bœsch <u@pkh.me>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

It should be possible to pass hwaccel frames through lavfi.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Also make sure the allocation and its check are properly done.

This macro is unused since 247fa6c27c.

This avoids null pointer dereferences in case memory allocation has failed

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Also only free it when it was not previously allocated to return to the
state prior to the failing function call

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

CC: libav-stable@libav.org
Bug-Id: CID 1250334

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Thatt hardcodes the number of formats into lavfi and will break when a
shared lavu is updated, adding new formats.

Make ff_parse_channel_layout() accept unknown layouts too.

The functions are unused

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

This is more consistent with naming in the rest of Libav.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Detect when filtering known layouts from an explicit list
results in an empty list.

Fix erratic behavior.

The following commit:
b97d61f avfilter/ff_merge_formats: only merge if doing so does not loose chroma or alpha
introduced an exception to avoid lossy conversions.
Add a comment to explain the logic.
Fix the call to avoid applying it on audio formats.

Fixes Ticket1280

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

The first parameter is a pointer and NULL is more correct

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Also remove it in once case when it is not needed.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

Signed-off-by: Martin Storsjö <martin@martin.st>

It was deprecated and removed after the recent 2->3 major bump.

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This is more consistent with naming in the rest of Libav.

I changed the *_set_common_* functions to only set unset formats, then
added a wrapper that calls them after the filters query_formats.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

It is only useful inside filters and we don't allow user filters for
now.

Those functions are only useful inside filters. It is better to not
support user filters until the API is more stable.

The list goes back to being simple integers,
and avfilter_make_format64_list is fixed to work
with the correct structure directly.

all_channel_layouts.h is not a real header file and breaks make checkheaders.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

The planar/packed switch and the packing_formats list is no longer
required, since the planar/packed information is now stored in the sample
format enum.

This is technically a major API break, possibly it should be not too
painful as we marked the audio filtering API as unstable.

It's more convenient to have them all in one file.