Before commit 2f76476549a01ae2c4ec2a440e4b14c5aba64869, avfilter.h
contained no typedef for AVFilterChannelLayouts; all references to it
were done using its struct tag. formats.h meanwhile contained the
definition of the struct and a typedef for it. Said commit now added a
typedef in avfilter.h, too, bringing it in line with AVFilterFormats;
yet this means that there are two typedefs for AVFilterChannelLayouts
(in contrast to AVFilterFormats which is only typedef'ed in avfilter.h).

The problem is that older versions of GCC don't like this and error out:
http://fate.ffmpeg.org/history.cgi?slot=x86_64-openbsd5.6-gcc4.2-conf2
is one of the FATE boxes that now fail to compile. So just remove the
typedef in formats.h.

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Part of the code expects valid lists, in particular no duplicates.
These tests allow to catch bugs in filters (unlikely but possible)
and to give a clear message when the error comes from the user
((a)formats) or the application (buffersink).

If we decide to switch to a more efficient merging algorithm,
possibly sorting the lists, these functions will be the preferred
place for pre-processing, and can be renamed accordingly.

This commit removes ff_parse_sample_format(), ff_parse_time_base() and
ff_query_formats_all_layouts() from libavfilter/formats.c. All of these
functions were completely unused. ff_parse_time_base() has not been used
at all since it had been added in 3448404a707b6e236a2ffa7b0453b3300de41b7b;
the last caller of ff_parse_sample_format has been removed in commit
d1c49bcae9. And the one and only caller of
ff_query_formats_all_layouts() (the asyncts filter) has been removed in
commit a8fe8d6b4a.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

The callers of the ff_merge_*() functions fall into two categories with
quite different needs:

One caller is can_merge_formats() which only wants to test for mergeability
without it merging anything. In order to do so, it duplicates the lists
it intends to test and resets their owners so that they are not modified
by ff_merge_*(). It also means that it needs to receive the merged list
(and not only an int containing whether the lists are mergeable) to
properly free it.

The other callers want the lists to be actually merged. But given the
fact that ff_merge_*() automatically updates the owners of the lists,
they only want the information whether the merge succeeded or not; they
don't want a link to the new list.

Therefore this commit splits these functions in two: ff_merge_*() for
the latter callers and ff_can_merge_*() for the former.
ff_merge_*() doesn't need to return a pointer to the combined list at all
and hence these functions have been modified to return an int, which
allows to distinguish between incompability and memory allocation failures.

ff_can_merge_*() meanwhile doesn't modify its arguments at all obviating
the need for copies. This in turn implies that there is no reason to
return a pointer to the new list, as nothing needs to be freed. These
functions therefore return an int as well. This allowed to completely
remove can_merge_formats() in avfiltergraph.c.

Notice that no ff_can_merge_channel_layouts() has been created, because
there is currently no caller for this. It could be added if needed.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Despite its name, this function is not part of the public API, as
formats.h, the header containing its declaration, is a private header.
The formats API was once public API, but that changed long ago
(b74a1da49db5ebed51aceae6cacc2329288a92c1, the commit scheduling it to
become private, is from 2012). That avfilter_make_format64_list() was
forgotten is probably a result of the confusion resulting from the
libav-ffmpeg split.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

It is unused since 8cbb055760 and it
actually coincides with avfilter_make_format64_list().

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

Since the default in the libav fork is to only allow known layouts, making
unknown layouts allowed by default here can be a security risk for filters
directly merged from libav. However, usually it is simple to detect such cases,
use of av_get_channel_layout_nb_channels is a good indicator, so I suggest we
change this regardless.

See http://ffmpeg.org/pipermail/ffmpeg-devel/2016-November/203204.html.

This patch indirectly adds unknown channel layout support for filters where
query_formats is not specified:

abench
afifo
ainterleave
anullsink
apad
aperms
arealtime
aselect
asendcmd
asetnsamples
asetpts
asettb
ashowinfo
azmq

It introduces a query_formats callback for the asyncts filter, which only
supports known channel layouts since it is using libavresample.

And it removes .query_formats callback from filters where it was only there to
support unknown layouts, as this is now the default:

aloop
ametadata
anull
asidedata
asplit
atrim

Acked-by: Nicolas George <george@nsup.org>
Signed-off-by: Marton Balint <cus@passwd.hu>

Signed-off-by: Paul B Mahol <onemda@gmail.com>

This uses the av_warn_unused_result attribute liberally to catch some forms of improper
usage of functions defined in avfilter/formats.h.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>

Also make sure the allocation and its check are properly done.

This is more consistent with naming in the rest of Libav.

The following commit:
b97d61f avfilter/ff_merge_formats: only merge if doing so does not loose chroma or alpha
introduced an exception to avoid lossy conversions.
Add a comment to explain the logic.
Fix the call to avoid applying it on audio formats.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

It is only useful inside filters and we don't allow user filters for
now.

Those functions are only useful inside filters. It is better to not
support user filters until the API is more stable.