Based on a patch by Nicolas George <nicolas.george <at> normalesup.org>

Fixes overread

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes invalid reads while initializing the dequant tables, which uses
the bit depth to determine the QP table size.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Alex Converse <alex.converse@gmail.com>

This fixes a over read.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

This is based on the reference implementation and fixes
a global out of array read.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

This fixes a out of global array read.
This change is based on the reference mpc imlementation.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

This makes the packetization spec compliant for cases where one single
GOB doesn't fit into an RTP packet.

Signed-off-by: Martin Storsjö <martin@martin.st>

Signed-off-by: Martin Storsjö <martin@martin.st>

Signed-off-by: Martin Storsjö <martin@martin.st>

Fixes out of array read.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

It appears there are corner cases with damaged input that can lead
to small overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Code ported from qatar/master, please see there for per line authorship.
Main authors AFAIK are Ronald and Justin. I have no authorship on this.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

* qatar/master: (58 commits)
  amrnbdec: check frame size before decoding.
  cscd: use negative error values to indicate decode_init() failures.
  h264: prevent overreads in intra PCM decoding.
  FATE: do not decode audio in the nuv test.
  dxa: set audio stream time base using the sample rate
  psx-str: do not allow seeking by bytes
  asfdec: Do not set AVCodecContext.frame_size
  vqf: set packet parameters after av_new_packet()
  mpegaudiodec: use DSPUtil.butterflies_float().
  FATE: add mp3 test for sample that exhibited false overreads
  fate: add cdxl test for bit line plane arrangement
  vmnc: return error on decode_init() failure.
  libvorbis: add/update error messages
  libvorbis: use AVFifoBuffer for output packet buffer
  libvorbis: remove unneeded e_o_s check
  libvorbis: check return values for functions that can return errors
  libvorbis: use float input instead of s16
  libvorbis: do not flush libvorbis analysis if dsp state was not initialized
  libvorbis: use VBR by default, with default quality of 3
  libvorbis: fix use of minrate/maxrate AVOptions
  ...

Conflicts:
	Changelog
	doc/APIchanges
	libavcodec/avcodec.h
	libavcodec/dpxenc.c
	libavcodec/libvorbis.c
	libavcodec/vmnc.c
	libavformat/asfdec.c
	libavformat/id3v2enc.c
	libavformat/internal.h
	libavformat/mp3enc.c
	libavformat/utils.c
	libavformat/version.h
	libswscale/utils.c
	tests/fate/video.mak
	tests/ref/fate/nuv
	tests/ref/fate/prores-alpha
	tests/ref/lavf/ffm
	tests/ref/vsynth1/prores
	tests/ref/vsynth2/prores

Merged-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

We already have sufficient coverage for 16-bit pcm.

Otherwise the values are overwritten.

Packets are not guranteed to be allocated by av_malloc().

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Fixes a out of array read.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Its not clear from the spec what to do with values larger than 127
so iam opting for the safe side and ask for a sample.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

related to b716542691
Error messages and audible artifacts were fixed in that commit.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

also use AVERROR codes for some return values instead of -1

simplifies the code and does less memmove()

vorbis_bitrate_flushpacket() does not return any packets that should not be
output in the bitstream.

libvorbis takes float input, so we can just deinterleave/reorder the input
as-is instead of also converting.

Fixes a segfault if init() fails before initializing the dsp state

- enable the options for audio encoding
- properly check for user-set maxrate
- use correct calling order in vorbis_encode_setup_managed()