rtpdec_asf: Fix integer underflow that could allow remote code execution

Fixes MSVR-11-0088
Fixes CVE-2011-4031
Credit:  Jeong Wook Oh of Microsoft and Microsoft Vulnerability Research (MSVR)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5ea091fb5a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

libavformat/rtpdec_asf.c

@@ -233,8 +233,14 @@ static int asfrtp_parse_packet(AVFormatContext *s, PayloadContext *asf,
 
                 int cur_len = start_off + len_off - off;
                 int prev_len = out_len;
+                void *newmem;
                 out_len += cur_len;
-                asf->buf = av_realloc(asf->buf, out_len);
+                if (FFMIN(cur_len, len - off) < 0)
+                    return -1;
+                newmem = av_realloc(asf->buf, out_len);
+                if (!newmem)
+                    return -1;
+                asf->buf = newmem;
                 memcpy(asf->buf + prev_len, buf + off,
                        FFMIN(cur_len, len - off));
                 avio_skip(pb, cur_len);