avformat/utils: check for integer overflow in av_get_frame_filename2()

Fixes: signed integer overflow: 317316873 * 10 cannot be represented in type 'int'
Fixes: 24708/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5731180885049344

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 03c479ce23)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavformat/utils.c

@@ -4745,8 +4745,11 @@ int av_get_frame_filename2(char *buf, int buf_size, const char *path, int number
         if (c == '%') {
             do {
                 nd = 0;
-                while (av_isdigit(*p))
+                while (av_isdigit(*p)) {
+                    if (nd >= INT_MAX / 10 - 255)
+                        goto fail;
                     nd = nd * 10 + *p++ - '0';
+                }
                 c = *p++;
             } while (av_isdigit(c));