Signed-off-by: Diego Biurrun <diego@biurrun.de>

Signed-off-by: Diego Biurrun <diego@biurrun.de>

This will be useful to test more aggressively for failures to mark XMM
registers as clobbered in Win64 builds, and prevent regressions thereof.

Based on a patch by Ramiro Polla <ramiro.polla@gmail.com>

Also prettyprint some comments in Assembly code.

This was accidently disabled.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Also implement sse2/ssse3/avx versions.

This allows combining multiple conditionals in a single statement.

This reverts commit fc115c80b7.

Tests are broken.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Fixes several "‘static’ is not at beginning of declaration" warnings.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Fixes problems in swscale-test where it gives a 3-member array to a
function expecting a 4-member array.

Fixes problems where rgbToRgbWrapper() is called even though it doesn't
support this particular conversion (e.g. converting from RGB444 to
anything). Thirdly, fixes issues where rgbToRgbWrapper() is called for
non-native endiannness conversions (e.g. RGB555BE on a LE system).
Fourthly, fixes crashes when converting from e.g. monowhite to
monowhite, which calls planarCopyWrapper() and overwrites/reads because
n_bytes != n_pixels.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Additional comments from Måns Rullgard have been integrated
by Reinhard Tartler.

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Fixes a compilation failure on win64.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Also implement SSE2/AVX variants.

Also document the function somewhat.

scale.asm keeps horizontal scaling functions, whereas output.asm gets
the vertical scaling/output functions.

Non-RGB output always uses full chroma interpolation.

Writes past the end of the destination buffer were occuring when its
stride was smaller than the stride of the source. Fixes Bug #183.

Fixes PAL8 to YUV conversion.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

This fixes integer multiplication overflows in RGB48 output
(vertical) scaling as detected by IOC. What happens is that for
certain types of filters (lanczos, spline, bicubic), the
intermediate sum of coefficients in the middle of a filter can
be larger than the fixed-point equivalent of 1.0, even if the
final sum is 1.0. This is fine and we support that.

However, at frame edges, initFilter() will merge the coefficients
for the off-screen pixels into the top or bottom pixel, such as
to emulate edge extension. This means that suddenly, a single
coefficient can be larger than the fixed-point equivalent of
1.0, which the vertical scaling routines do not support.

Therefore, remove the merging of coefficients for edges for
the vertical scaling filter, and instead add edge detection
to the scaler itself so that it copies the pointers (not data)
for the edges (i.e. it uses line[0] for line[-1] as well), so
that a single coefficient is never larger than the fixed-point
equivalent of 1.0.

This fixes the same overflow as in the RGB48/16-bit YUV scaling;
some filters can overflow both negatively and positively (e.g.
spline/lanczos), so we bias a signed integer so it's "half signed"
and "half unsigned", and can cover overflows in both directions
while maintaining full 31-bit depth.

Signed-off-by: Mans Rullgard <mans@mansr.com>

We're shifting individual components (8-bit, unsigned) left by 24,
so making them unsigned should give the same results without the
overflow.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

Add a macro to shorten the if condition.

For certain types of filters where the intermediate sum of coefficients
can go above the fixed-point equivalent of 1.0 in the middle of a filter,
the sum of a 31-bit calculation can overflow in both directions and can
thus not be represented in a 32-bit signed or unsigned integer. To work
around this, we subtract 0x40000000 from a signed integer base, so that
we're halfway signed/unsigned, which makes it fit even if it overflows.
After the filter finishes, we add the scaled bias back after a shift.

We use the same trick for 16-bit bpc YUV output routines.

Signed-off-by: Mans Rullgard <mans@mansr.com>

Signed-off-by: Mans Rullgard <mans@mansr.com>