Comment 19 for bug 551193

Created attachment 403292
Proposed patch

The problem comes from the macro mod() used in computation.

The code in fbComposite() from fbpict.c reads like this:

    if (srcRepeat)
    {
        y_src = mod (y_src - pSrc->pDrawable->y, pSrc->pDrawable->height);
        if (h_this > pSrc->pDrawable->height - y_src)
            h_this = pSrc->pDrawable->height - y_src;
        y_src += pSrc->pDrawable->y;
    }

While inspecting the values, we see that initially, y_src=871, pSrc->pDrawable->y=1024, pSrc->pDrawable->height=500

After computation of mod() y_src=895 (which is wrong) so that h_this = pSrc->pDrawable->height - y_src = -395

Passing a negative value to a CARD16 in mmx function will cause the crash. But the real problem is that the value returned by mod() is actually greater than the values passed which is not possible, so there should be no way that y_src is greater than pSrc->pDrawable->height and therefore h_this should/could not be negative.

mod() is defined as follow (earlier in that code):

# define mod(a,b) ((b) == 1 ? 0 : (a) >= 0 ? (a) % (b) : (b) - (-a) % (b))

Problem is that (-a) gets expanded as "-871 - 1024" (and *not* "- (871 - 1024)" as expected) and therefore "(b) - (-a) % (b)" = 500 - (-871 - 1024) = 895

TI think the following would be more appropriate:

# define mod(a,b) ((b) == 1 ? 0 : (a) >= 0 ? (a) % (b) : (b) - (-(a)) % (b))

That seems to fix the crash and produces the correct output.