Comment 9 for bug 618019

On Thu, Aug 19, 2010 at 6:10 AM, Gustavo Niemeyer <email address hidden> wrote:
> If iterating over thousands of objects has a fixed cost which you're
> unhappy about, I wouldn't like to increase that fixed cost by adding
> persistent debugging logic on that fast path.

Iteration is only part of it. I'm not worried about the python
overhead as yet - simple iteration is very fast.

> This is the logic of iteration of results within Storm's database.py:
>
>            for result in results:
>                yield tuple(self.from_database(result))
>
> It doesn't look good to add a tracer+timer in this loop.

That loop looks cheap (and may be, though storms complex variable
mapping is inside that loop.

Its not the area that matters though. This is the full loop:
        while True:
           results = self._connection._check_disconnect(
                self._raw_cursor.fetchmany)
            if not results:
                break
            for result in results:
                yield tuple(self.from_database(result))

The second like calls raw_cursor.fetchmany which AIUI *can block* and
*do networking*.

If I'm wrong, its a nonissue. Am I wrong?

> If you're wondering about network/db problems, I recommend monitoring
> these pieces more closely, rather than adding that logic to Storm.

Storm is our connection to the DB, we've done the same thing with
tracers to track the initial time for queries so far - I mean we can
write a replacement raw_cursor and db implementation, but if we need
to do that why are we using a storm tracer at all for the other
identical things?