ScopedCollection:CollectionResource:#message-page-resource timeouts for bugs with many messages

Now the top timeout in production.
sample url
https://api.launchpad.net/1.0/bugs/1/messages?ws.start=500&ws.size=75

Recent oops:

https://lp-oops.canonical.com/oops.py/?oopsid=1718C1010

Branch: launchpad-rev-9760.2
Revno: 9760
SQL time: 4344 ms
Non-sql time: 10882 ms
Total time: 15226 ms
Statement Count: 288

Repeating query pattern (due to inefficient lookups that the storm cache can't handle / lack of eager loading:

163. 7644 30ms SQL-launchpad-main-master SELECT COUNT(*) FROM Bug, BugMessage WHERE BugMessage.message = %s AND Bug.id = BugMessage.bug
164. 7676 4ms SQL-launchpad-main-master SELECT Bug.date_last_message, Bug.date_last_updated, Bug.date_made_private, Bug.datecreated, Bug.description, Bug.duplicateof, Bug.heat, Bug.heat_last_updated, Bug.id, Bug.latest_patch_uploaded, Bug.message_count, Bug.name, Bug.number_of_duplicates, Bug.owner, Bug.private, Bug.security_related, Bug.title, Bug.users_affected_count, Bug.users_unaffected_count, Bug.who_made_private FROM Bug, BugMessage WHERE BugMessage.message = %s AND Bug.id = BugMessage.bug ORDER BY Bug.id DESC LIMIT 1
165. 7703 17ms SQL-launchpad-main-master SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_closed, BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released, BugTask.date_incomplete, BugTask.date_inprogress, BugTask.date_left_closed, BugTask.date_left_new, BugTask.date_triaged, BugTask.datecreated, BugTask.distribution, BugTask.distroseries, BugTask.id, BugTask.importance, BugTask.milestone, BugTask.owner, BugTask.product, BugTask.productseries, BugTask.sourcepackagename, BugTask.status, BugTask.statusexplanation, BugTask.targetnamecache, "_prejoin1".account, "_prejoin1".creation_comment, "_prejoin1".creation_rationale, "_prejoin1".datecreated, "_prejoin1".defaultmembershipperiod, "_prejoin1".defaultrenewalperiod, "_prejoin1".displayname, "_prejoin1".hide_email_addresses, "_prejoin1".homepage_content, "_prejoin1".icon, "_prejoin1".id, "_prejoin1".logo, "_prejoin1".mailing_list_auto_subscribe_policy, "_prejoin1".merged, "_prejoin1".mugshot, "_prejoin1".name, "_prejoin1".personal_standing, "_prejoin1".personal_standing_reason, "_prejoin1".registrant, "_prejoin1".renewal_policy, "_prejoin1".subscriptionpolicy, "_prejoin1".teamdescription, "_prejoin1".teamowner, "_prejoin1".verbose_bugnotifications, "_prejoin1".visibility, "_prejoin2".answers_usage, "_prejoin2".blueprints_usage, "_prejoin2".owner, "_prejoin2".translations_usage, "_prejoin2".active, "_prejoin2".autoupdate, "_prejoin2".bug_reported_acknowledgement, "_prejoin2".bug_reporting_guidelines, "_prejoin2".bug_supervisor, "_prejoin2".bugtracker, "_prejoin2".date_next_suggest_packaging, "_prejoin2".datecreated, "_prejoin2".description, "_prejoin2".development_focus, "_prejoin2".displayname, "_prejoin2".downloadurl, "_prejoin2".driver, "_prejoin2".enable_bug_expiration, "_prejoin2".freshmeatproject, "_prejoin2".homepage_content, "_prejoin2".homepageurl, "_prejoin2".icon, "_prejoin2".id, "_prejoin2".lastdoap, "_prejoin2".license_approved, "_prejoin2".license_info, "_prejoin2".reviewed, "_prejoin2".logo, "_prejoin2".max_bug_heat, "_prejoin2".mugshot, "_prejoin2".name, "_prejoin2".official_answers, "_prejoin2".official_blueprints, "_prejoin2".official_malone, "_prejoin2".official_rosetta, "_prejoin2".private_bugs, "_prejoin2".programminglang, "_prejoin2".project, "_prejoin2".registrant, "_prejoin2".remote_product, "_prejoin2".reviewer_whiteboard, "_prejoin2".screenshotsurl, "_prejoin2".security_contact, "_prejoin2".sourceforgeproject, "_prejoin2".summary, "_prejoin2".title, "_prejoin2".translation_focus, "_prejoin2".translationgroup, "_prejoin2".translationpermission, "_prejoin2".wikiurl, "_prejoin3".id, "_prejoin3".name, "_prejoin4".account, "_prejoin4".creation_comment, "_prejoin4".creation_rationale, "_prejoin4".datecreated, "_prejoin4".defaultmembershipperiod, "_prejoin4".defaultrenewalperiod, "_prejoin4".displayname, "_prejoin4".hide_email_addresses, "_prejoin4".homepage_content, "_prejoin4".icon, "_prejoin4".id, "_prejoin4".logo, "_prejoin4".mailing_list_auto_subscribe_policy, "_prejoin4".merged, "_prejoin4".mugshot, "_prejoin4".name, "_prejoin4".personal_standing, "_prejoin4".personal_standing_reason, "_prejoin4".registrant, "_prejoin4".renewal_policy, "_prejoin4".subscriptionpolicy, "_prejoin4".teamdescription, "_prejoin4".teamowner, "_prejoin4".verbose_bugnotifications, "_prejoin4".visibility, "_prejoin5".bug, "_prejoin5".bugtracker, "_prejoin5".datecreated, "_prejoin5".id, "_prejoin5".last_error_type, "_prejoin5".lastchanged, "_prejoin5".lastchecked, "_prejoin5".next_check, "_prejoin5".owner, "_prejoin5".remote_importance, "_prejoin5".remotebug, "_prejoin5".remotestatus FROM BugTask LEFT JOIN Person AS "_prejoin1" ON BugTask.assignee = "_prejoin1".id LEFT JOIN Product AS "_prejoin2" ON BugTask.product = "_prejoin2".id LEFT JOIN SourcePackageName AS "_prejoin3" ON BugTask.sourcepackagename = "_prejoin3".id LEFT JOIN Person AS "_prejoin4" ON BugTask.owner = "_prejoin4".id LEFT JOIN BugWatch AS "_prejoin5" ON BugTask.bugwatch = "_prejoin5".id WHERE BugTask.bug = 136469 AND (1=1) ORDER BY BugTask.id
166. 7724 20ms SQL-launchpad-main-master SELECT Message.datecreated, Message.id, Message.owner, Message.parent, Message.raw, Message.rfc822msgid, Message.subject, "_prejoin1".account, "_prejoin1".creation_comment, "_prejoin1".creation_rationale, "_prejoin1".datecreated, "_prejoin1".defaultmembershipperiod, "_prejoin1".defaultrenewalperiod, "_prejoin1".displayname, "_prejoin1".hide_email_addresses, "_prejoin1".homepage_content, "_prejoin1".icon, "_prejoin1".id, "_prejoin1".logo, "_prejoin1".mailing_list_auto_subscribe_policy, "_prejoin1".merged, "_prejoin1".mugshot, "_prejoin1".name, "_prejoin1".personal_standing, "_prejoin1".personal_standing_reason, "_prejoin1".registrant, "_prejoin1".renewal_policy, "_prejoin1".subscriptionpolicy, "_prejoin1".teamdescription, "_prejoin1".teamowner, "_prejoin1".verbose_bugnotifications, "_prejoin1".visibility FROM BugMessage, Message LEFT JOIN Person AS "_prejoin1" ON Message.owner = "_prejoin1".id WHERE BugMessage.bug = %s AND Message.id = BugMessage.message AND (1=1) ORDER BY Message.datecreated, Message.id
167. 7814 22ms SQL-launchpad-main-master SELECT MessageChunk.blob, MessageChunk.content, MessageChunk.id, MessageChunk.message, MessageChunk.sequence FROM MessageChunk WHERE MessageChunk.message = %s ORDER BY MessageChunk.sequence

----

Currently it seems almost impossible to get *any* messages from bug 1 via the web API. In other words, with launchpadlib, root.bugs[1].messages[0] almost always times out.

For a bug with many (>50) messages, or when requesting a narrow slice of messages, getting messages via the web API is very inefficient because message list is fulfilled by Bug.indexed_messages, which pulls in every message. It can't return a result set - which would be dealt with efficiently - for two reasons: (1) it needs to wrap each message in a delegate to provide the comment index, and (2) it must check if the parent is within the set of messages.

We can address (2) by using a message_id_set:

    message_id_set = set(
        IResultSet(self.messages).values(Message.id))

Instead of the message_set as is currently used.

(1) is trickier. There may be some way to get Storm to materialize a query using a different object factory class, or it may also be possible to provide a custom marshaller for the lazr.restful.