Taskbrowser refresh is way too slow

Hi Steve,

There's indeed a lot of performance problems and I personnaly want to take care of them for 0.2. In fact, our biggest problems before 0.1 was the parsing in the editor and this was properly addressed. Now, we need to work on the task browser. All the problems you're describing are the refreshing of the taskbrowser : when you modify the title, enter a tag or filter on tag, this trigger a refresh of the browser.

I'm interesed by your insight about TreeModel because it's something I can't find intuitive. I agree that our taskbrowser is suboptimal so any help there would be greatly appreciated.

Hey,

sorry for the delay. I will just try to give a few basic modelfilter code examples. If you have any questions please don't hesitate to ask.

The idea of TreeModel, TreeModelFilter and TreeModelSort (if you need it) is the MVC pattern.
Model will be your model and contain all the Data. You can wrap it with a ModelFilter and filter/modify the data to be displayed. For advanced sorting you can wrap this with a ModelSort.

A wrapped model is referred to as child model. Be careful when handling with iters and paths: E.g you must not use a child_iter to reference a row in the ModelFilter. This can lead to nasty behaviour, warnings and even segfaults.

Anyway the basic stuff is easy:

 def setup(self):
  '''create new empty model '''
  self.tree = self.xml.get_widget('my_treeview')
  self.model = gtk.TreeStore(...)

  self.modelfilter = self.model.filter_new()
  self.modelfilter.set_visible_func(self._visible_func)
  self.tree.set_model(self.modelfilter)

 def _idle_insert_all_tasks(self, task_source):
  ''' Add all tasks in a lazy, pseudo-threading manner '''
  def _insert(task_source):
   for task in task_source:
    self.add_new_task(task)
    # you can also do some progress bar drawing here, if you like
    yield True
   yield False

  job = _insert(task_source)
  gobject.idle_add(job.next)

This is the interesting function for you: It will be called for each item in your underlying model. When it returns
true the row will be shown in the treeview otherwise it will just be hidden.
Whenever you change the description or tag (or something else) of you taks-row in the child model the treemodel
will automatically recall the visible_func on this single taks and you can decide to show it or not.

When you change a global property, e.g change visible tag you can let the model re-evaluate all iters.

 def _visible_func(self, model, titer):
  '''Determine whether iter should be visible in the treeview'''

  type_ = model[titer][C_TYPE]
  tag = model[titer][C_TAG]
  if type_ == 'task':
   return tag in self.active_tags
  ...

I have once read an article on planet.gnome about how to write a custom treemodel. Unluckily I cannot find it right now :-/

Best,
steve

http://davyd.livejournal.com/252351.html

Not what I wanted, but worth a read aswell.

Thanks steve-e, this is very interesting information! We'll surely try to use this in order to improve the browser. A refactoring is planned for the upcoming releases.

Really interesting, thanks for all the information :-)

Unluckily I cannot provide you information about how to write a custom TreeModel. Nevertheless the link I posted above is a good start and I think you should really follow that path.

You should not just use the TreeModel for the TaskBrowser but as your general runtime-datastore. With the model at hand, TaskEditor and TaskBrowser can operate on the same data.

For example, when you open a TaskEditor and now drag'n'drop another task into the subtree of that task in the TaskBrowser, you will proparbly want to update both the TaskEditor and the Browser. You should be able to achieve this by the change-signals emitted by your model and propagated to both views.

I had the chance to try those mechanisms on another pygtk applicatio, and it is indeed a much better to implement this stuff. However, the data structures of GTG have not been thought in this perspective, and I will certainly require to clarify things before being able to progress seriously on this bug. An effort has already been done on refactoring the browser.py file, since it makes modifications easier.

Looking forward to this, as the sluggish performance prevents me from using GTG at all (I hit the performance bottleneck with 70 tasks or so).

In the meantime I'm still using Tasks (now *THAT* is the kind of performance I expect for a GTG app to be "pleasurable" like Mr. Allen suggested in his book). Hopefully I'll be able to give GTG another try when 0.2.x comes out.

I lack the knowledge to help implement a fix, but I do have some basic notions of performance profiling (taken from Federico's fosdem talk in 2007), and this basically sounds to me like the app is, first and foremost, doing unneccessary work (before we start looking into algorythms and such). It begs the question: why does GTG refresh the treeview everytime you type a character in a note title? It should do that only once (when the user moves the focus out of the note's title)!

Hi JF,

Actually we should get rid of this by handling correctly the
treeviews. This means reimplementing completely the way we generate
the content of what we display to do it in a more gtk-ish way.
Unfortunately it also means that we'll have to rethink a part of our
data structures, something this always painful to do. I'm planning to
complete this ASAP since it is a much needed work before GTG could go
further.

By the way, you're right: we should certainly prevent GTG from doing
unnecessary work. However it could be possible that a change in the
title we'll be efficiently handled by the use of signals, etc.
(treeviews are actually quite well thought). Some work will probably
be also needed at the editor level, to prevent throwing a signal when
the user hasn't changed the title.

On Fri, Jul 10, 2009 at 4:27 AM, Jean-François Fortin
Tam<email address hidden> wrote:
> Looking forward to this, as the sluggish performance prevents me from
> using GTG at all (I hit the performance bottleneck with 70 tasks or so).
>
> In the meantime I'm still using Tasks (now *THAT* is the kind of
> performance I expect for a GTG app to be "pleasurable" like Mr. Allen
> suggested in his book). Hopefully I'll be able to give GTG another try
> when 0.2.x comes out.
>
> I lack the knowledge to help implement a fix, but I do have some basic
> notions of performance profiling (taken from Federico's fosdem talk in
> 2007), and this basically sounds to me like the app is, first and
> foremost, doing unneccessary work (before we start looking into
> algorythms and such). It begs the question: why does GTG refresh the
> treeview everytime you type a character in a note title? It should do
> that only once (when the user moves the focus out of the note's title)!
>
> --
> Taskbrowser refresh is way too slow
> https://bugs.launchpad.net/bugs/351867
> You received this bug notification because you are a member of Gtg
> developers, which is the registrant for Getting Things GNOME!.
>
> Status in Getting Things GNOME!: Confirmed
>
> Bug description:
> Hey,
>
> I recently stumble upon GTG and liked it instantly. Unluckily I have now discovered (after filling 200+ taks via the quick add entry) that it is too slow to be used productively.
>
> - Filling the task on startup takes rather long. There should be nothing complicated to it...
> - Typing a word in the note name row freezes GTG for ~2seconds. The same applies for adding a tag by typing  @..
> - Filtering of tags takes ~1 seconds, but this should happen instantly
>
> I only had a short look at your code and saw threading, locking code and a refresh method. Is that really needed?
> Often in such simple applications pseudo threads via python generators and gobject.idle_add are more than sufficient.
>
> IMHO performance could be improved by several orders of magnitude if you use a proper GTKTreeModel. With this at hand you don't need to care about updating or redrawing your stuff. You should also consider wrapping your file/source abstraction in a custom GTKModel that yields the correct signals when required (not that difficult...)
>
> I haven't looked at what happends when you add a tag. But maybe you can only do (what ever y...

FWIW, I am also confused by the need for threading and locking.

Bertrand is working on that in his refactoring branch

Bertrand, any progress on this?

And again, as the proposed fix (refactoring the hell out of gtg) seems quite complex, I have to reiterate the question: why not just limit GTG to refreshing the treeview only when the user moves the focus out of the note's title (instead of in realtime), as a stopgap measure?

Yes there's been progress: a whole GtkTreeModel has been implemented,
as suggested in this bug. From now on, the browser doesn't reload
every task at refresh. The improved code is already available in the
trunk. However the bug cannot be closed now since several regressions
must be adressed.

On Sat, Sep 5, 2009 at 9:42 PM, Jean-François Fortin
Tam<email address hidden> wrote:
> Bertrand, any progress on this?
>
> And again, as the proposed fix (refactoring the hell out of gtg) seems
> quite complex, I have to reiterate the question: why not just limit GTG
> to refreshing the treeview only when the user moves the focus out of the
> note's title (instead of in realtime), as a stopgap measure?
>
> --
> Taskbrowser refresh is way too slow
> https://bugs.launchpad.net/bugs/351867
> You received this bug notification because you are a bug assignee.
>

--
Bertrand Rousseau
Place communale 1, 1450 Chastre, Belgium
e-mail : <email address hidden>
tel : +32 485 96 69 86

Bertrand > I think that regressions are separate issue. This bug adress 3 issues :

- Filling the task on startup takes rather long. There should be nothing complicated to it...
- Typing a word in the note name row freezes GTG for ~2seconds. The same applies for adding a tag by typing @..
- Filtering of tags takes ~1 seconds, but this should happen instantly

This bug should be closed when those 3 points are considered as "fast enough for 2.0"

For me, your work already solved 2 and 3. Only the start is still a bit long.(altought acceptable in my case).

Regressions should be reported separately.

Hello,

it's me again :-)

Just want to thank you for your effort on this and GTG in general. Thank you! Keep up!

Kind Regards,
steve-e

Bertrand > I think you can fairly close this bug now. Maybe open a separate one for startup optimization but this will not be addressed for 0.2

Closing this bu then, since the refactoring of the TaskTree and some additional modifications of the requester and core object since to have sped thing up sufficiently now. As said by ploum in the previous comment, please address any startup optimization bug in a separate bug.

The performance problems with today's bzr checkout still make GTG unusable here, opened bug #476477 about that.