With Bindwood installed, Firefox is completely unresponsive

Anton,

Thanks for taking the time to file a bug. We've added some diagnostic debugging that can be enabled if FF is run with BINDWOOD_DEBUG in the process' environment. The easiest way is to run from the terminal (like you tried):

  $ BINDWOOD_DEBUG=1 firefox &

This will output a lot of debugging messages to the error console. The default FF error console only allows you to copy one message at a time, so you could install the Console2 extension ( https://addons.mozilla.org/en-US/firefox/addon/1815 ) to enable copying multiple lines.

Bindwood iterates over all your bookmarks, pushing each to your desktop couchdb (after first checking that it's not already present in the db). During testing and development, it's likely that we didn't provide our own instances with enough bookmark data to really stress test the setup. Off hand, do you have an idea of the number of bookmarks you have?

In the future, Bindwood will do a much better job after the first run, of keeping its network traffic to a bare minimum. As it stands now, it's a little too chatty.

Thanks for the quick reply!

The amount of bookmarks indeed seems to be part of the problem, yet I also discovered something else.

First some information:
I have 78 bookmarks in total, in a hierarchy that is two subfolders deep. The actual json-bookmark file exported by Firefox is 33KB in size.
My Internet connection is not very fast (max. 15KB/s upload).
I waited 15 minutes to see whether the initial sync would eventually complete, but this was not the case and Firefox was still unusable after all the time.

So I deleted all bookmarks. With nothing to sync, the browser now was usable from the start.
Then I restored my bookmarks, by loading a backup file from within Firefox.
Interestingly, this time the browser remained usable, beam.smp and firefox did not consume 100% CPU, and the sync began, at a rate of about 1 bookmark per minute.
There were no errors and warnings in the error console, only regular messages about the ongoing transfer. It would probably have completed successfully.

But out of curiosity I did not wait that long. Instead I restarted Firefox, with the result that, once again, CPU load spiked and the restarted browser froze.

In short:
-Start Firefox+Bindwood with existing bookmarks --> freeze, high CPU load
-Start Firefox+Bindwood without bookmarks, restore them from file --> regular sync begins

I also tried to get debug information according to your instructions. But with the browser frozen I was unable to access the error console.

This bug has disappeared after deleting all items labeled "Desktop Couch user authentication" in seahorse.
Firefox now responds to input. :-)

Well, my experience is that if you wait long enough (tens of minutes) firefox will eventually be responsive again. Of course, it's completely unsatisfactory to have to wait tens of minutes after starting a browser for it to be useful.

Any thoughts on why it's so slow to start up?

A bit more information: I couldn't reproduce this bug on another machine with a relatively fresh Karmic install and the same set of bookmarks.

The PC with the freeze bug has been running development releases since Hardy, so maybe something broke along the way.

Hi,

I could reproduce this bug with a fresh karmic beta install + all updates.
But on my computer it consumes not only cpu, but also all memory (8 Gig) and take the swap, after 3 Gig of swap I killed firefox.
After disabling the bindwood plugin via the safe mode feature, firefox works again.
I have 150-200 bookmarks in 25-30 folders with maximum hierarchy depth of 4.

My internet connection has 100 MBit/s upload.

I've got a few more informations:

I saved my bookmarks as backup, deleted all my bookmarks in firefox, activated bindwood and restarted firefox. After that bindwood worked, I created one new bookmark and it was synced correct to the desktop couch. After that I imported my backup and it took only 4 seconds to sync all my normal bookmarks to the desktop couch.
It works this way. It created 230 documents in the bookmarks database.

But after I restarted firefox, the browser consumes all cpu and memory again. So the problem is not the performance of bindwood to the desktop couch and to much data alone, the issue has to do more with the way firefox starts up and launchs it.

How can I access the firefox error console if firefox is unresponsive?

Bindwood is syncing live bookmarks, which spams desktopcouch on every startup with lots of duplicate entries.

Here is an excerpt of the raw content from my desktopcouch, these come from the default live bookmark in firefox.

{"id":"1c6d34c81fcb0048575fe523faa32558","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"204830b6140fea0a9f2122f10d194ec8","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"23aad0bf9adbba87fc79d16a0b6aefc3","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"2739083632b8a75477d56bffa4614796","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"404c789f7c65d7cea56e8c4fdf8f128a","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"5a98387397942df68f478dd4a52ed994","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"5fec432a3a41703bf2851aa804d1e6b1","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"749cb9fdd035bd5cf7876dbe7bed5601","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"9986ac59a3b71a061a522e28ea75175a","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"ba51772d47e29ddf3eecba2d971d885b","key":"'Bracelets' useless in arthritis","value":"news.bbc.co.uk/go..."},
{"id":"c9ff5b3c60c48c5ed01ace38482ac25b","key":"'Crash for cash' scam man jailed","value":"news.bbc.co.uk/go..."},
{"id":"ccc3bf62c55e74a8ee79e50d0e45176f","key":"'Crash for cash' scam man jailed","value":"news.bbc.co.uk/go..."},
{"id":"676d5cd1d63d45d624351592860fef85","key":"'Extremists hijack' military name","value":"news.bbc.co.uk/go..."},
{"id":"6a59843851a2019c2700d25d11023c58","key":"'Extremists hijack' military name","value":"news.bbc.co.uk/go..."},
{"id":"6dba534fd64064af38c1c27052c8eb3d","key":"'Extremists hijack' military name","value":"news.bbc.co.uk/go..."},
{"id":"d18dc1f55d21c63820926f753a6a2c5c","key":"'Extremists hijack' military name","value":"news.bbc.co.uk/go..."},
{"id":"dbc5a8f70ede4a7a5a051c0149aef632","key":"'Extremists hijack' military name","value":"news.bbc.co.uk/go..."},
{"id":"30de68d5a47be4fbd72c7c7f1b47bf9a","key":"'Giant' orb web spider discovered","value":"news.bbc.co.uk/go..."},
{"id":"7a9170ae5853ea94823aa095a4e90da0","key":"'Giant' orb web spider discovered","value":"news.bbc.co.uk/go..."},
{"id":"86b7d246d322233d86af4f74074419a8","key":"'Giant' orb web spider discovered","value":"news.bbc.co.uk/go..."},
{"id":"4cb52e26e4693d2858d60264ea516adf","key":"'Hoodies down' call for colleges","value":"news.bbc.co.uk/go..."},
{"id":"d58d343f43242594521211807a9b14f0","key":"'Hoodies down' call for colleges","value":"news.bbc.co.uk/go..."},
{"id":"01f6996b06ac859a9c5e15614114de7d","key":"'Quick test' for airport liquids","value":"news.bbc.co.uk/go..."},
{"id":"04a64e18bf83386a6d4ceee5251c1131","key":"'Quick test' for airport liquids","value":"news.bbc.co.uk/go..."},
{"id":"0e02ac6c983bf000f8bb7b8c4a5f5cd3","key":"'Quick test' for airport liquids","value":"news.bbc.co.uk/go..."},
{"id":"42c41ab69ad13c7730e009be3f0476fa","key":"'Quick test' for airport liquids","value":"news.bbc.co.uk/go..."},
{"id":"4ebd93ba67fae6836426d0a4b7b083df","key":"'Quick test' for airport liquids","valu...

I think I see what the problem is and how to correct it. There are means to avoid livemarks folders from the folder iteration, and we haven't been doing that. I'm currently testing a solution to this now, and hope to have a fix shortly.

This is important show case for CouchDB, and will be a popular feature I think. We should move the fix into Karmic if it is feasible.

I've deleted my live bookmarks folder to see if that stops the grinding but as soon as I launch firefox it seems that bindwood repopulates the browser immediately.

Jorge O. Castro wrote:
> I've deleted my live bookmarks folder to see if that stops the grinding
> but as soon as I launch firefox it seems that bindwood repopulates the
> browser immediately.

Yes, we saw that during testing, too. The branch that fixes this
behavior is almost through review. After it's released there are two
ways forward for those suffering from this:

   1) Move aside your bookmarks Couch database (you'll need to make sure
that desktop couch is stopped), and let Bindwood recreate the bookmarks
database from scratch - it will avoid syncing the livemarks if present)

   2) Manually delete the documents from Couch (preferably after
disabling Bindwood temporarily). Another bug that was fixed in the
branch under testing was that new bookmarks created by Firefox were
stamped as coming from the toolbarFolder by default, even if they were
created (say) in the livemarks folder. Then, when Bindwood pulls those
bookmarks and has to recreate them, they'd end up in the toolbar. Not at
all what anyone really wants.

Sorry for the trouble, folks. It's about to get a whole lot better.

Zac

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading bindwood_0.4.2-0ubuntu1.dsc: done.
  Uploading bindwood_0.4.2.orig.tar.gz: done.
  Uploading bindwood_0.4.2-0ubuntu1.diff.gz: done.
  Uploading bindwood_0.4.2-0ubuntu1_source.changes: done.
Successfully uploaded packages.

This bug was fixed in the package bindwood - 0.4.2-0ubuntu1

---------------
bindwood (0.4.2-0ubuntu1) karmic; urgency=low

  * new bug fix release 0.4.2
    - fix LP: #459068 - Create a proper folder for storing Couchdb/UbuntuOne book
      marks
    - fix LP: #443121 - With Bindwood installed, Firefox is completely unresponsive

 -- Alexander Sack <email address hidden> Mon, 26 Oct 2009 17:26:22 +0100

I'm not convinced that this is fixed. I'm running a fresh install (as of this morning) of karmic rc (bindwood 0.4.2-0ubuntu2), and having bindwood installed means firefox is unresponsive for the first minute or so that it is open. I have ~100 bookmarks.

I'll see if I can attach some debugging output later.

mister_pink wrote:
> I'm not convinced that this is fixed. I'm running a fresh install (as of
> this morning) of karmic rc (bindwood 0.4.2-0ubuntu2), and having
> bindwood installed means firefox is unresponsive for the first minute or
> so that it is open. I have ~100 bookmarks.
>
> I'll see if I can attach some debugging output later.

In its current incarnation, there will be a linear lag at startup,
depending on the number of bookmarks stored in Firefox. At startup, we
iterate over all the local bookmarks and push them to CouchDB if they're
not already present. This is a known limitation, but isn't going to land
before Karmic. We'll be pushing updates to our PPA in the meanwhile
between Karmic and Lucid releases, so please stay tuned if Bindwood is
something you plan on using.

Thanks,

Zac

Yeah, still seeing a big lag on firefox startup.

Is there a fix for this in a PPA that we should be testing? Or should we just disable bindwood until a fix is available?

Same problem with my bookmarks.
Number of bookmarks: 314
Number of newsfeeds: 8
Number of folders: 13
Firefox just freezes after I launch it with bindwood enabled.

Jeremy: I'm working on a new branch that will likely impose a stiff lag penalty on first launch, but will drastically reduce the lag after initial sync. As soon as I can, I'll upload a test package to my PPA, and let everyone here know.

And by "impose a stiff lag penalty", I mean a linear lag penalty based on the number of bookmarks in your profile. Those with more bookmarks will notice a longer lag at first startup, but things should be much faster later on.

So what exactly is the work around? And why is this marked "Fixed" if people are still affected by it?

I see people talking about removing a "Live Bookmarks" folder, but I don't see that anywhere. And I see stuff about removing couchdb stuff, but I don't know anything about that. A detailed explaination of what to do would help those who show up here because they're affected by this.

Zachery: This still seems like a bad solution. Why isn't this just done in the background, so that firefox remains responsive while the bookmark sync is happening?

Jeremy Nickurak wrote:
> So what exactly is the work around? And why is this marked "Fixed" if
> people are still affected by it?

The workaround right now, if it's really crippling, is to disable the
Bindwood extension. It was marked Fixed because the initial bug was
about the unresponsiveness and bloat caused by the livemarks.

> I see people talking about removing a "Live Bookmarks" folder, but I
> don't see that anywhere. And I see stuff about removing couchdb stuff,
> but I don't know anything about that. A detailed explaination of what to
> do would help those who show up here because they're affected by this.

> Zachery: This still seems like a bad solution. Why isn't this just done
> in the background, so that firefox remains responsive while the bookmark
> sync is happening?

The simplest explanation is that I didn't want to dive into dealing with
threads until I saw a need. In retrospect, it was a naive decision.

I agree with you: there shouldn't be any responsiveness penalty for the
user, regardless of bookmarks corpus size. So I'm now exploring
Firefox's thread manager to make all bookmarks collection and network
operations happen on a background thread.

Thanks,

Zac

I've reopened this bug and marked it "in progress"

Hi Zac,

Thanks for the update and the work on this. Look forward to testing to the updated package.

All the best
Brendan

Lastest Karmic, fully Updated (19:30 UK time, 11/11/2009), fresh install of bindwood through synaptic, kills Firefox on both PCs (unresponsive; 100% CPU load). json file is 86k - accumulated debris of years of bookmarking, I admit, but they're all in neat folders by subject, it's not unreasonable... and backup/restore/copy-between-PCs of that json file locally takes seconds, so I don't really understand what the problem is (or for that matter why it's taken two hours for Ubuntu One to upload about 10M of files, which again should only take seconds ... what's going on?)

Any news here?

I learned a lot about Chrome, Arora, and Seamonkey while trying to figure out what froze my FF.
Just found this thread 10 minutes ago and finally got back up and running in FF.

I have thousands of bookmarks. No hope for me with bindwood until this is fixed.

Wrote a python script to be launched by System > Preferences > Preferred Applications > Mail Reader > Custom
to enable other browsers to do what FF does with Edit > Preferences > Applications > mailto
and will share it with any interested parties.

Hi, all.

I've got some news for everyone affected by this. While the updated package isn't yet in Universe, you can get an updated package from my PPA:

  ppa:urbanape/bindwood-exp

The updated version should be:

  $ apt-cache policy bindwood
  bindwood:
    Installed: 0.5-0ubuntu1~ppa1
    Candidate: 0.5-0ubuntu1~ppa1
    Version table:
   *** 0.5-0ubuntu1~ppa1 0
          500 http://ppa.launchpad.net karmic/main Packages
          100 /var/lib/dpkg/status
       0.4.2-0ubuntu2 0
          500 http://us.archive.ubuntu.com karmic/universe Packages

This update doesn't solve moving the work to a background thread, but it does alleviate the vast majority of work we do at (subsequent) startup and while the current session is running.

Please give it a try. On the first startup after installing it, Firefox will still take a short time to become responsive, but after the initial sync, it should be far more usable. I'm still working on making all this happen in the background, as it should, but this should get you all back to being productive with FF.

Thanks for your patience,

Zac

I haven't seen any response to this, but I've gotten good feedback from internal testers. I'll be pushing an SRU with this behavior while I work on the threaded version.

Sorry I failed to communicate, but I have been running this since you posted it on 11/23 and have had no issues at all. Thumbs up.

Hi Zac,

Installed the new version as requested and all seems fine. As you say, the 1st startup after install is slow but after that all seems ok.

Thanks again.

All the best
Brendan

Works fine on my two machines running karmic 64bit :)

Zac, thank you for fixing this!

Zac,

This now works for me. Thanks for fixing it.

Tim

installed 0.5-0ubuntu1~ppa1 on karmic 32bit and it just ate up cpu/memory/swap until I ran out :(

This is not fixed on 64 bit Karmic as of 2010-01-13
The Bindwood addon instantly freezes firefox and cause huge memory leak. All memory gets used.

Binary package hint: bindwood

It also uses half of my swap partition. Screen shot attached.

I disabled all other firefox addons and it still causes the massive memory leak. System becomes unresponsive firefox greys out.
Have to use killall firefox to regain control. Memory then returns to normal.

I had to use ubuntu-bug without bindwood installed in order to submit a bug report.

ProblemType: Bug
Architecture: amd64
Date: Tue Jan 12 16:07:14 2010
DistroRelease: Ubuntu 9.10
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release Candidate amd64 (20091020.3)
NonfreeKernelModules: nvidia
Package: bindwood (not installed)
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-17.54-generic
SourcePackage: bindwood
Uname: Linux 2.6.31-17-generic x86_64

See 506496 for ubuntu-bug attachements.

Hi,

Must say I'm not seeing this and I'm on amd64 too...

Not sure what to suggest, just works for me.

Cheers
B

This is an important bug, and Zac is fixing it for Lucid. It's not a round-the-clock bug that needs to be marked as critical though, the workaround is to uninstall bindwood.

I just updated my system to lucid 64 bit. I have been using Ubuntu One for backups since Karmic without problems. As soon as i installed and enabled the bindwood extension, FF 3.6 became unresponsive. I do have over a 100 bookmarks, including some live ones. As long as i disable the bindwood extension, FF 3.6 is very quick. With it enabled, it's not usable.

The same is happening to me as described by Martin G Miller (#40)

Hey there, I was also affected by this bug. I installed bindwood, xul-ext-bindwood 1.0.4-0ubuntu1
i686, lucid with same error which was described by the reporter.

My .local/share/desktop-couch/bookmarks.couch-file are 4GB large (400 bookmarks), I have the same issue at five machines, all lucid. Firefox are unresponsive and turns grey from time to time.

xul-ext-bindwood 1.0.4-ubuntu1

Experienced no unresponsiveness. But today all my bookmarks are gone. My bookmarks.couch file is 324,0 MB! I had merely 50 bookmarks. Surely they've got "compressed" into that file. Anyone lost his/her bookmarks too? What are the chances for recovery?

I disabled Bindwood 1.0.4 and my Firefox is running great again. I didn't really need it installed anyway.
Thank you!

My bookmarks.couch file is 8GB, what the hell?

promptly uninstalling bindwood now, i would rather copy and paste than waste 8GB of disk space on something that doesn't really work

I have seen something that could be interesting, maybe you already know:
Opening ~/.local/share/desktop-couch/couchdb.html you could see a manager of the couchdb, the problem seems to be that FF records a lot of versions for each bookmark (maybe each time you access) and can be solved using "Compact", deleting all old versions and "deleted" bookmarks, now it's 0.8MB.
This is only a temporal solution.

But you can also configure the max size of the database (in "configuration", at the right), for me (Lucid 64, FF 3.6 updated) it's by default at 4GB, the same size I have read here. I change it to 10MB and I will report if it works.
It's the max_document_size, there is another one, max_attachment_chunk_size, that I don't really know what is it, but it was also at 4GB and I changed it to 10MB too.

Maybe if you psypher haven't delete it (or even if you do, maybe) can check the size for you.

NoScript's backup-settings-as-a-bookmark feature seems to cause excessive couchdb growth as well.

About 20 bookmarks and that noscript-settings on Maverick resulted in 245 "documents" in bookmarks taking 9 megabytes of space, with sequence number at 1172 in mere couple days. At times Firefox seemed to be slowly accessing couchdb on every keypress on form fields where javascript was blocked by NoScript.