checkout operation consume too much resources

Bug #269456 reported by Benoit Pierre
26
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
John A Meinel

Bug Description

On my tree, which is fairly big (430Mo, 7222 files in 1233 directories), remove-tree/checkout operations with version 1.5 take:

# time bzr remove-tree
bzr remove-tree 0.03s user 0.01s system 0% cpu 1:58.59 total

# time bzr checkout
bzr checkout 0.01s user 0.01s system 0% cpu 3:36.56 total

During checkout, memory usage peaked at around 60Mo.

With version 1.6.1:

# time bzr remove-tree
bzr remove-tree 0.01s user 0.01s system 0% cpu 2:05.02 total

# time bzr checkout
bzr checkout 0.01s user 0.03s system 0% cpu 10:21.28 total

I killed bzr after 10 minutes of wait, memory usage peaked at ~400Mo!

All tests were done on Windows, using the standalone installer versions. I have the same problem on Linux: bzr hog all the memory, and I kill it before completion. The only way to go around this problem for me is to kill the checkout operation, clean .bzr/checkout/limbo and .bzr/checkout/pending-deletion, break the lock, and do a series of revert. A single revert on the whole tree seems to suffer from the same problem as checkout, so I really have to slice it into smaller revert operations, one sub-directory at a time.

Tags: windows
Revision history for this message
John A Meinel (jameinel) wrote :

So, I believe the specific issue is that the underlying code is buffering all of the working tree before writing it to disk. The apis in question are now requesting all the content in one pass, which is good, but the one handling it should be a bit more aware, and only spool out the request one file at a time.

Changed in bzr:
importance: Undecided → High
status: New → Triaged
John A Meinel (jameinel)
Changed in bzr:
assignee: nobody → jameinel
status: Triaged → In Progress
John A Meinel (jameinel)
Changed in bzr:
status: In Progress → Fix Committed
Revision history for this message
Michael Steele (mikesteele81) wrote :

I see you've committed a patch. I would be happy to test that out if you'll send me a patch for dev or give me a URL to pull from.

Revision history for this message
John A Meinel (jameinel) wrote :

I went ahead and associated the branch with the bug. It has also been submitted for review at:
http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C48E64F01.1000106%40arbash-meinel.com%3E

Revision history for this message
Michael Steele (mikesteele81) wrote :

I found your patch on the mailing list.

I'm now able to extract my 1.3GB repository. Memory usage hovered around 50MB instead of 1GB.

John A Meinel (jameinel)
Changed in bzr:
milestone: none → 1.9rc1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.