MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

Bug #456077 reported by Jorgen Austvik
44
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
Martin Pool
2.0
Fix Released
High
Martin Pool
2.1
Fix Released
Undecided
Unassigned

Bug Description

This is a Sun/MySQL - Canonical escalation imported into Launchpad by the Canonical Support Team (internal case 6614 )

Original escalation: 2009-11-16 10:12 UTC

It takes more than 10 hours to branch mysql 5.0.
In this attempt it used > 4GB virtual memory, > 3 GB real memory, almost no CPU, almost no disk and almost no network.

Before this try, I tried to check out mysql 5.1 (but without the "time" command). I stopped the command after 8 hours (1.5G transfered, still streaming) when my machine was trashing furiously because of swapping.

This is a brand new MacBook Pro with 4GB memory, and it isn't able to cope with the memory requirements of a command line program that is supposed to download some bits.

steve:mysql austvik$ mkdir mysql
steve:mysql austvik$ cd mysql
steve:mysql austvik$ bzr init-repo .
Shared repository with trees (format: 2a)
Location:
  shared repository: .
steve:mysql austvik$ time bzr branch lp:mysql-server/5.0 5.0
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
bzr: ERROR: Connection error: while sending GET /~mysql/mysql-server/mysql-5.0/.bzr/repository/packs/27fd486dfda34acee6644eb49b58ddaa.pack: [Errno 60] Operation timed out

real 632m55.352s
user 560m48.451s
sys 6m24.630s
steve:mysql austvik$ ls -al
total 0
drwxr-xr-x 4 austvik staff 136 Oct 19 19:49 .
drwxr-xr-x 3 austvik staff 102 Oct 19 09:58 ..
drwxr-xr-x 6 austvik staff 204 Oct 19 19:49 .bzr
drwxr-xr-x 3 austvik staff 102 Oct 19 19:49 5.0
steve:mysql austvik$ find 5.0/ | wc -l
       5
steve:mysql austvik$ find .bzr | wc -l
      13
steve:mysql austvik$ bzr --version
Bazaar (bzr) 2.0.1
  Python interpreter: /usr/bin/python 2.6.1
  Python standard library: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
  Platform: Darwin-10.0.0-i386-64bit
  bzrlib: /Library/Python/2.6/site-packages/bzrlib
  Bazaar configuration: /Users/austvik/.bazaar
  Bazaar log file: /Users/austvik/.bzr.log

Problems:
real 632m55.352s

How to fix:
I believe the enormous memory requirements slow things down a lot

I don't care about "bzr: ERROR: Connection error" in this bug report. It is probably a side effect of bazaar using so much memory that the whole computer becomes unresponsive over time.

Related branches

Revision history for this message
Ian Clatworthy (ian-clatworthy) wrote :

Hi Jorgen,

Thanks for the bug report. I'm sorry it took so long. Looking at https://code.launchpad.net/~mysql/mysql-server/mysql-5.1, it looks like that branch is still in our 1.x format. I suspect the primary reason it took so long was because it was actually converting content to the new format on the fly, rather than just transferring it. I'll let one of the more network-savvy developers confirm that.

You may want to check with the MySql team as to when they plan to upgrade their branches to our new format before going much further. If it's a while away, you may need to change the format you're using locally instead of using the default 2.x one.

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 456077] [NEW] bzr branch: unacceptable slow

Two things occured here:

You branched over HTTP which means it did not stream. And MYSql need to
upgrade to the 2a format, which improves memory usage and performance.
The branching performance of very large projects (like mysql5) with
older formats cannot be comprehensively improved because the amount of
effort is quadratic.

If you do 'bzr lp-login' before branching you will get a streaming
branch which is a lot faster and should use less memory.

If the MYSql branches are upgraded, then things will get better still.

 status invalid

I will note that we have significant memory improvements in the works
for 2.1 and there are bugs open on that.

Changed in bzr:
status: New → Invalid
Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 456077] Re: bzr branch: unacceptable slow

On Tue, 2009-10-20 at 07:15 +0000, Ian Clatworthy wrote:
> Hi Jorgen,
>
> Thanks for the bug report. I'm sorry it took so long. Looking at
> https://code.launchpad.net/~mysql/mysql-server/mysql-5.1, it looks like
> that branch is still in our 1.x format. I suspect the primary reason it
> took so long was because it was actually converting content to the new
> format on the fly, rather than just transferring it. I'll let one of the
> more network-savvy developers confirm that.
>
> You may want to check with the MySql team as to when they plan to
> upgrade their branches to our new format before going much further. If
> it's a while away, you may need to change the format you're using
> locally instead of using the default 2.x one.

Conversion will have added to the overhead yes; however we did fix the
huge-roundtrip-count bug on such conversions for 2.0.0, so it should
have been tolerable. Likely the inventory cache was a large contributing
factor to the memory utilisation though.

-Rob

Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote : Re: bzr branch: unacceptable slow

Hi Ian and Robert, thanks for your answers.

Some questions:
1. How do I set the 1.x format?
2. Should one get a warning if HTTP is used, saying "this is slow, do 'bzr lp-login' for fast streaming"?
3. Should one get a warning if two different formats are used, saying which formats is used, and that this is slow?

I will try bzr lp-login.
I will try to get the version upgraded

-J

Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote :

I can confirm that 'bzr lp-login' really helps utilizing my network. Now it is consistently around 550KB/sec download speed - earlier it was a 550KB/s spike every ~10 seconds, and nothing in between.

It also reduced memory usage 10X, from 4GB to 400MB, and I was actually able to branch mysql-5.0 in 16m28.121s.

Memory leak when downloading with HTTP?

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 456077] Re: bzr branch: unacceptable slow

On Tue, 2009-10-20 at 08:01 +0000, Jorgen Austvik wrote:
> I can confirm that 'bzr lp-login' really helps utilizing my network. Now
> it is consistently around 550KB/sec download speed - earlier it was a
> 550KB/s spike every ~10 seconds, and nothing in between.
>
> It also reduced memory usage 10X, from 4GB to 400MB, and I was actually
> able to branch mysql-5.0 in 16m28.121s.
>
> Memory leak when downloading with HTTP?

Conversions take a bunch of memory, and there are file system and index
caches, all of which add up. Streaming means that two machines share the
load, and there are better lifetime management as well.

-Rob

Revision history for this message
John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Tue, 2009-10-20 at 07:15 +0000, Ian Clatworthy wrote:
>> Hi Jorgen,
>>
>> Thanks for the bug report. I'm sorry it took so long. Looking at
>> https://code.launchpad.net/~mysql/mysql-server/mysql-5.1, it looks like
>> that branch is still in our 1.x format. I suspect the primary reason it
>> took so long was because it was actually converting content to the new
>> format on the fly, rather than just transferring it. I'll let one of the
>> more network-savvy developers confirm that.
>>
>> You may want to check with the MySql team as to when they plan to
>> upgrade their branches to our new format before going much further. If
>> it's a while away, you may need to change the format you're using
>> locally instead of using the default 2.x one.
>
> Conversion will have added to the overhead yes; however we did fix the
> huge-roundtrip-count bug on such conversions for 2.0.0, so it should
> have been tolerable. Likely the inventory cache was a large contributing
> factor to the memory utilisation though.
>
> -Rob
>

It took about 48 hours to convert the mysql repo I have to 2a. So if he
was doing cross-format conversions on the fly, that would be an
approximate ballpark of how long it would take to get a local copy.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrd3rcACgkQJdeBCYSNAAPosACgpj6Zexwuq6PLBTTbYpr4/L2/
/i8AoKoy38Wg4FgAmiwLPFV1s+GFnzXn
=92hq
-----END PGP SIGNATURE-----

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote : Re: bzr branch: unacceptable slow

Hello. What do you guys mean by "two machines" in "Streaming means that two machines share the
load": the server and my client machine? Or does Launchpad have multiple servers sending to one client machine (like a torrent)?
We haven't had such download issues when branching from our internal server which has repo format 1.9 but where we use bzr+ssh - does this mean the problem is specific of branching over http?
I will likely have more questions later regarding the upgrade of MySQL repositories, internal and Launchpad, to format 2a.
Thanks!

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 456077] Re: bzr branch: unacceptable slow

On Tue, 2009-10-20 at 21:05 +0000, GuilhemBichot wrote:
> Hello. What do you guys mean by "two machines" in "Streaming means that two machines share the
> load": the server and my client machine?

Yes. The server sends plain texts, and inventory deltas, the client
doesn't buffer, just extracts and compresses.

> Or does Launchpad have multiple servers sending to one client machine (like a torrent)?
> We haven't had such download issues when branching from our internal server which has repo format 1.9 but where we use bzr+ssh

> - does this mean the problem is specific of branching over http?

Its a major component of it, yes.

> I will likely have more questions later regarding the upgrade of MySQL repositories, internal and Launchpad, to format 2a.
> Thanks!

Sure thing.

-Rob

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote : Re: bzr branch: unacceptable slow

Hello, I got a new report of this, with different conditions, which may help to see clearer.
Colleague uses bzr 2.0.2 (from debian).
Does
 bzr init-repo repo
 cd repo
 bzr branch bzr+ssh://our_central_host/some_branch
it runs for 6 hours and is not completed.
Turns out that the repo he created is in 2a format, because that's the default of init-repo in bzr 2.0.
And our_central_repo is in format "1.9 or 1.14" according to "bzr info -v".
So what happens is likely on-the-fly conversion and he's up for 48 hours, John says above.
Then I say "kill this job, and recreate your repo with 'bzr init-repo --1.9' ". Then the branch completes in 45 minutes.
It's bzr+ssh, not http.
So we have this problem:
as more and more people upgrade to bzr 2.0 (in his case, it was an automatic upgrade, so I didn't expect him to read lots of release notes anyway), they will more and more hit the on-the-fly 48-hour conversion. It will hit Community people branching from Launchpad (and will surely discourage them, I don't think 100% of them will search for a solution on IRC or forums).
Solutions:
1) could you please consider printing a message on stdout when bzr sees that it will do on-the-fly conversion? Something like "we will do on-the-fly conversion from format X to format Y; it may be slow; if you don't want this, create a repo with format X, see 'bzr help init-repo'"? Printing a message has been suggested by a colleague and I like this idea, it does not cost much and can help other colleagues and the Community.
2) in the end we should upgrade our central host to format 2a. But there are lots of things to examine before, among which:
 - how bugfree is 2a? our team is 100 people (Windows, OS X, Linux, Solaris)
 - what minimum bzr version will colleagues need to install to be able to communicate with our central host when it is in 2a format?
 - will mirroring to launchpad break?
 - etc
so this is not for today, which is why I'd like 1) to be considered.
Thanks!

tags: added: mysql
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

One more subquestion for 2): as the upgrade takes 48 hours, which is too long downtime, we'll need to design a multi-step upgrade; I guess:
take original repo offline; copy the original repo (minutes); take the original repo online;
upgrade the copy's format (48 hours);
take the original repo offline; pull the original's repos changes into the copy (minutes); take the original repo online; eof.

Something observed by the colleague when on-the-fly conversion happened:
he had:
his laptop (bzr 2.0, 2a repo) branching from our central host (bzr 1.17, 1.9-or-1.14 repo):
- laptop: no bzr cpu use, no disk size increase
- no network traffic
- central host: bzr running at 100% cpu (user), increasing slowly in memory use ~450M
So it looks like the central host (bzr 1.17) knows the 2a format and performs the conversion while the laptop waits for the converted result?

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 456077] Re: bzr branch: unacceptable slow

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

GuilhemBichot wrote:
> One more subquestion for 2): as the upgrade takes 48 hours, which is too long downtime, we'll need to design a multi-step upgrade; I guess:
> take original repo offline; copy the original repo (minutes); take the original repo online;
> upgrade the copy's format (48 hours);
> take the original repo offline; pull the original's repos changes into the copy (minutes); take the original repo online; eof.

Right. We've actually outlined that process here:
http://doc.bazaar-vcs.org/latest/en/upgrade-guide/index.html?highlight=upgrade#migrating-data-to-the-new-default-format

>
> Something observed by the colleague when on-the-fly conversion happened:
> he had:
> his laptop (bzr 2.0, 2a repo) branching from our central host (bzr 1.17, 1.9-or-1.14 repo):
> - laptop: no bzr cpu use, no disk size increase
> - no network traffic
> - central host: bzr running at 100% cpu (user), increasing slowly in memory use ~450M
> So it looks like the central host (bzr 1.17) knows the 2a format and performs the conversion while the laptop waits for the converted result?
>

So bzr 1.17 does know about the 2a format, but the server does not do
the conversion. There are periods of time where the server is computing
what to send next where the client will be waiting on data.

I honestly can't say for sure what the server memory consumption profile
was w/ bzr 1.17. I would say that it really depends on how long this was
observed. If it was ~1 minute, then it would probably just be paging in
information about what needs to be sent. (That would be long, but
possible I suppose). If it was an hour, then it certainly is something
to investigate, as that is a lot longer than I would have expected.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkr0TuQACgkQJdeBCYSNAAMhlgCdEZsso+8NYxZIQN4WVLcNdtz/
ImcAoM/+k9IUD6eV2zQyxNEpYV0xtDvO
=5Sqz
-----END PGP SIGNATURE-----

Revision history for this message
Matthew Fuller (fullermd) wrote :

> 1) could you please consider printing a message on stdout when bzr
> sees that it will do on-the-fly conversion?

Please. Even if there's not an expensive conversion like 1.9->2a to
do, I think it should print a notice when the formats differ.

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote : Re: bzr branch: unacceptable slow

> 1) could you please consider printing a message on stdout when bzr
> sees that it will do on-the-fly conversion?
I fear that without this warning, all MySQL developers are going to hit the 48-hour branch problem, as they'll all upgrade to bzr 2 one day or the other.

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

A third colleage was just bitten by this silent very long auto-conversion. I would really like suggestion 1) to be considered and implemented soon.

summary: - bzr branch: unacceptable slow
+ MYSQL/BZR P3: bzr branch: unacceptable slow
description: updated
visibility: public → private
Revision history for this message
Martin Pool (mbp) wrote : Re: MYSQL/BZR P3: bzr branch: unacceptable slow

reopening

Changed in bzr:
importance: Undecided → Medium
status: Invalid → Incomplete
Revision history for this message
Tim Penhey (thumper) wrote :

There is a bug where branching into an empty shared repository where it does a linear walk back from the tip revision.

Perhaps this should also be checked, tweaked, or special cased.

Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote :

Hi, Martin. I see that you set the status to "Incomplete", and that the description of this status in Launchpad is "Incomplete: the bug report is incomplete and needs more information before it can be triaged".

I have tried to figure out which information is needed by searching for question marks in the text above, and they all come from comments by either Guilhem or me. If you need information from the user, please indicate what kind of information you need and who you need it from.

Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote :

Hi, Dragomir.

I would like to state that I have not added any information to this thread that I believe need to be private. I also see that Guilhem has anonymized his paths etc.

I don't think there is any security issue here, just slowness.

Why has the visibility been changed to private?

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

To Tim: this slowness also happens with bzr+ssh:// when branching from our central host to collagues' machines, so isn't specific of Launchpad.

Revision history for this message
Martin Pool (mbp) wrote :

Hi,

It's not incomplete in the sense of blocked, that was just transient while editing.

I think we understand the problem:

 * there should be a warning as you suggest (easy)
 * there should be an option to set the default format (easy, bug 484101)
 * conversion should be faster - probably a separate bug; gradually being fixed separately

--
Martin <http://launchpad.net/~mbp/>

Changed in bzr:
status: Incomplete → Confirmed
summary: - MYSQL/BZR P3: bzr branch: unacceptable slow
+ MYSQL/BZR P3: cross format fetch is mysteriously very slow
Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote : Re: MYSQL/BZR P3: cross format fetch is mysteriously very slow

On the #drizzle IRC channel yesterday somebody had the same problem, but I couldn't give a link to this report since it is private.

Can somebody please give a good reason for keeping this report private?

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

removing the "private" mark, nothing secret in this.

visibility: private → public
Revision history for this message
Jorgen Austvik (jorgen-austvik) wrote : Re: MYSQL/BZR P3: cross format fetch is very slow

Cannot be "mysteriously" when Martin Pool writes "I think we understand the problem" and the definition of mystery is "something not understood or beyond understanding" - http://www.merriam-webster.com/dictionary/mystery.

summary: - MYSQL/BZR P3: cross format fetch is mysteriously very slow
+ MYSQL/BZR P3: cross format fetch is very slow
Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: cross format fetch is very slow

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jorgen Austvik wrote:
> Cannot be "mysteriously" when Martin Pool writes "I think we understand
> the problem" and the definition of mystery is "something not understood
> or beyond understanding" - http://www.merriam-
> webster.com/dictionary/mystery.
>
> ** Summary changed:
>
> - MYSQL/BZR P3: cross format fetch is mysteriously very slow
> + MYSQL/BZR P3: cross format fetch is very slow
>

The point was that it was mysterious to the *users* who encountered it.
Not that it was a mystery to the developers.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksEDygACgkQJdeBCYSNAANOvQCfQL9Cl0Zd3NvysGb62j3u5TOT
J0YAniWafIujbcgkL9tkDu28pBtR/rEs
=ht9t
-----END PGP SIGNATURE-----

Martin Pool (mbp)
summary: - MYSQL/BZR P3: cross format fetch is very slow
+ MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch
tags: added: easy
removed: branch
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

Hello. One user hit it again today.
Please, please, do something for this bug. Printing a simple warning would greatly help. See Martin Pool's comment above where he wrote
" * there should be a warning as you suggest (easy)".

John A Meinel (jameinel)
Changed in bzr:
milestone: none → 2.1.0
Martin Pool (mbp)
Changed in bzr:
assignee: nobody → Martin Pool (mbp)
status: Confirmed → In Progress
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

Thanks Martin.
Note that the fetch is between a _remote_ machine and the developer's local machine, via bzr+ssh.
So I'm not sure the patch will work (changelog says 'Give a warning when fetching between local repositories").
I just tested it, and after 5 minutes, bzr on the remote host is at 100% CPU, and I don't see any warning on my machine's screen.
If you want to create a test scenario mimicking our problem, just do:
local> bzr init-repo r # make sure to use bzr 2.x
local> cd r
local> bzr branch bzr+ssh://some_remote_machine/some_branch
You should see 100% CPU on the remote machine and a very long branch time (with a MySQL branch it is not finished after a day, I am even told that the process on the local machine finally dumps core).
Also, when we kill this "bzr branch" job on the local machine (using Control-C), the "bzr" process on the remote machine keeps running, at 100% CPU, at least for 10 minutes (after which I killed it) which isn't quite good as the remote machine is a shared machine.

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

I forgot an important point, for the test scenario: the branch on the remote machine must be in a shared repo of format *1.9*. So that 1.9->2a conversion happens.

Revision history for this message
Andrew Bennetts (spiv) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

GuilhemBichot wrote:
> Thanks Martin.
> Note that the fetch is between a _remote_ machine and the developer's local
> machine, via bzr+ssh.
> So I'm not sure the patch will work (changelog says 'Give a warning when
> fetching between local repositories").

That's true. Martin did say in the merge proposal that he'd shortly follow it
with a patch for the non-local code path.

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

2010/1/12 Andrew Bennetts <email address hidden>:
> GuilhemBichot wrote:
>> Thanks Martin.
>> Note that the fetch is between a _remote_ machine and the developer's local
>> machine, via bzr+ssh.
>> So I'm not sure the patch will work (changelog says 'Give a warning when
>> fetching between local repositories").
>
> That's true.  Martin did say in the merge proposal that he'd shortly follow it
> with a patch for the non-local code path.

That's right: for better or worse this needs to be hooked in to two
different places, or at least to get two different tests. The
local-only one is now on its way to 2.0, and I'll go back to the
network one now.

--
Martin <http://launchpad.net/~mbp/>

Martin Pool (mbp)
Changed in bzr:
importance: Medium → High
Revision history for this message
Martin Pool (mbp) wrote :

A branch that fixes the network case too is now on its way through pqm.

Revision history for this message
Martin Pool (mbp) wrote :

This should be fixed in the next release; further feedback is welcome of course.

Changed in bzr:
milestone: 2.1.0 → 2.1.0rc1
status: In Progress → Fix Released
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

I have bzr revno 4967 (pulled this morning).
I branched, over bzr+ssh://, a MySQL code branch residing on a 1.9-format repo on a remote machine, into a 2a repo on my machine, and so far I am here:
| 300973KB 232KB/s | Fetching revisions:Inserting stream
So, 300MB downloaded and still not a warning; has been running for 4 hours and bzr is at 100% CPU on the remote side (no CPU usage on mine).
Has the fix been tested over bzr+ssh? If not, please do... Thanks!

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

And 30 minutes after killing the above "bzr branch" job on my local machine, the bzr process on the remote machine is still running at 100% CPU. I suspect that this is a bug. There should be reasonably periodic detection, on the remote side, that the local side process has gone away. Otherwise the warning, even if it is printed, loses interest (it warns you, you control-C your bzr command, but it still hogs the remote machine for a long time).
Some piece of information, maybe useless: on the local machine, when I killed bzr after it had exchanged 300MB on the network and ran for 4 hours, my branch was only 8KB.

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

On the remote machine:
Bazaar (bzr) 1.17dev
  Python interpreter: /usr/bin/python 2.4.4

Revision history for this message
Matt Nordhoff (mnordhoff) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

GuilhemBichot wrote:
> Some piece of information, maybe useless: on the local machine, when
> I killed bzr after it had exchanged 300MB on the network and ran for
> 4 hours, my branch was only 8KB.

Did you check the size before or after killing it? There should have
been a pile of data in .bzr/repository/upload, which bzr would have
deleted when it exited.
--
Matt Nordhoff

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Nordhoff wrote:
> GuilhemBichot wrote:
>> Some piece of information, maybe useless: on the local machine, when
>> I killed bzr after it had exchanged 300MB on the network and ran for
>> 4 hours, my branch was only 8KB.
>
> Did you check the size before or after killing it? There should have
> been a pile of data in .bzr/repository/upload, which bzr would have
> deleted when it exited.

Given the symptoms, I'm going to guess something else.

1) I believe the "Generic" fetch code has to stay 'knit' compatible. So
it always fetches

  texts
  inventories
  revisions (and signatures)

In that order.

2) To determine what texts need to be sent, it first has to read the
inventory data looking for text keys that need to be transmitted.

3) The code Martin added to give a warning uses the fact that
inventories are transmitted differently when there is a cross-format
fetch. Which according to (1) means that all the texts have to be
transmitted before it issues the warning.

And in the cross-format case, I think we'll be recompressing all of
those 300+MB of text content before we give the warning.

Obviously this is not satisfactory for the case we are trying to solve.

I think moving the check to be part of "_get_source()" might work. In
that code we'll have a check for "source.network_name() ==
target.network_name()", and we can warn when they aren't identical.

It isn't foolproof. In that fetching from a pack-0.92 format into a 1.9
format has no extra overhead. (The only change is the index format,
which are always generated locally anyway.)

It is possible that we would want to instead special case specific
conversion combinations.

I'll bump this back to needs to be worked on, since the current status
is not satisfactory.

 status: confirmed

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktQjTUACgkQJdeBCYSNAANEIACfRo2g3b9mIZUHbe5ue7JXljNs
gKQAoMcw4dgpD0htmE5uFi41mg4PrzOn
=gbZA
-----END PGP SIGNATURE-----

Changed in bzr:
status: Fix Released → Confirmed
Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

On Fri, 2010-01-15 at 15:43 +0000, John A Meinel wrote:
>
> Given the symptoms, I'm going to guess something else.
>
> 1) I believe the "Generic" fetch code has to stay 'knit' compatible.
> So
> it always fetches

The streaming interface knows what format its sending to, so it doesn't
need to stay strictly knit compatible. It just has to be knit compatible
when sending to knits. HTH

-Rob

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

If there's any info I can provide to help, let me know. I'd really like to have an early warning printed, and also that the 100%-CPU remote bzr process dies quickly when the local process is interrupted.

Revision history for this message
Matthew Fuller (fullermd) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

> It isn't foolproof. In that fetching from a pack-0.92 format into a
> 1.9 format has no extra overhead.

I don't think a little overwarning is that big a deal. And even
leaving performance to one side, a hint as to when a project has
changed formats since you made your copy of it is nice.

John A Meinel (jameinel)
Changed in bzr:
status: Confirmed → Fix Released
Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

Hi John. What do you mean by "fix released"? I was under impression that the warning isn't issued in a reasonable amount of time and so more work was needed as you wrote previously. Thanks for help.

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

I think John changed it (maybe semi-automatically) because my patch related
to it has landed. From your feedback it's clear it is not really fixed so we
should reopen it. I'm on vacation this week but will go back to it soon, if
no one else gets it first.

- Martin

On 21/01/2010 10:41 PM, "GuilhemBichot" <email address hidden>
wrote:

Hi John. What do you mean by "fix released"? I was under impression that
the warning isn't issued in a reasonable amount of time and so more work
was needed as you wrote previously. Thanks for help.

-- MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch
https://bugs.launchpad.n...

Revision history for this message
Martin Pool (mbp) wrote :

There is an additional small bug in the currently landed implementation, which is that the message appears while upgrading asking you to upgrade, which looks a bit silly.

Changed in bzr:
status: Fix Released → Confirmed
Martin Pool (mbp)
Changed in bzr:
status: Confirmed → In Progress
Revision history for this message
Martin Pool (mbp) wrote :

The other related bugs are <https://bugs.launchpad.net/bzr/+bug/513157> and <https://bugs.launchpad.net/bzr/+bug/515356>.

Perhaps we should revert this from 2.0, because it's not clearly a sru-qualifying bugfix, and there has been some fallout.

Revision history for this message
Martin Pool (mbp) wrote :

The attached branch now gives a message earlier in the process.

Revision history for this message
GuilhemBichot (guilhem-bichot) wrote :

Hello Martin. I don't know what you mean by "revert this from 2.0", but surely I am happy that the warning is printed, and have had many colleagues hit by the silent super-slow upgrade, which is why I insisted on getting a warning.
So if this is to be reverted, this is an issue for us again.
Our central repo is still 1.9/1.14. I am making preparations for the upgrade to format 2a, but there are things which slow me down on the upgrade route: corruption as in https://bugs.launchpad.net/bzr/+bug/527035, slower annotate as in https://bugs.edge.launchpad.net/bzr/+bug/153787. So the upgrade may only happen in a few months.
So if the warning is reverted, colleagues will be bitten again for a few months.

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

GuilhemBichot wrote:
> Hello Martin. I don't know what you mean by "revert this from 2.0", but surely I am happy that the warning is printed, and have had many colleagues hit by the silent super-slow upgrade, which is why I insisted on getting a warning.
> So if this is to be reverted, this is an issue for us again.
> Our central repo is still 1.9/1.14. I am making preparations for the upgrade to format 2a, but there are things which slow me down on the upgrade route: corruption as in https://bugs.launchpad.net/bzr/+bug/527035, slower annotate as in https://bugs.edge.launchpad.net/bzr/+bug/153787. So the upgrade may only happen in a few months.
> So if the warning is reverted, colleagues will be bitten again for a few months.
>

i think what he is saying is that he is reverting the old fix (which
gave the message very late) in favor of the new one which gives it earlier.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuFNGAACgkQJdeBCYSNAAPKTQCcDpwPnMKmajYSh6Ip6euR42+j
/4MAoKOLr4s3KSrdfZVCZRZ9la2alTSR
=4Z/n
-----END PGP SIGNATURE-----

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 456077] Re: MYSQL/BZR P3: bzr doesn't explain it's doing a slow cross-format fetch

Hi Gulhem,

What I meant was that perhaps, while the message is incorrectly shown
on upgrades etc, we should pull it out, but I'm no longer planning to
do that. I now have a fix for the fallout, that causes it to be shown
much earlier and shown correctly for remote branches and upgrades. I
hope that'll be landed next week and then this will be properly fixed.

Revision history for this message
Martin Pool (mbp) wrote :
Revision history for this message
Martin Pool (mbp) wrote :

This should now really be fixed in 2.2b1, 2.1.1, and 2.0.5.

Changed in bzr:
milestone: 2.1.0rc1 → 2.2b1
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.