arm64 autopkgtests are flaky

3 failures out of 54 test runs in the past 30 days, that's a 5.6% failure rate.

1 failure out of 57 test runs in the past 30 days, that's a 1.75% failure rate.

Christian raised this with us just now - I think we should consider taking a look at this.

Looking at

http://autopkgtest.ubuntu.com/packages/libreoffice/hirsute/arm64
http://autopkgtest.ubuntu.com/packages/libreoffice/hirsute/armhf

the tests are basically OK but one of the uicheck-* ones ends up timing out often enough that it can be described as flaky. When this happens it causes a problem for other migrations, doubly so as the runs take many hours to complete whether they fail or not, which slows down migration speed.

If it's not possible to fix the timeouts (feels like it might be hard or take a while) then disabling these subtests (only) on arm would be an option - stable tests with less coverage are better than flaky ones with more.

We had

https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=baf5a581da7ce3bb27a5b8ed44fadeb6e245e1ff

in the past which might be a basis for what to do now, but that would want to be extended to arm64 too.

This recently showed up often enough that I bothered to find this bug :-/
I think it degraded again into a state where we should consider:

IRC:
[13:16] <cpaelzer> Was there any hope to not expect this to be a retry-monster forever?
[13:17] <Laney> The desktop team should be asked if they can work on making that less flaky
[13:17] <Laney> I don't think ignoring the tests there is a good answer
[13:18] <Laney> Reducing the coverage would be preferable if that gets it to be more stable, for example, assuming the flakiness can't be easily fixed

I have a simple test stats gatherer [1] that also shows the subtests which helps a lot in those cases to identify potential candidates to skip.

I attached the output, best viewed in monospace
AFAICS up until recently we had <15% on all architectures.
But in Hirsute on arm* I see a bump to 22% of the subtest "uicheck-sw".
Could be bad-luck / noise - but maybe it is worth considering to skip that on arm?

[1]: https://git.launchpad.net/~ubuntu-server/ubuntu-helpers/tree/cpaelzer/check-autopkgtest-stats.sh

I agree that the uicheck-sw test has increased in flakiness. How about instead of removing the test or sinking the resources into actually fixing them right now, I can mark them flaky so that at least a failed test looks like a skipped test:

https://salsa.debian.org/ci-team/autopkgtest/blob/master/doc/README.package-tests.rst

On Fri, Nov 20, 2020 at 9:00 PM Heather Ellsworth
<email address hidden> wrote:
>
> I agree that the uicheck-sw test has increased in flakiness. How about
> instead of removing the test or sinking the resources into actually
> fixing them right now, I can mark them flaky so that at least a failed
> test looks like a skipped test:

Would work for me, can we mark flaky-on-armhf only (I'd not be aware)?

Tagging notfixing, the bug is assigned and we will try to get it fixed rather sooner than later but we don't believe it's a release issue

> we don't believe it's a release issue

It would be good to elaborate when making such comments.

It's a drag on the archive in general when we have flaky tests, and even more so when they take many hours and potentially multiple retries to succeed.

What's the criteria you're using here? Are we saying that "release" is only for user facing issues and bugs which have a negative impact on development of Ubuntu itself aren't to go through this process? If so, I think the team should come up with a way of being able to raise such issues.

That would seem to rule out FTBFS for example too.

> Would work for me, can we mark flaky-on-armhf only (I'd not be aware)?

The only way I could think to do this would be to duplicate the test and use "Architecture:" to run the "Restrictions: flaky" one on armhf arm64 only and the other one otherwise.

In this case the failure is a *timeout* though so I'm not sure that Restrictions: flaky works since it is being killed from the outside by autopkgtest itself (the test runner). That would need to be tested to make sure it does. You could do that by making a simple (in another package, doesn't have to be libreoffice) which has "Restricitons: flaky" and runs something like "sleep infinity". See what happens when autopkgtest times out the test - you can pass --timeout-test to reduce this value for testing.

Onto the test itself a bit.

When the "uicheck-sw" test passes, it takes in the region of 20-25 minutes. Picking two recent results:

armhf:

autopkgtest [05:43:20]: test uicheck-sw: [-----------------------
autopkgtest [06:03:54]: test uicheck-sw: -----------------------]
autopkgtest [06:03:58]: test uicheck-sw: - - - - - - - - - - results - - - - - - - - - -
uicheck-sw PASS

arm64:

autopkgtest [02:52:01]: test uicheck-sw: [-----------------------
autopkgtest [03:05:33]: test uicheck-sw: -----------------------]
autopkgtest [03:05:34]: test uicheck-sw: - - - - - - - - - - results - - - - - - - - - -
uicheck-sw PASS

but the timeout we have (this is a "large" test, it runs with higher timeouts than we use by default) is after over 5 *hours* 30 minutes. It's clearly wildly variable, that's more than 10× as much time and it still timed out.

If this is to do with the test runners being "loaded" (slower) vs "unloaded" (faster), then it could be a race condition that happens when the test runs slowly that causes it to get stuck?

I'm not sure if you want to go down the route of investigating the actual failure, though, but it does feel to me like this "shouldn't" be failing. We're not usually close to the limits and just being tipped over them sometimes, does look like the test is actually getting stuck. But it would require someone to actually look at the tests themselves to determine that.

Clearly having the tests working would be ideal; we want assurance that Libreoffice works on armhf and arm64.

In the absence of this, since we're just mashing the retry button and not actually trying to fix them (ATM), making this stop blocking us one way or another would be better than the status quo.

> That would seem to rule out FTBFS for example too.

We have been rls-not-fixing build failures in the past as well yes, especially in non lts series

After some more investigation, here are the main points that make this issue a tricky one to debug:
1. The test failure is not isolated. While the uicheck-sw test is the culprit, it is just an umbrella for many other tests. And it's those smaller tests that the failure occurs all over. This makes it hard to just "fix the tests".
2. I have not been able to reproduce this locally on a rpi4. Laney pointed out that perhaps having multiple containers running the same test might generate the load needed (assuming that plays a key role in the failures seen on LP) and I'll certainly give this a shot.
3. Currently if you look at the recent hirsute armhf and arm64 LO test results, you'll see that the rate of failure is way down (about 1/10 times). But just a couple of weeks ago the failure rate was much higher. This suggests that it's load on the nodes causing the problem because that's when the infra was quite busy.
4. Marking the test simply as "flaky" won't work because it's actually autopkgtest killing the process

The approach going forward is to try and be clever about letting this uicheck-sw test timeout and not be considered a failure:
https://git.launchpad.net/~hellsworth/+git/libreoffice/commit/?h=ubuntu-hirsute-7.0-workaround-arm-flakiness&id=63170c4e1824dcee45b670738ff47a6174a991a8

A test build on both armhf and arm64 to try out these changes:
https://launchpad.net/~hellsworth/+archive/ubuntu/lo-arm-flaky-testing

This issue is not restricted to arm* but have been seen occasionally on amd64.
It effects the binary package build too, where these tests are part of "make check".

There are a lot tests disabled in 'sw/qa/uitest/writer_dialogs/openDialogs.py' for similar reasons.

So while it locks up at "test_21_.uno:EditGlossary (openDialogs.openDialogs) ... " I would give that a try:
https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=0136e1c7a8f8823cbd0e5027ed203ee0d2f71262

On Wed, Dec 09, 2020 at 10:46:52PM -0000, Rico Tzschichholz wrote:
> This issue is not restricted to arm* but have been seen occasionally on amd64.
> It effects the binary package build too, where these tests are part of "make check".
>
> There are a lot tests disabled in
> 'sw/qa/uitest/writer_dialogs/openDialogs.py' for similar reasons.
>
> So while it locks up at "test_21_.uno:EditGlossary (openDialogs.openDialogs) ... " I would give that a try:
> https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=0136e1c7a8f8823cbd0e5027ed203ee0d2f71262

Thanks for the bug reference - I had an unconfirmed theory that the test
was deadlocking somewhere!

I think Heather mentioned seeing this in other tests too - it would be
good to grep the failing logs to see... seems like the string when it
starts a test doesn't contain a \n, so it looks like:

  test_21_.uno:EditGlossary (openDialogs.openDialogs) ... autopkgtest [22:36:52]: ERROR: timed out on command "...

If it's only this one, or only a small number of tests which can hang
(on slow / loaded machines particularly it seems) always, then skipping
those few would be a reasonable solution, indeed.

But if this can apply to any of the uicheck tests, then I think I'm
still in favour of using timeout(1) to paper over the deadlock, by
setting our own timeout, lower than the autopkgtest one, and skipping if
we hit it.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

This bug was fixed in the package libreoffice - 1:7.0.4~rc2-0ubuntu2

---------------
libreoffice (1:7.0.4~rc2-0ubuntu2) hirsute; urgency=medium

  * Revert "Re-enable coinmp (LP: #1905422)" until MIR is sorted out

 -- Rico Tzschichholz <email address hidden> Mon, 14 Dec 2020 10:05:19 +0100

This bug was fixed in the package libreoffice - 1:7.1.1~rc2-2ubuntu1

---------------
libreoffice (1:7.1.1~rc2-2ubuntu1) hirsute; urgency=medium

  * Fix autopkgtest failures

 -- Rico Tzschichholz <email address hidden> Mon, 01 Mar 2021 11:17:32 +0100

Hello Olivier, or anyone else affected,

Accepted libreoffice into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libreoffice/1:7.0.5-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Groovy verified per LP: #1913353.

This bug was fixed in the package libreoffice - 1:7.0.5-0ubuntu0.20.10.1

---------------
libreoffice (1:7.0.5-0ubuntu0.20.10.1) groovy; urgency=medium

  * New upstream release (LP: #1913353)

  [ Rico Tzschichholz ]
  * Update yaru icon style "2021-01-31"
  * Move libreoffice-sdbc-hsqldb to Recommends (LP: #1916786)
  * Fix Calc crash in ScSelectionTransferObj (LP: #1897784)
  * Drop fix-bluez-external.diff and unowinreg-static-libgcc.diff,
    applied upstream

  [ Heather Ellsworth ]
  * Add timeout to uicheck-sw test to allow it to handle timeouts under
    load better (LP: #1785262)

libreoffice (1:7.0.4-3) unstable; urgency=medium

  * debian/tests/control.in: *really* add libreoffice-writer dependency
    to uicheck-sc test

libreoffice (1:7.0.4-2) unstable; urgency=medium

  * debian/test/control: make uicheck-sc depend on libreoffice-writer, too
    (the openDialogs/uno.Show:License Dialog test opens a new "Writer/Web"
    document...)

libreoffice (1:7.0.4-1) unstable; urgency=medium

  * LibreOffice 7.0.4 final release (identical to rc2)

  * debian/patches/pdfium-m68k.diff: fix pdfium build on m68k

  * debian/rules, debian/control*in: s/noinsttests/noinsttest/, thanks
    lintian
  * debian/rules:
    - revert clang (>= 1:11) build-dep for buster-backports; doesn't exist in
      buster and we resort back to gcc
    - don't rm LICENSE.html, it is used by
      Help -> License Information -> Show License
  * debian/control.mediawiki.in: remove Homepage: (closes: #978713)
  * debian/*.mime: stop quoting %s (closes: #950319)

libreoffice (1:7.0.4~rc2-1) unstable; urgency=medium

  * New upstream release candidate

libreoffice (1:7.0.4~rc1-1) unstable; urgency=medium

  * New upstream release candidate

  * debian/patches/unowinreg-static-libgcc.diff: use -static-libgcc, see
    LO master commit 01241113947fc7bd7f7b765dd897bb023c8ca99c

  * debian/rules:
    - set MYSQL_FLAVOUR=mariadb (as it's used anyway, and upstreams internal
      copy is mariadb-connector-c anyway) and build-depend on libmariadb-dev
      instead of libmariadbclient-dev-compat (closes: #975481)
    - build-depend on clang (>= 1:11) on armel
    - fix nocheck -b builds...
    - use nowindows build profile for (not) building unowinreg
  * debian/rules, debian/control*in:
    - honour noinsttests build profile for (not) building
      -subsequentcheckbase/-smoketest-test-data
  * debian/rules, debian/control*in:
    - move unowinreg.dll into /usr/lib/i686-w64-mingw32 (the mingw g++
      compiler installs its .dlls into /usr/lib/gcc/i686-w64-mingw32)

 -- Rico Tzschichholz <email address hidden> Mon, 15 Mar 2021 19:01:12 +0100

The verification of the Stable Release Update for libreoffice has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.