FTBFS in arm64: backtrace failures in build-time tests

I found this reported to Debian at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962630 and fixed with https://salsa.debian.org/debian/dovecot/-/merge_requests/8 at 1:2.3.10.1+dfsg1-2 version.

Applying and checking ...

It seems that it is applied, except that later the libunwind-dev dependency was added again... checking if that can be the origin of this FTBFS.

It passed locally, although throwing debug info that wasn't shown when libunwind was used:

   dh_auto_test
        make -j8 check "TESTSUITEFLAGS=-j8 --verbose" VERBOSE=1
[...]
make[5]: Entering directory '/<<PKGBUILDDIR>>/src/lib'
for bin in test-lib; do \
  if ! ./$bin; then exit 1; fi; \
done
[..]
array_free ........................................................... : ok
Debug: backtrace: './test-lib(+0x80448) [0xaaaae5780448] -> ./test-lib(test_backtrace+0x48) [0xaaaae572e518] -> ./test-lib(+0x7da94) [0xaaaae577da94] -> ./test-lib(test_run_named_with_fatals+0x84) [0xaaaae577e0e8] -> /lib/aarch64-linux-gnu/libc.so.6(+0x27580) [0xffff83c57580] -> /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff83c57658] -> ./test-lib(_start+0x30) [0xaaaae57298f0]'
backtrace_append ..................................................... : ok
Debug: backtrace: './test-lib(+0x80448) [0xaaaae5780448] -> ./test-lib(backtrace_get+0x28) [0xaaaae57805b8] -> ./test-lib(test_backtrace+0xf4) [0xaaaae572e5c4] -> ./test-lib(+0x7da94) [0xaaaae577da94] -> ./test-lib(test_run_named_with_fatals+0x84) [0xaaaae577e0e8] -> /lib/aarch64-linux-gnu/libc.so.6(+0x27580) [0xffff83c57580] -> /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffff83c57658] -> ./test-lib(_start+0x30) [0xaaaae57298f0]'
backtrace_get ........................................................ : ok
[..]

All tests passed and the package was built locally.

What I wonder is, why this didn't happen before? libunwind was added to dovecot again in version 1:2.3.17.1+dfsg1-1 and the last buildlog for dovecot on arm64 was ok (using libunwind-dev 1.6.2-3 too) [1].

Checking the build on PPA [2] for all architectures...

[1] https://launchpadlibrarian.net/682563409/buildlog_ubuntu-mantic-arm64.dovecot_1%3A2.3.20+dfsg1-1ubuntu2_BUILDING.txt.gz
[2] https://launchpad.net/~mirespace/+archive/ubuntu/dovecot-lp2036268

Because I feel the above is like a regression, I checked the default toolchain to see the differences, and I found that -mbranch-protection=standard is used by default now on mantic, and it was not used in the mantic build [1] prior to this mantic rebuild.

Checking it ...

[1] https://launchpadlibrarian.net/682563409/buildlog_ubuntu-mantic-arm64.dovecot_1%3A2.3.20+dfsg1-1ubuntu2_BUILDING.txt.gz

Stripping the -mbranch-protection flag results in a successful build, similar to what we got before the rebuild:

make[5]: Entering directory '/<<PKGBUILDDIR>>/src/lib'
for bin in test-lib; do \
  if ! ./$bin; then exit 1; fi; \
done
aqueue ............................................................... : ok
array elem ........................................................... : ok
array count/empty .................................................... : ok
array foreach ........................................................ : ok
array foreach reverse ................................................ : ok
array foreach_elem ro/rw strings ..................................... : ok
array reverse ........................................................ : ok
array_lsearch ........................................................ : ok
array compare (ushort) ............................................... : ok
array compare (char*) ................................................ : ok
array_free_without_data .............................................. : ok
array_free ........................................................... : ok
backtrace_append ..................................................... : ok
backtrace_get ........................................................ : ok
base32_encode() with padding ......................................... : ok
base32_encode() no padding ........................................... : ok
base32hex_encode() with padding ...................................... : ok
base32hex_encode() no padding ........................................ : ok

I wonder if enabling only PAC or only BTI would should be enough, instead of removing them completely (-mbranch-protection=none).

I found a year-ago discussion [1] that precisely mentions an scenario where libunwind can be affected by this.

[1] https://groups.google.com/g/linux.debian.maint.dpkg/c/OJHmr8XjuzQ

-mbranch-protection=bti doesn't break the tests at build-time

Building using the -funwind-tables flag without stripping the default mbranch-protection as in 1:2.3.16+dfsg1-3 version doesn't work.:

CFLAGS ......... : -std=gnu99 -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/dovecot-1:2.3.20+dfsg1-1ubuntu3+ppa2 -funwind-tables -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2

[...]

gcc -DHAVE_CONFIG_H -I. -I../.. -I../../src/lib-test -Wdate-time -D_FORTIFY_SOURCE=2 -std=gnu99 -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/dovecot-1:2.3.20+dfsg1-1ubuntu3+ppa2 -funwind-tables -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 -c -o test_lib-test-backtrace.o `test -f 'test-backtrace.c' || echo './'`test-backtrace.c

[...]

test-backtrace.c:14: Assert failed: strstr(str_c(bt), "test_backtrace") != NULL
backtrace_append ..................................................... : FAILED
test-backtrace.c:38: Assert failed: strstr(bt, "test_backtrace") != NULL
backtrace_get ........................................................ : FAILED

I tried to get the point of view of a more expert one, but Final freeze is tomorrow, so moving forward with this:

1. - The fix will consist of lowering the branch-protection to bti for arm64 architectures (better than none).
2. - I will check if there are other packages that uses libunwind that could be affected on arm64 by setting -mbranch-protection to standard by default (i.e.: dotnet6).
3. - If point 2 is true, I will fill a bug against libunwind.

This bug was fixed in the package dovecot - 1:2.3.20+dfsg1-1ubuntu3

---------------
dovecot (1:2.3.20+dfsg1-1ubuntu3) mantic; urgency=medium

  * d/rules: -mbranch-protection=bti for arm64 architectures
   (LP: #2036268).

 -- Miriam España Acebal <email address hidden> Wed, 04 Oct 2023 16:33:20 +0200

I believe libunwind is affected and reported this https://github.com/libunwind/libunwind/issues/647