Jaunty i386 popen() misbehaves on x86_64 kernel 2.6.26

I would be the second person who tested this on a second Jaunty install and a second Lenny install

I reproduced this with a vanilla v2.6.26 kernel (on x86_64 with defconfig). I used git bisect to find the first commit on which the problem is exhibited:

commit 8ab32bb89b5b9bf06147c31947eba65f0f21c3c0
Author: Roland McGrath <email address hidden>
Date: Sun Mar 16 21:57:41 2008 -0700

    x86: ia32 ptrace vs -ENOSYS

    When we're stopped at syscall entry tracing, ptrace can change the %eax
    value from -ENOSYS to something else. If no system call is actually made
    because the syscall number (now in orig_eax) is bad, then the %eax value
    set by ptrace should be returned to the user. But, instead it gets reset
    to -ENOSYS again. This is a regression from the native 32-bit kernel.

    This change fixes it by leaving the return value alone after entry tracing.

    Signed-off-by: Roland McGrath <email address hidden>
    Signed-off-by: Ingo Molnar <email address hidden>

By defconfig, I meant defconfig plus CONFIG_IA32_EMULATION=y (without which it is not possible to run the i386 binary); that is the only option I changed.

And the first commit after v2.6.26 on which the problem is NOT exhibited is:

commit ed8cae8ba01348bfd83333f4648dd807b04d7f08
Author: Ulrich Drepper <email address hidden>
Date: Wed Jul 23 21:29:30 2008 -0700

    flag parameters: pipe

    This patch introduces the new syscall pipe2 which is like pipe but it also
    takes an additional parameter which takes a flag value. This patch implements
    the handling of O_CLOEXEC for the flag. I did not add support for the new
    syscall for the architectures which have a special sys_pipe implementation. I
    think the maintainers of those archs have the chance to go with the unified
    implementation but that's up to them.

    The implementation introduces do_pipe_flags. I did that instead of changing
    all callers of do_pipe because some of the callers are written in assembler.
    I would probably screw up changing the assembly code. To avoid breaking code
    do_pipe is now a small wrapper around do_pipe_flags. Once all callers are
    changed over to do_pipe_flags the old do_pipe function can be removed.

It seems that __have_pipe2 is being incorrectly set to 1 when running under kernels inside this range (8ab32bb..ed8cae8). Before 8ab32bb, __have_pipe2 is set to -1 and the program works correctly; starting with ed8cae8, pipe2() is actually implemented so the program works correctly.

Fascinating. The problem is that pipe2() is not actually returning -1 and setting errno = ENOSYS like it should. Instead, it is returning 331, which is __NR_pipe2, and leaving errno == 0.

This affects every x86_64 kernel from v2.6.26 (8ab32bb) through v2.6.29-rc3, and was fixed by

commit c09249f8d1b84344eca882547afdbffee8c09d14
Author: Roland McGrath <email address hidden>
Date: Fri Feb 6 18:15:18 2009 -0800

    x86-64: fix int $0x80 -ENOSYS return

    One of my past fixes to this code introduced a different new bug.
    When using 32-bit "int $0x80" entry for a bogus syscall number,
    the return value is not correctly set to -ENOSYS. This only happens
    when neither syscall-audit nor syscall tracing is enabled (i.e., never
    seen if auditd ever started).

    The fix makes the int $0x80 path match the sysenter and syscall paths.

    Reported-by: Dmitry V. Levin <email address hidden>
    Signed-off-by: Roland McGrath <email address hidden>

To fix this on the kernel side, please add commit c09249f8d1b84344eca882547afdbffee8c09d14 to the Intrepid and Jaunty kernels.

It may be possible for glibc to work around this problem on unpatched kernels, though it would not be very pretty.

> Correct output:
> $ uname -r
> 2.6.28-8-generic
> $ ./popen-test
> popen returned 0x1ec6400
> ferror returned 0
> fread returned 19
> ferror returned 0
> pclose returned 0
> Incorrect output:
> $ uname -r
> 2.6.26-1-amd64
> $ ./popen-test
> popen returned 0x935f688
> ferror returned 0
> SHOULD NOT DISPLAY

I am confused. Your own output above shows that this works fine on the
Ubuntu Jaunty kernel. We don't need to patch Jaunty kernels to get the
right result?

It is possible this is triggered by the fact that we do indeed default to running with auditing. That would make the problem moot for a default Intrepid and Jaunty install. I would need a complelling use case to support and SRU of this change given the default install is unaffected.

Kernels starting with 2.6.27 do not have that particular problem with popen() because they implement the pipe2() system call, so popen() does not tickle the missing syscall detection bug. But the attached source can be used to reproduce the missing syscall detection bug directly, by displaying the return value of syscall(6666666).

There are likely to be more and more problems arising from this bug as new syscalls get implemented.

To reproduce, statically compile this on an Intrepid or Jaunty i386 system:
  gcc -static bad-syscall-test.c -o bad-syscall-test
and then run the resulting binary on an Intrepid or Jaunty amd64 system.

Correct output:
  $ uname -rm
  2.6.24-etchnhalf.1-amd64 x86_64
  $ ./bad-syscall-test
  syscall(6666666) returned -1
  errno is 38 (Function not implemented)
Incorrect output:
  $ uname -rm
  2.6.28-8-generic x86_64
  $ ./bad-syscall-test
  syscall(6666666) returned 6666666
  errno is 0 (Success)

It seems that the only sensible fix is to fix the kernel. If this will affect new system calls then this does seem like something worth fixing correctly.

@Anders -- I have pulled down that kernel fix and applied it to Jaunty and Intrepid, could you test the kernels at the URL below and confirm you are happy that they fix the issue to your satisfaction. Please report any testing back here. Kernels are at the URLs below:

    http://people.ubuntu.com/~apw/lp339743-intrepid/
    http://people.ubuntu.com/~apw/lp339743-jaunty/

Thanks. I tested the Jaunty amd64 kernel using the same bad-syscall-test binary; it works correctly and fixes the problem. I will test the Intrepid kernel soon.

Yep, the Intrepid amd64 kernel also fixes the problem.

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-jaunty.git;a=commit;h=73eec19c0301e46b46184cc8c16704ee6b9e26a5

This bug was fixed in the package linux - 2.6.28-10.32

---------------
linux (2.6.28-10.32) jaunty; urgency=low

  [ Amit Kucheria ]

  * Delete prepare-ppa-source script

  [ Andy Isaacson ]

  * SAUCE: FSAM7400: select CHECK_SIGNATURE
  * SAUCE: LIRC_PVR150: depends on VIDEO_IVTV
    - LP: #341477

  [ Ayaz Abdulla ]

  * SAUCE: forcedeth: msi interrupt fix
    - LP: #288281

  [ Brad Figg ]

  * Updating armel configs to remove PREEMPT

  [ Catalin Marinas ]

  * Fix the VFP handling on the Feroceon CPU

  [ Huaxu Wan ]

  * SAUCE: (drop after 2.6.28) [Jaunty] iwlagn: fix iwlagn DMA mapping
    direction

  [ Ike Panhc ]

  * squashfs: correct misspelling
    - LP: #322306

  [ Theodore Ts'o ]

  * SAUCE: (drop after 2.6.28) ext4: add EXT4_IOC_ALLOC_DA_BLKS ioctl
  * SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
    blocks on close
  * SAUCE: (drop after 2.6.28) ext4: Automatically allocate delay allocated
    blocks on rename
    - LP: #317781

  [ Tyler Hicks ]

  * SAUCE: (drop after 2.6.28) eCryptfs: Don't encrypt file key with
    filename key
    - LP: #342128

  [ Upstream Kernel Changes ]

  * ALS: hda - Add support of iMac 24 Aluminium
  * USB: fix broken OTG makefile reference
  * ALSA: hda - add another MacBook Pro 3,1 SSID
  * ALSA: hda - Add model entry for HP dv4
  * x86-64: fix int $0x80 -ENOSYS return
    - LP: #339743

 -- Tim Gardner <email address hidden> Thu, 12 Mar 2009 19:16:07 -0600

Accepted linux into intrepid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I have reproduced the bad return value of int $0x080 calls on the amd64 kernels in intrepid-updates, currently 2.6.27-14.37, and confirmed that the kernels in intrepid-proposed, version 2.6.27-14.36 (needing to be updated), fixes the issue. I've added a slightly modified testcase to the lp:qa-regression-testing bzr tree (I invoke int $0x080 directly via inline asm rather than call syscall(2) so that binaries can be built directly on amd64). Marking verification-done.

This bug was fixed in the package linux - 2.6.27-14.41

---------------
linux (2.6.27-14.41) intrepid-proposed; urgency=low

  [ Stefan Bader ]

  * Revert "SAUCE: input: Blacklist digitizers from joydev.c"
    - LP: #300143

linux (2.6.27-14.40) intrepid-proposed; urgency=low

  [ Amit Kucheria ]

  * Disable DEVKMEM for all archs on Intrepid
    - LP: #354221
  * SAUCE: Quirk for BT USB device on MacbookPro to be reset before use
    - LP: #332443

  [ Andy Isaacson ]

  * LIRC_PVR150: depends on VIDEO_IVTV
    - LP: #341477
  * SAUCE: FSAM7400: select CHECK_SIGNATURE
    - LP: #341712

  [ Andy Whitcroft ]

  * SAUCE: hotkey quirks for various Zepto Znote and Fujitsu Amilo laptops
    - LP: #330259
  * SAUCE: unusual devs: add an entry for the ScanLogic SL11R-IDE 0.78
    - LP: #336189

  [ Anton Veretenenko ]

  * SAUCE: sony-laptop: add support for Sony Vaio FW series function/media
    keys
    - LP: #307592

  [ Ayaz Abdulla ]

  * SAUCE: forcedeth: msi interrupt fix
    - LP: #288281

  [ Chuck Short ]

  * SAUCE: [USB] Unusual Device support for Gold MP3 Player Energy
    - LP: #125250

  [ Ike Panhc ]

  * squashfs: correct misspelling
    - LP: #322306
  * SAUCE: Fixing symbol name in HECI module
    - LP: #336549
  * Copy header files for various kernel media driver
    - LP: #322732

  [ Stefan Bader ]

  * SAUCE: vgacon: Return the upper half of 512 character fonts
    - LP: #355057
  * SAUCE: input: Blacklist digitizers from joydev.c
    - LP: #300143

  [ Upstream Kernel Changes ]

  * libata: make sure port is thawed when skipping resets
    - LP: #269652
  * x86-64: fix int $0x80 -ENOSYS return
    - LP: #339743
  * rt2x00: Fix race conditions in flag handling
    - LP: #258985
  * USB: cdc-acm: Add another conexant modem to the quirks
    - LP: #323829
  * Bluetooth: Add fine grained mem_flags usage to btusb driver
    - LP: #268502
  * Bluetooth: Handle bulk URBs in btusb driver from notify callback
    - LP: #268502
  * Bluetooth: Submit bulk URBs along with interrupt URBs
    - LP: #268502
  * hwmon: (abituguru3) Match partial DMI board name strings
    - LP: #298798
  * x86: mtrr: don't modify RdDram/WrDram bits of fixed MTRRs
    - LP: #292619
  * sis190: add identifier for Atheros AR8021 PHY
    - LP: #247889
  * ath9k: implement IO serialization
    - LP: #373034
  * ath9k: AR9280 PCI devices must serialize IO as well
    - LP: #373034
  * acer-wmi: fix regression in backlight detection
    - LP: #333386

 -- Stefan Bader <email address hidden> Wed, 26 Aug 2009 11:48:11 +0200

I just checked sources of karmic and don't see there any prebuilt openvz enabled kernels. I thought it was this bug that was preventing the openvz-enablde kernel to be built and put into the repository.