Azure: swiotlb patch needed for CVM

Bug #1971701 reported by Tim Gardner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-azure (Ubuntu)
Invalid
Undecided
Unassigned
Jammy
Fix Released
Medium
Tim Gardner

Bug Description

SRU Justification

[Impact]

[Azure][CVM] Include the swiotlb patch to increase the disk/network performance

Description
As we discussed, there will be new CVM-supporting linux-azure kernels that're based on v5.13 and v5.15. Here I'm requesting the below patch to be included into the two kernels because it can significantly improve the disk/network performance:

swiotlb: Split up single swiotlb lock: https://github.com/intel/tdx/commit/4529b5784c141782c72ec9bd9a92df2b68cb7d45

We have tested the patch with the upstream 5.16-rc8.
BTW, the patch is unlikely to be in the mainline kernel, as the community is trying to resolve the lock contention issue in the swiotlb code using a different per-device per-queue implementation, which would need quite some time to be finalized -- before that happens, we need this out-of-tree patch to achieve good disk/network performance for CVM GA on Azure.

(BTW, the v5.4-based linux-azure-cvm kernel does not need the patch, because it uses a private bounce buffer implementation: drivers/hv/hv_bounce.c, which doesn’t have the I/O performance issue caused by lock contention in the mainline kernel’s swiotlb code.)

[Test Case]

[Microsoft tested]

I tried the April-27 amd64 test kernel and it worked great for me:
1. The test kernels booted up successfully with 256 virtual CPUs + 100 GB memory.
2. The kernel worked when I changed the MTU of the NetVSC NIC.
3. The Hyper-V HeartBeat/TimeSync/ShutDown VMBsus devices also worked as expected.
4. I did some quick disk I/O and network stress tests and found no issue.

When I did the above tests, I changed the low MMIO size to 3GB (which is the setting for a VM on Azure today) by "set-vm decui-u2004-cvm -LowMemoryMappedIoSpace 3GB".

Our test team will do more testing, including performance test. We expect the performance of this v5.15 test kernel should be on par with the v5.4 linux-azure-cvm kernel.

[Where things could go wrong]

Networking could fail or continue to suffer from poor performance.

[Other Info]

SF: #00332721

CVE References

Tim Gardner (timg-tpi)
affects: linux (Ubuntu) → linux-azure (Ubuntu)
description: updated
Tim Gardner (timg-tpi)
Changed in linux-azure (Ubuntu Jammy):
assignee: nobody → Tim Gardner (timg-tpi)
importance: Undecided → Medium
status: New → In Progress
Stefan Bader (smb)
Changed in linux-azure (Ubuntu):
status: New → Invalid
Tim Gardner (timg-tpi)
Changed in linux-azure (Ubuntu Jammy):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.15.0-1006.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-jammy
Revision history for this message
Tim Gardner (timg-tpi) wrote :

Microsoft tested. Marking verification done.

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (105.5 KiB)

This bug was fixed in the package linux-azure - 5.15.0-1008.9

---------------
linux-azure (5.15.0-1008.9) jammy; urgency=medium

  * jammy/linux-azure: 5.15.0-1008.9 -proposed tracker (LP: #1974294)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.04.18)

  * [Azure] WARNING: CPU: 0 PID: 499 at include/linux/dma-mapping.h:555
    netvsc_probe+0x3c9/0x3e0 (LP: #1975717)
    - Drivers: hv: vmbus: Rework use of DMA_BIT_MASK(64)
    - Drivers: hv: vmbus: Fix initialization of device object in
      vmbus_device_register()

  * config CONFIG_HISI_PMU for kunpeng920 (LP: #1956086)
    - [Config] azure: CONFIG_HISI_PMU=m

  * linux: CONFIG_SERIAL_8250_MID=y (LP: #1967338)
    - [Config] azure: CONFIG_SERIAL_8250_MID=y

  * Support AMD P-State cpufreq control mechanism (LP: #1956509) // Enable
    speakup kernel modules to allow the speakup screen reader to function
    (LP: #1967702)
    - [Config] azure: Update configs after rebase

  * Azure: swiotlb patch needed for CVM (LP: #1971701) // [Azure][CVM] Fix
    swiotlb_max_mapping_size() for potential bounce buffer allocation failure in
    storvsc (LP: #1973169)
    - SAUCE: swiotlb: Max mapping size takes min align mask into account

  * Azure: swiotlb patch needed for CVM (LP: #1971701)
    - SAUCE: treewide: Replace the use of mem_encrypt_active() with
      cc_platform_has()
    - SAUCE: swiotlb: use bitmap to track free slots
    - SAUCE: swiotlb: allocate memory in a cache-friendly way
    - SAUCE: swiotlb: Split up single swiotlb lock

  * jammy/linux-azure: Update cifs to 5.15 backport (LP: #1970977)
    - improve error message when mount options conflict with posix
    - cifs: call cifs_reconnect when a connection is marked
    - cifs: call helper functions for marking channels for reconnect
    - cifs: mark sessions for reconnection in helper function
    - treewide: Replace zero-length arrays with flexible-array members
    - smb3: fix incorrect session setup check for multiuser mounts
    - cifs: truncate the inode and mapping when we simulate fcollapse
    - cifs: use a different reconnect helper for non-cifsd threads
    - cifs: do not skip link targets when an I/O fails
    - cifs: convert the path to utf16 in smb2_query_info_compound
    - cifs: change smb2_query_info_compound to use a cached fid, if available
    - cifs: fix bad fids sent over wire
    - cifs: fix incorrect use of list iterator after the loop
    - move more common protocol header definitions to smbfs_common
    - smb3: move defines for ioctl protocol header and SMB2 sizes to smbfs_common
    - smb3: move defines for query info and query fsinfo to smbfs_common
    - smb3: cleanup and clarify status of tree connections
    - smb3: fix ksmbd bigendian bug in oplock break, and move its struct to
      smbfs_common
    - fs: Remove ->readpages address space operation
    - cifs: fix potential race with cifsd thread
    - cifs: remove check of list iterator against head past the loop body
    - cifs: force new session setup and tcon for dfs
    - cifs: update internal module number
    - cifs: Check the IOCB_DIRECT flag, not O_DIRECT
    - cifs: Split the smb3_a...

Changed in linux-azure (Ubuntu Jammy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.