Re: [PATCH 0/8] Force detach PCI devices for ACPI-based and PCIe native hot-unplug

[Dongli Zhang <dongli.zhang@xxxxxxxxxx>]

On Mon, Sep 7, 2026 1:55:42AM -0700, Michael S. Tsirkin wrote:
> On Mon, Sep 07, 2026 at 01:26:57AM -0700, Dongli Zhang wrote:
>> 
>> 
>> On Thu, Sep 3, 2026 8:28:35AM -0700, Igor Mammedov wrote:
>> > On Wed, 26 Aug 2026 09:15:47 -0700
>> > Dongli Zhang <dongli.zhang@xxxxxxxxxx> wrote:
>> > 
>> >> On Mon, Aug 24, 2026 7:44:49AM -0700, Daniel P. Berrangé wrote:
>> >> > On Sun, Aug 23, 2026 at 06:13:30PM -0700, Dongli Zhang wrote:  
>> >> >> Hot-unplugging a PCI device can require cooperation from the guest. For
>> >> >> ACPI PCI hotplug, QEMU notifies the guest through ACPI and the guest
>> >> >> eventually writes the ACPI PCI eject register. For PCIe native hotplug,
>> >> >> QEMU notifies the guest through the PCIe hotplug mechanism and waits 
>> >> >> for
>> >> >> the slot unplug flow to complete. Only after that completion does QEMU
>> >> >> unrealize the device and emit DEVICE_DELETED.
>> >> >> 
>> >> >> This can leave a device stuck in the unplug pending state when the 
>> >> >> guest
>> >> >> does not cooperate. Examples include:
>> >> >> 
>> >> >> 1. The guest has panicked, or the relevant ACPI/PCI hotplug driver is
>> >> >> unavailable.
>> >> >> 
>> >> >> 2. The guest is stalled and cannot handle the hot-unplug event. For
>> >> >> example, stalling the Linux [irq/9-acpi] kernel thread can reproduce 
>> >> >> this
>> >> >> for ACPI-based hot-unplug.
>> >> >> 
>> >> >> 3. The device was attached to a slot that the guest cannot use. For
>> >> >> example, a pcie-root-port only supports slot 0. If a device is added 
>> >> >> to a
>> >> >> non-zero slot below a pcie-root-port, the guest may never discover the
>> >> >> device and therefore may never complete the unplug request.
>> > 
>> > all of above is actually expected, no (functioning) driver => no 
>> > hotplug/unplug.
>> > it's the guest problem. Once device it exposed to guest its life-cycle
>> > not longer owned by QEMU.
>> > 
>> > That's what one would see in real hw as well, you press eject button
>> > but it will not do anything if OS doesn't process it.
>> > also see comment at the end.
>> 
>> Users are generally more tolerant of issues with real hardware.
>> 
>> In virtualization and cloud environments, PCI hotplug is more commonly used 
>> for
>> NICs and storage devices. Users are less tolerant of disruption or unexpected
>> failures.
> 
> 
> Simply put, surprise removal exists in the hardware.  Emulating that
> makes sense, at a high level. Nor is it too hard.
> However, guests, especially Linux, do not
> handle it all that well generally. Exactly because users
> would tend to impatiently reach for that tool, then
> blame QEMU after a crash, we avoided emulating that.
> 
> 
> I'd expect much more in the way of research into how guests behave,
> perhaps some ways to limit it to devices that work well, and
> likely some linux patches to make it work better, before we commit to
> supporting such interfaces.

Here is my summary of the current situation and plan:

1. Folks are not against adding an option to force-detach a PCI device. It
should emulate surprise removal, which exists in real hardware. This requires
adding the emulation to QEMU.

2. However, the more important issue is ensuring that surprise removal works
well in guest operating systems, such as Linux and Windows, and for a defined
subset of devices, including virtio devices, VFIO-assigned devices, Intel,
Mellanox, and Broadcom NICs, and NVMe devices.

>From QEMU’s perspective, it is preferable not to expose an interface that may
cause a guest crash and lead users to blame QEMU for a guest-side limitation.

3. More research and development work is required to support surprise removal in
both QEMU and guest operating systems, especially the Linux kernel.

4. So far, from QEMU's perspective, everyone suggest rebooting the guest VM for
many of the use cases I mentioned

5. By instrumenting QEMU code and possibly adding additional metadata, we can
track whether a PCI device has been accessed by the guest VM, especially during
PCI hotplug. If the device has never been used by the guest VM, it may be safer
to force-detach it.

> 
> 
>> > 
>> >> >> 
>> >> >> The non-zero slot case has also been discussed in:
>> >> >> 
>> 
>> [snip]
>> 
>> >> >   
>> >> 
>> >> Thank you very much!
>> >> 
>> >> I see that the issue has been fixed. The ticket mentions the following.
>> >> 
>> >> "What I am observing is that it seems when the slot ID != 0, the guest OS 
>> >> seems
>> >> to ignore this and we never seem to hit ich9_pm_device_unplug_cb()."
>> >> 
>> >> Based on my experience and evaluation, ACPI-based hotplug is more likely 
>> >> to
>> >> encounter an issue where the guest VM does not respond to an unplug 
>> >> operation.
>> > 
>> > I'm not sure it's a good idea to delete device when guest still thinks 
>> > it's there
>> > (you can make guesses on QEMU side if it's in use, how useful those are is 
>> > questionable).
>> 
>> By instrumenting the QEMU functions related to device hotplug and PCI
>> initialization, we may be able to make informed assumptions.
> 
> So try. But just know that pci initialization is commonly done by firmware, 
> not
> the driver.

Thank you very much for the confirmation. This also provides users with
telemetry to narrow down the potential reasons why QEMU does not send the
DEVICE_DELETED QMP event.

> 
>> > 
>> > as far as I know, ACPI hotplug has no notion of surprise removal (pls 
>> > educate me if it's not the case),
>> > so I wouldn't do what you are proposing here at all, it's basically asking 
>> > for disaster to happen.
>> > And all this is basically for dealing with abused qemu flexibility.
>> > 
>> > Please (re)formulate usecase and make it more clear as what is eludes me
>> > no matter how many times i've read this cover letter.
>> 
>> Here are some use cases in virtualization and cloud environments:
>> 
>> 1. Suppose there is a QEMU user configuration error and a PCI device is 
>> attached
>> to slot 1 of a pcie-root-port that uses ACPI-based hotplug. The device may 
>> not
>> be detected or ejected. As a result, there is no way to detach it from the
>> user's QEMU instance until the guest VM reboots.
> 
> sounds vague. if users can not configure qemu what are the chances
> they will use force detach responsibly?

I should have clarified the difference between the QEMU user and the VM owner.
The QEMU user is the software that manages VMs, such as libvirt or any other
software that communicates with QEMU through QMP. The VM owner does not have
access to QEMU.

However, any mistake made by the QEMU user can affect the VM owner. For example,
if the QEMU user mistakenly adds a PCI device to slot 1 of a pcie-root-port, the
VM owner cannot detach the device from the QEMU instance until the guest 
reboots.

In this case, the QEMU user is at fault, but the VM owner is affected.

A force or surprise removal option helps the QEMU user recover from a
configuration error without requiring action from the VM owner.

> 
>> 2. For an unknown reason in the customer's guest kernel (Linux, Windows, or
>> BSD), a PCI device may still be referenced by the guest kernel or its 
>> services.
>> As a result, the guest never writes the eject register for ACPI-based 
>> hotplug,
>> and QEMU cannot detach the device. The customer may blame QEMU for not 
>> removing
>> it. A force-detach option could provide an escape hatch, with a warning that 
>> it
>> may make the VM unstable or insecure.
> 
> So just reboot the guest.

I agree.

Sometimes, the VM owner blames QEMU for not detaching a PCI device, even though
QEMU is technically waiting for the guest VM to write to the EJ register :(

> 
>> 3. Suppose a VM is stuck because of a guest kernel bug, such as a Linux 
>> kernel
>> panic without a kdump kernel being triggered. The guest kernel is 
>> unresponsive.
>> Force detach could allow the block device to be temporarily attached to 
>> another
>> VM without resetting the currently panicked VM.
> 
> So just reboot the guest.

I agree.

Sometimes, the VM owner blames QEMU for not detaching a PCI device, even though
QEMU is technically waiting for the guest VM to write to the EJ register :(

> 
>> 4. Provide a mechanism to demonstrate that force detach is unsafe. Otherwise,
>> users may repeatedly attempt it and mistakenly conclude that QEMU is at 
>> fault :)
> 
> We have that - we don't support unsafe detach.

Thank you very much!

Dongli Zhang