[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [PATCH 6/9] dom0 PCI: document the change



Create how-to for SR-IOV user and device driver developer.

Signed-off-by: Yu Zhao <yu.zhao@xxxxxxxxx>

diff -r f9bb37cf38ed -r 6b4452adda52 Documentation/DocBook/kernel-api.tmpl
--- a/Documentation/DocBook/kernel-api.tmpl     Sat Sep 27 01:27:42 2008 -0400
+++ b/Documentation/DocBook/kernel-api.tmpl     Sat Sep 27 01:28:05 2008 -0400
@@ -311,6 +311,7 @@
      </sect1>

      <sect1><title>PCI Support Library</title>
+!Iinclude/linux/pci.h
 !Edrivers/pci/pci.c
 !Edrivers/pci/pci-driver.c
 !Edrivers/pci/remove.c
@@ -323,6 +324,7 @@
 -->
 !Edrivers/pci/probe.c
 !Edrivers/pci/rom.c
+!Edrivers/pci/iov.c
      </sect1>
      <sect1><title>PCI Hotplug Support Library</title>
 !Edrivers/pci/hotplug/pci_hotplug_core.c
diff -r f9bb37cf38ed -r 6b4452adda52 Documentation/PCI/pci-iov-howto.txt
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/Documentation/PCI/pci-iov-howto.txt       Sat Sep 27 01:28:05 2008 -0400
@@ -0,0 +1,227 @@
+               PCI Express Single Root I/O Virtualization HOWTO
+                       Copyright (C) 2008 Intel Corporation
+
+
+1. Overview
+
+1.1 What is SR-IOV
+
+Single Root I/O Virtualization (SR-IOV) is a PCI Express Extended
+capability which makes one physical device appear as multiple virtual
+devices. The physical device is referred to as Physical Function while
+the virtual devices are referred to as Virtual Functions. Allocation
+of Virtual Functions can be dynamically controlled by Physical Function
+via registers encapsulated in the capability. By default, this feature
+is not enabled and the Physical Function behaves as traditional PCIe
+device. Once it's turned on, each Virtual Function's PCI configuration
+space can be accessed by its own Bus, Device and Function Number (Routing
+ID). And each Virtual Function also has PCI Memory Space, which is used
+to map its register set. Virtual Function device driver operates on the
+register set so it can be functional and appear as a real existing PCI
+device.
+
+1.2 What is ARI
+
+Alternative Routing-ID Interpretation (ARI) allows a PCI Express Endpoint
+to use its device number field as part of function number. Traditionally,
+an Endpoint can only have 8 functions, and the device number of all
+Endpoints is zero. With ARI enabled, an Endpoint can have up to 256
+functions by using device number in conjunction with function number to
+indicate a function in the device. This is almost transparent to the Linux
+kernel because the Linux kernel still can use 8-bit bus number field plus
+8-bit devfn number field to locate a function. ARI is managed via the ARI
+Forwarding bit in the Device Capabilities 2 register of the PCI Express
+Capability on the Root Port or the Downstream Port and a new ARI Capability
+on the Endpoint.
+
+
+2. User Guide
+
+2.1 How can I manage SR-IOV
+
+If a device supports SR-IOV, then there should be some entries under
+Physical Function's PCI device directory. These entries are in directory:
+       - /sys/bus/pci/devices/BB:DD.F/iov/
+               (BB:DD:F is bus:dev:fun)
+and
+       - /sys/bus/pci/devices/BB:DD.F/iov/N
+               (N is VF number from 0 to initialvfs-1)
+
+To enable or disable SR-IOV:
+       - /sys/bus/pci/devices/BB:DD.F/iov/enable
+               (writing 1/0 means enable/disable VFs, state change will
+                notify PF driver)
+
+To change number of Virtual Functions:
+       - /sys/bus/pci/devices/BB:DD.F/iov/numvfs
+               (writing positive integer to this file will change NumVFs)
+
+The total and initial number of VFs can get from:
+       - /sys/bus/pci/devices/BB:DD.F/iov/totalvfs
+       - /sys/bus/pci/devices/BB:DD.F/iov/initialvfs
+
+The identifier of a VF that belongs to this PF can get from:
+       - /sys/bus/pci/devices/BB:DD.F/iov/N/rid
+               (for all class of devices)
+
+For network device, there are:
+       - /sys/bus/pci/devices/BB:DD.F/iov/N/mac
+       - /sys/bus/pci/devices/BB:DD.F/iov/N/vlan
+               (value update will notify PF driver)
+
+2.2 How can I use Virtual Functions
+
+Virtual Functions are treated as hot-plugged PCI devices in the kernel,
+so they should be able to work in the same way as real PCI devices.
+NOTE: Virtual Function device driver must be loaded to make it work.
+
+
+3. Developer Guide
+
+3.1 SR-IOV APIs
+
+To register SR-IOV service, Physical Function device driver needs to call:
+       int pci_iov_register(struct pci_dev *dev,
+               int (*notify)(struct pci_dev *, u32), char **entries)
+
+Note: entries could be NULL if PF driver doesn't want to create new entries
+under /sys/bus/pci/devices/BB:DD.F/iov/N/.
+
+To unregister SR-IOV service, Physical Function device driver needs to call:
+       void pci_iov_unregister(struct pci_dev *dev)
+
+To enable SR-IOV, Physical Function device driver needs to call:
+       int pci_iov_enable(struct pci_dev *dev, int numvfs)
+
+To disable SR-IOV, Physical Function device driver needs to call:
+       void pci_iov_disable(struct pci_dev *dev)
+
+Note: above two functions sleeps 1 second waiting on hardware transaction
+completion according to SR-IOV specification.
+
+To read or write VFs configuration:
+       - int pci_iov_read_config(struct pci_dev *dev, int id,
+                       char *entry, char *buf, int size);
+       - int pci_iov_write_config(struct pci_dev *dev, int id,
+                       char *entry, char *buf);
+3.2 Usage example
+
+Following piece of code illustrates the usage of APIs above.
+
+static char *entries[] = { "foo", "bar", NULL };
+
+static int callback(struct pci_dev *dev, u32 event)
+{
+       int err;
+       int vfn;
+       int numvfs;
+
+       if (event & PCI_IOV_ENABLE) {
+               /*
+                * request to enable SR-IOV, NumVFs is available.
+                * Note: if the PF want to support PM, it has to
+                * check the device power state here to see if
+                * the request is allowed or not.
+                */
+
+               numvfs = event & PCI_IOV_NUM_VIRTFN;
+
+       } else if (event & PCI_IOV_DISABLE) {
+               /*
+                * request to disable SR-IOV.
+                */
+               ...
+
+       } else if (event & PCI_IOV_RD_CONF) {
+               /*
+                * request to read VF configuration, Virtual
+                * Function Number is available.
+                */
+
+               vfn = event & PCI_IOV_VIRTFN_ID;
+
+               /* pass the config to SR-IOV code so user can read it */
+               err = pci_iov_write_config(dev, vfn, entry, buf);
+
+       } else if (event & PCI_IOV_WR_CONF) {
+               /*
+                * request to write VF configuration, Virtual
+                * Function Number is available.
+                */
+
+               vfn = event & PCI_IOV_VIRTFN_ID;
+
+               /* read the config that has been written by user */
+               err = pci_iov_read_config(dev, vfn, entry, buf, size);
+
+       } else
+               return -EINVAL;
+
+       return err;
+}
+
+static int __devinit dev_probe(struct pci_dev *dev,
+                               const struct pci_device_id *id)
+{
+       int err;
+
+       err = pci_iov_register(dev, callback, entries);
+       ...
+
+       err = pci_iov_enable(dev, nr_virtfn, callback);
+
+       ...
+
+       return err;
+}
+
+static void __devexit dev_remove(struct pci_dev *dev)
+{
+       ...
+
+       pci_iov_disable(dev);
+
+       ...
+
+       pci_iov_unregister(dev);
+
+       ...
+}
+
+#ifdef CONFIG_PM
+/*
+ * If Physical Function supports the power management, then the
+ * SR-IOV needs to be disabled before the adapter goes to sleep,
+ * because Virtual Functions will not work when the adapter is in
+ * the power-saving mode.
+ * The SR-IOV can be enabled again after the adapter wakes up.
+ */
+static int dev_suspend(struct pci_dev *dev, pm_message_t state)
+{
+       ...
+
+       pci_iov_disable(dev);
+
+       ...
+}
+
+static int dev_resume(struct pci_dev *dev)
+{
+       ...
+
+       pci_iov_enable(dev, numvfs);
+
+       ...
+}
+#endif
+
+static struct pci_driver dev_driver = {
+       .name =         "SR-IOV Physical Function driver",
+       .id_table =     dev_id_table,
+       .probe =        dev_probe,
+       .remove =       __devexit_p(dev_remove),
+#ifdef CONFIG_PM
+       .suspend =      dev_suspend,
+       .resume =       dev_resume,
+#endif
+};

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.