Xen project Mailing List

[Xen-devel] [RFC v1 0/8] x86/init: Linux linker tables

From: "Luis R. Rodriguez" <mcgrof@xxxxxxxx> A long time ago in a galaxy far, far away... Konrad Rzeszutek Wilk posted patches which eventually got merged to help with modularizing the IOMMUs we have on x86 [0]. This work was done due to the complex relationship that exists on IOMMUs and the requirements on careful execution. The solution also provided a mechanism which jettisoned unused IOMMUs during run-time. During review, even though the code was merged, hpa did note that we tend to encounter this type of problem "often enough that we should implement a generic facility for it" [1], hpa acknowledged that it obviously has to be based on sections and even noted that perhaps we might be able in the future to automate its creation. He noted that the gPXE folks had done just this with linker tables and suggested that "presumably we'd need a few different flavors for init tables and so on, but this would make it a generic mechanism." The IOMMU code got merged and this was left on someone's mental backburner. I've had an itch to scratch recently to try to avoid issues which are possible if one does not jettison other code carefully due to the large complexity of implicit dependencies of certain code on x86 in particular with possible dead code on x86 due to paravirtualization, and the IOMMU jettison strategy turned out to be my favorite solution so far. I've taken on hpa's suggestions from back in the day to review gPXE's solution to see if we could embrace it on Linux for a generic section solution to help jettison code carefully. What this patch set does exactly: This RFC patch set attempts to add support for such a generic solution. In the end, it turns out that the best solution possible was the best of both worlds: a combination of what Konrad had implemented in addition to what Michael Brown had implemented on the gXPE front. The IOMMU solution enables simple semantic annotations for dependency relationships, this however requires a run time sort. The gPXE solution grants the option to simply sort at build time. One of gPXE's solution primary goals however was also to help avoid bit-rot on code that's possible from #ifdef'ery. The Linux linker table solution enables developers to pick and choose what they need, with linker tables being the simplest solution. Contrary to gPXE which strives to force compilation of all linker table solutions we let developers pick *when* they want this as part of their solution. As can be seen from the suggested x86 init specific use of linker tables proposed you can also take advantage of both, linker sorting, optional compilation when needed (at developer's discretion), and even careful semantics annotation for dependency / relationship annotations. Although the x86 init solution here is heavily inspired by the IOMMU solution it diverges with strong semantics, and a new optional subarchitecture annotation. Sorting of init sequences is structure specific, as such each subsystem must defing their own solution unless semantics could be shared. I considered sharing semantics but in the end this proved pointless so this keeps things separate. A series of changes were made to the x86 init sequence in contrast to the IOMMU solution to be *extremely pedantic* on semantics, review of this changes can be studied on the table-init tree [2]. Quick review of gPXE's solution and prospects on further changes: In my review from gPXE's solution it was not clear what hpa meant by gXPE folks having automated this process, they actually use linker tables all around, forcing compilation of *everything* and just do linking of enabled features at link time. You still need to build linker tables on your own. What I do see more potential for in the future is enabling to evolve stronger semantics over time, and this would also be subsystem specific. This will be evident in this patch set on the x86 init use of linker tables. I also see potential in strenghtening semantics for linker sorting, any of these types of features however would impose requiring newer binutils. For instance, gPXE's linker solution currently relies on SORT(), that defaults to SORT_BY_NAME(). This sorts lexicographically, gPXE's solution uses two digits to enable SORT_BY_NAME()'s lexicographical sort to sort orde by numeric priority. Since one is in control of order-level numbers one can provide guarantee that this sort should work as intended, however binutils also now has a SORT_BY_INIT_PRIORITY() which sorts specifically based on digits. SORT_BY_INIT_PRIORITY() was designed specifically for init_array sections though. Refer to the userspace mockup solution table-init git tree [2] commit 6deba47ee1ad461e90 for more details on this. One thing I can envision to help here further are prospects for future linker enhancements, these however must be carefully considered in light of possible requirements of newer binutils and also of compatibility with other toolchains. For now we resort to the good 'ol SORT() which even Linux has made use of for ages already. In that regards here, nothing new here. gPXE folks however did find some fun ICC compatiblity issues, and have also developed fixes for them, these were obviously carried over but should be reviewed carefully. Lastly, I should point out that essentially what we're developing are different forms loose and strong semantics -- in the most complex form what we really are after are feature graphs. Mauro explained to me that for media driver they needed to build a feature graphs, IMHO that could be a next level of strong semantics we might want to consider. For now though the combination of gPXE's linker-table solution based on linker order-level sort, and our old IOMMU init solution and its simple dependency map (and possible extensions, see subarchitecture) should likely suffice as a light weight solution for where we need semantics for at least on the x86 init path. Motivation and possible enhancements: To understand what made me tick to work on this feel free to read the dead code concerns with paravirtualization posts I've made, which also go into and detail Xen's alternative entry point [4] [5]. That's not the end of possibilities to help address to possible "dead code" on Linux, I have other ideas but I have a bit more work to do before publishing anything about it. If anything -- later work should likely supplement this solution further. Currently the earliest I was able to make use of boot_params on x86-64 for linker tables was after load_idt(), so I've decided to stick with the earliest init call for x86 init sequences starting on x86_64_start_reservation(). However, if we're able to make use of boot_params earlier (help appreciated), in particular just boot_params.hdr.hardware_subarch we should be able to make clearer and careful annotations on early x86-64 init code. Using the boot_params.hdr.hardware_subarch and requiring clear subarchitecture support annotations on early init code should provide a *proactive* means to avoid issues such as the cr4 shadow oversight [6] which later caused crashes on Xen. The latest similar type of issue is when KAsan was introduced, after it was integrated I suspected KAsan would probably break if enabled on Xen, I reported this in March 2015 [7] and Andrey confirmed this might be the case but since he wanted it enabled for allyesconfig and allmodconfig he could not think of a clean way to disable it or address support for it then [8]. During the development of this patch set I confirmed KAsan crashes Xen dom0 and therefore should obviously crash Xen PV guests as well. Using the same kernel KAsan still worked on KVM and bare-metal. Provided we had early access to boot_params.hdr.hardware_subarch, in x86_64_start_kernel() we could annotate kasan_early_init() and friends not supported on Xen, however since we don't have a way to disable KAsan at run time at this time we wouldn't have any other option but to crash early. Because of this though we should perhaps just consider disabling KAsan for Xen configs and KAsan folks might just have to bite the bullet. If KAsan folks are gung-ho about not disabling KAsan when Xen is enabled and if its easier to add support to disable KAsan at run time than it is to add KAsan support for Xen one alternative might be to use linker tables to annotate this and disable KAsan at run time for Xen. One of the points of the use of the x86 linker table solution and specific semantics there in is to *force* developers to think about requirements on x86 carefully. So for instance by requiring itemizing supported subarchitectures *early* on in development we should be able to avoid proactively issues such as the crash due to the cr4 shadow changes not added on the Xen entry path, and also missing the requirement to develop a solution for Kasan for Xen. The proposed x86 linker table solution is not the first to use boot_params.hdr.hardware_subarch, its first use was actually on i386, and for lguest. We take advantage of it to avoid further extending pv_ops, and *actually* to see if we could simplify pv_ops over time further. This patch set also includes a renaming of paravirt_enabled() to paravirt_legacy() An unexpected long term side goal which *might* be possible due to linker tables on x86 is to help unify the different C entry points for Linux x86-64. I've taken a stab at it after these patches [9] but it fails on Xen, likely because the stack is not set up right due to the different calls / argument requirements. If this is desirable folks could take a look and perhaps help on that front. Where to get code alternatively and testing: All of these patches are RFCs, if anything the only patches worth consdering merging sooner rather than later might be "paravirt: rename paravirt_enabled to paravirt_legacy" and "x86/boot: add BIT() to boot/bitops.h". After review I can send those separately in patch form if agreeable. Since these are all just RFCs I've based these patches on top of Linus' v4.4-rc5. If you want all patches in one file you can get them here [10], and a respective linux-next version which applies on top of next-20151215 here [11]. I also have two git trees set up with branches for this code, one based on Linus' tree [12] and another based on linux-next next-20151215 [13]. I'll see what issues zero-day bot testing finds out. I've just run time tested this on x86-64 bare metal, KVM, and Xen dom0 so far. [0] https://marc.info/?l=linux-kernel&m=128284562303565&w=2 [1] https://marc.info/?l=linux-kernel&m=128285216913266&w=2 [2] https://github.com/mcgrof/table-init/ [3] https://github.com/mcgrof/table-init/commit/6deba47ee1ad461e90e0fbba226a535cfc1c58f3 [4] http://www.do-not-panic.com/2015/12/avoiding-dead-code-pvops-not-silver-bullet.html [5] http://www.do-not-panic.com/2015/12/xen-and-x86-linux-zero-page.html [6] http://lists.xenproject.org/archives/html/xen-devel/2015-02/msg02742.html [7] http://lkml.kernel.org/r/CAB=NE6Xs5fepzNtymzT4CueeJZ0KMPETpda114DpL4eMtDswtw@xxxxxxxxxxxxxx [8] http://lkml.kernel.org/r/54F5B3DA.70203@xxxxxxxxxxx [9] http://drvbp1.linux-foundation.org/~mcgrof/patches/2015/12/15/x86-merge-x86-init-v1.patch [10] http://drvbp1.linux-foundation.org/~mcgrof/patches/2015/12/15/pend-20151215-rfc-v1-linker-tables.patch [11] http://drvbp1.linux-foundation.org/~mcgrof/patches/2015/12/15/pend-next-20151215-rfc-v1-linker-tables.patch [12] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux.git/log/?h=20151215-rfc-v1-linker-tables [13] https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20151215-rfc-v1-linker-tables Luis R. Rodriguez (8): paravirt: rename paravirt_enabled to paravirt_legacy tables.h: add linker table support x86/boot: add BIT() to boot/bitops.h x86/init: add linker table support x86/init: move ebda reservations into linker table x86/init: use linker table for i386 early setup x86/init: user linker table for ce4100 early setup x86/init: use linker table for mid early setup Documentation/kbuild/makefiles.txt | 19 ++ arch/x86/Kconfig.debug | 47 ++++ arch/x86/boot/bitops.h | 2 + arch/x86/boot/boot.h | 2 +- arch/x86/include/asm/bios_ebda.h | 2 - arch/x86/include/asm/paravirt.h | 4 +- arch/x86/include/asm/paravirt_types.h | 11 +- arch/x86/include/asm/processor.h | 2 +- arch/x86/include/asm/setup.h | 12 - arch/x86/include/asm/x86_init.h | 1 + arch/x86/include/asm/x86_init_fn.h | 267 ++++++++++++++++++++ arch/x86/kernel/Makefile | 4 +- arch/x86/kernel/apm_32.c | 2 +- arch/x86/kernel/asm-offsets.c | 2 +- arch/x86/kernel/cpu/intel.c | 2 +- arch/x86/kernel/cpu/microcode/core.c | 2 +- arch/x86/kernel/dbg-tables/Makefile | 18 ++ arch/x86/kernel/dbg-tables/alpha.c | 10 + arch/x86/kernel/dbg-tables/beta.c | 18 ++ arch/x86/kernel/dbg-tables/delta.c | 10 + arch/x86/kernel/dbg-tables/gamma.c | 18 ++ arch/x86/kernel/dbg-tables/gamma.h | 3 + arch/x86/kernel/{head.c => ebda.c} | 6 +- arch/x86/kernel/head32.c | 22 +- arch/x86/kernel/head64.c | 4 +- arch/x86/kernel/init.c | 55 +++++ arch/x86/kernel/kvm.c | 9 +- arch/x86/kernel/paravirt.c | 2 +- arch/x86/kernel/sort-init.c | 114 +++++++++ arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 25 ++ arch/x86/lguest/boot.c | 2 +- arch/x86/mm/dump_pagetables.c | 2 +- arch/x86/platform/ce4100/ce4100.c | 4 +- arch/x86/platform/intel-mid/intel-mid.c | 4 +- arch/x86/tools/relocs.c | 1 + arch/x86/xen/enlighten.c | 2 +- drivers/pnp/pnpbios/core.c | 2 +- include/linux/tables.h | 421 ++++++++++++++++++++++++++++++++ scripts/Makefile.build | 4 +- scripts/Makefile.clean | 1 + scripts/Makefile.lib | 12 + 42 files changed, 1091 insertions(+), 61 deletions(-) create mode 100644 arch/x86/include/asm/x86_init_fn.h create mode 100644 arch/x86/kernel/dbg-tables/Makefile create mode 100644 arch/x86/kernel/dbg-tables/alpha.c create mode 100644 arch/x86/kernel/dbg-tables/beta.c create mode 100644 arch/x86/kernel/dbg-tables/delta.c create mode 100644 arch/x86/kernel/dbg-tables/gamma.c create mode 100644 arch/x86/kernel/dbg-tables/gamma.h rename arch/x86/kernel/{head.c => ebda.c} (94%) create mode 100644 arch/x86/kernel/init.c create mode 100644 arch/x86/kernel/sort-init.c create mode 100644 include/linux/tables.h -- 2.6.2 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.