Re: Strange symbols_lookup() behaviour in test-symbols on arm64 CI

[Jan Beulich <jbeulich@xxxxxxxx>]

On 11.12.2025 12:44, Mykola Kvach wrote:
> On Thu, Dec 11, 2025 at 1:15 PM Jan Beulich <jbeulich@xxxxxxxx> wrote:
>>
>> On 11.12.2025 11:29, Mykola Kvach wrote:
>>> While working on an arm64 s2ram series for Xen I have hit what looks
>>> like very strange behaviour in symbols_lookup() as exercised by 
>>> test-symbols.
>>>
>>> The series is in the branch referenced at [1]. All patches there except
>>> the last one build and pass CI; adding only the last patch makes the CI
>>> job referenced at [2] start failing.
>>>
>>> Note that the tests in that job are built without CONFIG_SYSTEM_SUSPEND
>>> enabled, so most of the code introduced by the s2ram branch is not
>>> compiled at all for that configuration. That is why I initially did not
>>> expect my series to affect this job.
>>>
>>> To investigate, I tried to reproduce the issue locally. I downloaded the
>>> xen-config artifact from the failing job [3] and used it to build Xen
>>> with my local aarch64 cross compiler. With this local toolchain
>>> I could not reproduce the failure, and the resulting .config changed 
>>> slightly
>>> compared to the job's config. The relevant part of the diff looks like this:
>>>
>>>     diff --git a/xen/.config b/xen-config
>>>     index 057553f510..44dcf6bacc 100644
>>>     --- a/xen/.config
>>>     +++ b/xen-config
>>>     @@ -3,11 +3,11 @@
>>>      # Xen/arm 4.22-unstable Configuration
>>>      #
>>>      CONFIG_CC_IS_GCC=y
>>>     -CONFIG_GCC_VERSION=130300
>>>     +CONFIG_GCC_VERSION=120201
>>>      CONFIG_CLANG_VERSION=0
>>>      CONFIG_LD_IS_GNU=y
>>>      CONFIG_CC_HAS_ASM_INLINE=y
>>>     -CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
>>>     +CONFIG_GCC_ASM_GOTO_OUTPUT_BROKEN=y
>>>      CONFIG_FUNCTION_ALIGNMENT_4B=y
>>>      CONFIG_FUNCTION_ALIGNMENT=4
>>>      CONFIG_ARM_64=y
>>>
>>> So there is at least a difference in GCC version and asm-goto related
>>> Kconfig options between the CI environment and my local one.
>>>
>>> After that I tried rebuilding inside the same Docker image that GitLab
>>> CI uses:
>>>
>>>     registry.gitlab.com/xen-project/xen/alpine:3.18-arm64v8
>>>
>>> When I build Xen in that container, using the same branch, the problem
>>> reproduces in the same way as in the CI job.
>>>
>>> Even more confusingly, adding extra prints in test_symbols just before
>>> the calls to test_lookup() makes the problem disappear. This made me
>>> suspect some undefined behaviour or logic issue that is very sensitive
>>> to optimisation or layout changes.
>>
>> All symptoms described make me suspect you're hitting a problem we're
>> already in the process of hunting down. Can you please take [1], make
>> the small adjustment necessary to Arm's linking rule, and see whether
>> you get a build failure in the case where right now you get a boot time
>> crash? Of course no other changes to code or data layout should be done,
>> or else you may observe false negatives.
> 
> I tested the issue with the provided patch, and it is still reproducible.
> 
> This is my working branch:
> 
> e8d5baab50 (HEAD -> reg) symbols: check table sizes don't change
> between linking passes 2 and 3
> e53439fdfc (xen_gitlab/reg) xen/arm: Add support for system suspend
> triggered by hardware domain
> eaa461f3b5 xen/arm: Implement PSCI SYSTEM_SUSPEND call (host interface)
> 4236fff9a4 xen/arm: Save/restore context on suspend/resume
> a150f3d4bb xen/arm: Resume memory management on Xen resume
> 
> You can find the following line in the attached Xen boot log:
> 
> (XEN) [ 0.010785] Latest ChangeSet: Tue Dec 9 11:11:40 2025 +0100 
> git:e8d5baab50

But this means the build succeeded. Which in turn suggests you're observing
a different issue.

Jan