Xen project Mailing List

On Tue, Nov 3, 2015 at 7:48 AM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:

On Mon, 2015-11-02 at 13:02 -0500, Jintack Lim wrote:
>
>
> On Mon, Nov 2, 2015 at 10:33 AM, Ian Campbell <ian.campbell@xxxxxxxxxx>
> wrote:
> > On Sun, 2015-10-25 at 18:51 -0400, Jintack Lim wrote:
> > >
> > >
> > > On Fri, Oct 23, 2015 at 9:44 AM, Ian Campbell <ian.campbell@xxxxxxxxx
> > m>
> > > wrote:
> > > > On Fri, 2015-10-23 at 09:22 -0400, Jintack Lim wrote:
> > > > > Hi,
> > > > >
> > > > > I'm working on ARM64 (AppliedMicroâ X-Gene).
> > > > > I wanted to run perf to collect some hardware events, but I got
> > some
> > > > > trouble.
> > > > > Basically I have two questions.
> > > > >
> > > > > 1.
> > > > > I found that pmu is not registered in the device tree,
> > > > > while it is registered on native linux (installed on the same
> > type of
> > > > > machine) .
> > > > > Can I just modify device tree at booting time to make it look
> > like
> > > > native
> > > > > linux?
> > > > > Will it work?
> > > >
> > > > > Additional information is at the bottom.
> > > > >
> > > > > 2.
> > > > > I found that running perf on Xen 4.5.0 resulted in segmentation
> > fault
> > > > (or
> > > > > system crash),
> > > > > however it runs well on the latest Xen on git server.
> > > > > Can somebody point the patch that makes this change?
> > > >
> > > > 4.5.0 was vulnerable to XSA-93:
> > > >
> > > > http://xenbits.xen.org/xsa/advisory-93.html
> > > >
> > > > The core issue with PMU is that Xen currently does not know how to
> > context
> > > > switch the relevant state when changing guests. Fixing that would
> > be more
> > > > work than just fiddling with device tree.
> > > >
> > > > You might be able to bodge something useful (but unsupported)
> > together by
> > > > just allowing only dom0 to access these registers and to see the
> > values
> > > > collected from the whole system (i.e. don't worry about context
> > switching).
> > > > How useful such numbers would be depends somewhat on what you are
> > hoping to
> > > > find out from this exercise.
> > > My machine has 8 cores,
> > > and I'm running dom0 on core 0~3, and domU on 4~7.
> > > I wonder if running perf on dom0 can profile hardware events on
> > cores, on
> > > which dom0 is not running.
> >
> > Please remember: performance counter use from guests is not something
> > currently supported by Xen on ARM, so perf on dom0 cannot, in a
> > formally
> > supported sense, profile anything at all.
> >
> > If you are talking about using the bodge mentioned above then of course
> > it
> > can only profile things running on processors to which it has (pinned,
> > 1:1)
> > access and there may be other more serious caveats I've not thought of
> > because this is not something which is supported.
> Thanks, Ian.
>
> I have two questions.
>
> 1. What's the reason why perf is not supported by Xen on ARM?
> Is it because of what you've mentioned before?
> "The core issue with PMU is that Xen currently does not know how to
> context
> switch the relevant state when changing guests. "

Nobody has done the work to support it.

Thanks!

> 2. I'm trying to make it work in a limited environment (pined, 1:1) on Dom0.
> It's a bit awkward (and I'm sorry) to ask questions about something not supported,
> but even a simple comment will be really helpful!
>
> The problem I have is that Xen is getting FIQ for pmu overflow interrupts.
> It looks like group 0 interrupts are enabled and FIQen bit is not set, by this line inÂxen/arch/arm/gic-v2.c
> writel_gicc(GICC_CTL_ENABLE|GICC_CTL_EOI, GICC_CTLR);
> So, I think FIQ should not come, can it?
>
> Instead of getting panic because of FIQ,
> I tried to make it run as normal IRQ by
> modifying entry.S, and routing FIQ to normal IRQ function by calling do_trap_fiq.
> System didn't go panic, but it was just hanging there.
>
> I'm a bit lost where to start to fix this.
> I think getting IRQ instead of FIQ is the first step.
> Do you have any suggestions?

I'm afraid I do not have any clever thoughts here. I'm sure that some
actual development work would be needed to make FIQ work and not just
trying to hack around it by treating them as IRQs, but I don't know exactly
what this would entail.

I figured this out with Christoffer Dall's help.

On X-gene, IRQBypDisGrp1 and IFQBypDisGrp1 bits are set in non-secure GICC_CTLR copy on native Linux.

This is not to bypass legacy interrupt signal.

Xen clears those bits in gic_v2_cpuinit() function. [1]

After I modified it to preserve bypass bits as native Linux does,

FIQ didn't come any more, and I got pmu overflow interrupts in Dom0!

I have another question though, not related to FIQ.

Xen seems to assume that only SPIs are delivered to domain,

and they are set inÂroute_irq_to_guest() when creating domain.

However pmu interrupt is PPI (28 on X-gene), and it needs to beÂ

enabled for each cpu. (i.e. set enable bit in GICD_ISENABLER0)

So, currently pmu interrupt is only enabled on cpu0.

What would be the best way to make pmu interrupts be routed to all cpus if any?

One way I can think of is to useÂrequest_irq() function like timer interrupts,

and get interrupts in Xen and inject interrupts to domain in turn.

Or simply calling route_irq_to_guest() on each cpu will do?

If so, I'm not sure using vcpu[0] in current Xen code in gic_route_irq_to_guest() may have some side effect.

Thanks,

JintackÂ

[1]Âhttp://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/gic-v2.c;h=01e36b5aaacee8db52b7178ad4b6654dbdf059f9;hb=HEAD#l325

Ian.

Re: [Xen-users] perf hardware event on ARM64