Re: [Xen-users] leap second mess

[G.Bakalarski@xxxxxxxxxx]

> On Sun, 2012-07-01 at 16:32 +0100, Tony Lill wrote:
>> Apparently this is a known bug
>>
>> http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/
>
> AFAIK there isn't anything Xen specific here, is there?

No and yes. At first we thought it was XEN only issue.
Our Xen domUs where mostly unresposive. With xm console domU
command we could login to domU after 20+ min waiting.
Most machines had load in a range 150-300 and full CPUs load
while doing really nothing.

But today noticed that base metal servers were also affected
but not so hard. Eg. bare metal machine which used to use
1-2 CPUs with appropriate load (normal) , used today more than 10 CPUs
with load 10+ but effective performance was some 5 times slower (i.e.
it could managed 5 time less records in a minute).
But it was resposive - this is why we did not notice a problem yesterday.
After trick with date command the behaviour returned to normal

>
> If there is then we'll need more details about exactly what was running
> (dom0 and domU OS, userspace workloads etc) on the machines in question.

At first we were convinced that mostly Xen 4.1 + kernel 3.2.0-1 (debian
wheezy first 3.2 kernel). But today w found older machines with debian squeeze
kernel 2.6.32-5-xen where affected. But as I wrote above some  bare metal
machines with kernel 3.2.0-2 also affected.

Our main applications are java and postresql based with java mostly
generating mess.
But also simple longer scp (overnight copies) were also broken
- probably due to leap second or extremally high load.

Maybe context switching is a reason why Xen machines were so much
attected ???

GB



>
>> On 07/01/2012 11:23 AM, Tony Lill wrote:
>> > Yeah, that was fun. According to my monitoring, context switches
>> > and interrupts increased by a factor of 10 or more when the leap
>> > second was added.
>> >
>> > On 07/01/2012 08:59 AM, G.Bakalarski@xxxxxxxxxx wrote:
>> >> Hi list
>> >
>> >> Maybe everybody already knows this, but
>> >
>> >> today many of our domUs got crazy - unusually
>> >
>> >> high load on machines doing nothing (e.g. load of 200
>> >
>> >> on machine which usually has load 2-4).
>> >
>> >> Simple command:
>> >
>> >> date; date `date +"%m%d%H%M%C%y.%S"`; date
>> >
>> >> magically make peace ...
>> >
>> >
>> >> [shocked]
>> >
>> >
>> >> GB
>> >
>> >
>> >> _______________________________________________ Xen-users mailing
>> >>  list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
>> >
>> >
>> >
>> > _______________________________________________ Xen-users mailing
>> > list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
>> >
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-users
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxx
> http://lists.xen.org/xen-users
>



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users