Re: [Xen-devel] Security vulnerability process, and CVE-2012-0217

[John Creol <iamcreo@xxxxxxxxx>]

Just a quick perspective. I'm part of a smaller downstream "Service provider" 
-- albeit with "Many thousands" of Xen instances running vs. "Hundreds of 
thousands."

About a month before disclosure we were aware of patching/rebooting/notices 
that Linode was performing, based on word our own customers and our ear to the 
ground. From what we could understand we were concerned. We asked around. 
Unfortunately, as a true consumer of Xen we had no real "in" to confirm the 
scope of the issue. 

So we did what we could to prepare - we ensured we were up to date, that our 
rapid patch deployment process worked, and ramped up internal processes to look 
for problems. We've always asked ourselves, what would happen if a hypervisor 
was compromised - a doomsday scenario - and prepared as best we could. We'd 
heard enough to suspect this was what was in the wild and spent quite a few 
sleepless nights thinking about it.

Once the embargo was lifted we had the unique perspective of watching as the 
bug was updated, corrected, and the various CVE's and details rolled out. We 
were on top of it that quickly - we had to be. It was the worst 
case scenario for us - we're intel based and allow extremely custom levels of 
access to Xen so customers can control their environment. 

Not knowing if a vulnerability was in the wild, we took a 
few precautionary steps - for example, disabling 64bit images from new 
deployments, and pushing existing ones to HVM if a customer initiated a reboot. 
Luckily these are switches we can flip in our platform.

On the hypervisor, since we build our own packages we were able to quickly 
build, test, and deploy to our hosts. This was accomplished in most cases 
without customer interruption due to live migration or save/restores, but it 
was still a tense 24-72 hours with hundreds of physical servers to patch in 
multiple worldwide locations.

As we were working through the process, we saw Linode's self congratulatory 
blog post disclosing the vulnerability, and the fact that their customers 
didn't have to worry - they'd been on it for weeks. Wow, we sure wish we could 
be on the pre-disclosure list. We could market that. Unfortunately when it 
comes to the Xen.org disclosure process, size matters.

Do we blame Linode or the Xen.org team? No. Do we think the process was fair? 
No. Would we do anything differently if we were in your position? Maybe not. 
That said, it's clear that personal, business, and other decisions played far 
too great a part in the conversation here. When you have CEO's calling in, and 
jobs on the line, for what amounts to a community project (and 
a vulnerability disclosed by a community member) with hundreds of thousands of 
downstream users that's a real shame -- and a very real advantage to the 
/service/ providers on the list.

The net for us is that we were already doing the right things - our processes 
and Infrastructure are set up in such a way that /when/ the next critical Xen 
vulnerability is released - maybe directly into the wild as an easily runable 
exploit for everyone to get their hands on - we'll be able to respond quickly 
and on multiple fronts.

We can appreciate the process aspects of the discussion, and the scope and 
scale of such large Xen based providers. That said, I would also hope there is 
some renewed thought going into hot patching of the dom0 / hypervisor / etc. 

As someone else mentioned, the actual disclosure could be far better off being 
served by an outside organization like CERT that has established processes in 
place and can move forward without undue influence from a small group of 
players.

Thanks
John

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel