|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] Security vulnerability process, and CVE-2012-0217
Introduction
------------
The Xen.org security response team is charged with implementing the
Xen security response process, the current version of which can be
found here:
http://www.xen.org/projects/security_vulnerability_process.html
Over the past two months we on that team have been involved with
XSA-7 / CVE-2012-0217 and its various fallout.
During this exercise we have encountered some problems with the
process. Also, we have had to make some difficult decisions. We feel
it is essential for keeping us honest that we explain to the community
what we did, and when. There's a detailed timeline at the bottom of
this mail, listing the important events; reading it will give you a
much clearer picture of the problems we encountered and why we are
asking for various issues to be clarified.
In this message I'll go through some of the problems we encountered.
Where there are uncontroversial improvements for the process we
propose them here. For the more controversial issues, in this message
we will raise the questions and discuss the issues in a neutral way,
and make our individual responses separately.
The outcome of this discussion will be a set of changes to be agreed
on and/or voted on using the existing Xen.org governance processes:
http://www.xen.org/projects/governance.html
This discussion will take place on the xen-devel mailing list.
We expect it to take some weeks.
Summary of the most important difficulties we encountered
---------------------------------------------------------
One member of the predisclosure list, only a few days before the
embargo date, mounted a sustained and eventually successful lobbying
campaign to get the embargo extened.
There were leaks during the embargo period, after it had been
extended, which we experienced as enquiries from community members.
We were forced to made several ad-hoc decisions with insufficient
guidance from the process document.
We discovered too late in the process that the set of vulnerable
operating systems was substantially wider than we thought.
The big issues
--------------
1. Purpose of the process
The first point is that we think the security vulnerability process
document should have an explanation of what its goals are. This would
have greatly assisted us when making some of the more difficult
decisions.
In particular, if the process explained its purpose, we would be able
to refer to the spirit of the document when making interpretation
decisions or dealing with issues not clearly resolved by the document.
In this context we need to decide whether:
(a) the predisclosure arrangements are just there so that
organisations can backport patches, develop and test updated
versions, and so forth;
(b) the arrangements are also intended to allow operators to deploy
fixed versions.
Of course this needs to deal clearly with the common stituation of an
organisation running Xen which is a direct consumer of Xen.org source
code.
2. Extension of embargo dates
The most controversial decision was whether the embargo date might be
extended after it had originally been set. We resisted these
suggestions, referring to the process document, which does not
contemplate extending the disclosure period. However, when a
predisclosure list member pressured the discoverer into requesting an
extension, we felt the best interpretation of the process document
required us to acquiesce.
Specifically, of course, we as a team would like clearer guidance from
the process about whether, and if so under what circumstances, a
predisclosure period should be extended.
3. Decisionmaking
It was suggested to us several times, and indeed seemed to be regarded
by some as an underlying principle, that the predisclosure list
members should be making the decisions about disclosure schedule etc.
The question of who is to make the decisions, and on what basis, needs
to be addressed.
4. Length of (default) predisclosure period
Several members of the predisclosure list suggested that the
predisclosure period was too short. Others were ready within the
predisclosure period's timeframe, and were disappointed to see it
extended and to have to delay their fixes.
5. Criteria for predisclosure list membership
We had one request from a public Xen cloud provider to be provided
with predisclosure information. However it appeared to us that they
didn't meet the size threshold in the process document.
The size threshold is of course open to discussion.
We need to clarify whether upstreams and hardware vendors should be on
the predisclosure list. Currently we have one notable upstream vendor
on that list.
Our preliminary view is that the predisclosure list should be used for
downstream notifications. Upstreams, including hardware
manufacturers, should be brought in to the disclosure process as
needed. And as noted, our process for making sure we do that properly
needs to be formalised.
It is probably time for a review of the membership of the
predisclosure list, in any case, after we have incorporated any
changes to the criteria which are agreed as a result of this
discussion.
6. Sharing of information and code between predisclosure participants
We need the process document to be clearer about what kinds of
communications are contemplated _between_ members of the predisclosure
list.
One particular issue here which also relates to the predisclosure
membership criteria, is whether large indirect consumers of Xen should
be on the predisclosure list in their own right. That would allow
them to deploy the fix before the embargo date. It would also allow
them to prepare for testing and deployment, before the fix is
available from their vendor (who would in this scenario also be
entitled to be a predisclosure list member).
In this context, the process needs to clarify whether vendors may
release fixed binaries during the predisclosure period. Certainly
these should not be released other than to predisclosure list members,
since the nature of a bug can often by discovered by reverse
engineering. But can they be released to predisclousre list members ?
More minor policy questions
---------------------------
7. Public communications during the embargo period
We need to clarify what individual vendors are allowed to say during
the embargo period. It was brought to our attention that one public
cloud provider told its users that their VMs would be migrated due to
"a vendor issue" which would be revealed in "a few weeks". Our view
was that that was fine.
As a starting point, it seems to us that it is OK to disclose that
there is an issue, including its XSA and CVE numbers and its planned
disclosure date; but that it is not OK to disclose the impact, scope,
set of vulnerable systems, and certainly not the nature of the
vulnerability.
And we need to clarify what information the security team should give
out when we get an enquiry from a community member about an embargoed
problem.
8. Predisclosure subscription process, and email address criteria
We certainly need to clarify the subscription process to the list, to
make it clear that the organisation's security team should request all
subscriptions and that self-subscriptions via the mailing list
interface will be rejected out of hand. The current arrangements have
caused quite a lot of admin work, to confirm various change requests.
The predisclosure list contains a number of individuals at various
organisations. One vendor seemed to get into difficulty because it
had only one individual on the predisclosure list, which seems to have
unfortunately kept their established team in the dark during the early
stages of the process. The process document requires that downstreams
on the list have established security teams. Should we require that
such response teams' contact details be publicly advertised ? Should
we insist that only response teams' role addresses may be included on
the list ?
9. Vulnerability process scope
We need to clarify the scope of the process document. Currently it
looks like it covers every Xen.org project.
While it would be a nice ideal to support every Xen.org project this
way, in practice the team behind security@xen do not have the
expertise or resources to fix problems in (say) XCP, or the ARM PV
port. Our capability is concentrated on the "Upstream Xen" codebase
(xen-*.hg and its sub-repositories) and the Xen support code in Linux.
Perhaps this could be dealt with by making it clear that problems are
handled on a best effort basis.
10. Exploits
We need a clear policy about releasing proof of concept exploits -
whether, when and who to.
11. Transparency
We think the process document should explicitly state that any
potentially controversial private decisions made by the Xen.org
security response team should be disclosed after the vulnerability
itself is disclosed.
Lacunae in the process
----------------------
12. Holidays
The original planned disclosure date, 2012-05-01, was a public holiday
in many places. We should try to avoid holidays, and Fridays. We
need to discuss which territories' holidays should be checked and make
a list for inclusion in the process.
13. Patch development and review
The patch development process is too closed and not robust enough.
We need to be much clearer both about the need for security patches to
fix things in the smallest, simplest and most reliable way, and not
fix or "improve" matters in unrelated ways. Our internal code review
needs to be much better.
We need to clarify that other issues discovered during the course of
investigating a security disclosure will not be publicly discussed
without first consulting the Xen.org security team, regardless of
their actual relationship to the vulnerability at hand or expected
security impact. Otherwise there is a substantial risk that such
changes will draw attention to embargoed problems.
It may be that it would be a better idea to send out earlier
advisories without patches to the predisclosure list, to enable those
predisclosure list members who would like to do so to help with
development and testing of the patches. If so this needs to be
catered for.
14. Early consideration of which other organisations to bring in
Our process needs to have, very early on, an explicit step to of
deciding which other projects/organisations may also be vulnerable and
may therefore need to be part of the same disclosure process. It also
needs to make sure that we ask for any help (for example from
upstreams or hardware vendors) as soon as possible.
We also need to explicitly determine the availability of various key
players over the disclosure timeline as an early step. One of the
difficulties we encountered was that one of the more important team
members was unexpectedly out of their office at a critical point.
15. Process automation
Many of the advisory versions sent to the predisclosure list contained
clerical errors. Our processes for constructing these need to be
made more automatic.
Thanks for your attention,
Ian.
on behalf of the Xen.org security response team
Timeline (here "us" is the Xen.org security team)
-------------------------------------------------
2012-04-09 Xen.org and FreeBSD notified of the problem (which
will become XSA-7 / CVE-2012-0217) by the discoverer.
2012-04-11 Discoverer tells us that they are happy with following the
Xen.org process for this vulnerability, and asks for
objections from FreeBSD.
2012-04-11 We ask the Debian security team to allocate us a CVE
candidate number. (We have had difficulty with getting
these from Mitre.)
2012-04-12 We confirm in response to a question from Debian
that we think Debian is vulnerable, giving some more
information but not enough to find the problem.
2012-04-12 We contact AMD to ask them to confirm whether any
AMD processors are vulnerable.
2012-04-12 Debian allocates us CVE-2012-0217. We make sure that
everyone currently aware of the issue is told.
2012-04-13 Prompted by the discoverer, we contacts NetBSD
so that they can check if they also are vulnerable,
given that we believe that NetBSD may share the relevant
code with FreeBSD.
2012-04-16 Near-final draft of the predisclosure advisory is ready,
including initial versions of patches for xen-unstable,
Xen 4.1 and Xen 4.0.
2012-04-17 The problem which will later become XSA-8 /
CVE-2012-0218 is spotted and a fix committed to
xen-unstable.hg, without mentioning that this bug is a
security vulnerability. In our view this was a mistake.
2012-04-17 XSA-7 v1 is sent out, with an embargo end date of
2012-05-01.
2012-04-17 We become aware of the earlier commit and assign it
XSA-8. XSA-7 v2 is sent out, with a slightly improved
patch and mentioning that XSA-8 will follow.
2012-04-18 We ask Mitre and Debian for a CVE for XSA-8. We send
XSA-7 v3, and XSA-8 to the predisclosure list.
2012-04-20 We send XSA-8 v2, which contains backported patches and
corrections to the advisory text widening the set of
possibly affected systems.
2012-04-20 We send XSA-7 v4, which contains backported patches and
the information that AMD CPUs are not vulnerable
(following confirmation from AMD).
2012-04-20 A member of the predisclosure list complains to us that
the disclosure schedule is too short, and making various
other points about the process. We reply with a
reference to the process document; we suggest that if the
organisation feels that the process needs improving this
should be discussed in public.
2012-04-20 We are asked by a member of the predisclosure list whether
it would be OK to release fixed binaries to a customer of
theirs who is also a member of the predisclosure list.
On the 2012-04-23, after internal discussions and
consulting with the discoverer, we reply to say that we
think that would be OK.
2012-04-20 We are asked by a member of the predisclousre list
whether there is any way they can verify whether their
systems are vulnerable, or perhaps whether we can share
any proof of concept exploit code. Again, we consulted
with the discoverer.
2012-04-21 The discoverer reproduces the problem on a widely-deployed
proprietary operating system that runs on x86, and
suggests to us that the embargo period should therefore
be extended. We reply:
Our process doesn't contemplate the extension of the embargo
after the release of information to our predisclosure list.
It would in theory be possible for us to email the
predisclosure list ask them to extend the embargo date.
However, from a practical point of view, if anyone didn't get
the email they might go ahead and disclose on the original
date, leaving us all scrambling madly with an unexpected
disclosure. And from a process point of view, that would be
departing substantially from our published process, so I'm very
reluctant.
Certainly you are entitled to notify [proprietary OS vendor]
right away and it would be sensible to do so.
With hindsight it was an error not to consider, as part
of the formal process, which other operating systems
might be vulnerable.
2012-04-26 We obtain the number CVE-2012-0218 from Debian for XSA-8.
2012-04-26 We send XSA-7 v5 with improved wording.
We send XSA-8 v3, with the CVE number.
2012-04-28 A member of the predisclosure list requests an extension
of the disclosure timeframe, and in the same message
requests a change to the email address subscribed to the
predisclosure list. We reply declining, referring to our
process document, and once again inviting public
discussion of the process itself.
2012-04-28 The member of the predisclosure list asking for an
extension asks to be put in touch with all the other
members of the predisclosure list and proposes a
conference call.
2012-04-30 We decline to give further details of the predisclosure
list members, other than the organisational identities
listed. We again decline to extend the disclosure date.
2012-04-30 The member asking for an extension reiterates their
request, citing support from various other named
predisclosure list members - in at least some cases,
exaggerating the level of such support.
2012-04-30 Various other members of the predisclosure list report to
us having been asked to support the member asking for a
delay. CEOs of several companies were involved. In some
cases the other members support the request. We again
decline.
2012-04-30 The member requesting an embargo suggests holding a straw
poll of predisclosure list members on the question of a
delay. Again, we decline.
2012-04-30 The member requesting an embargo places intense pressure
on the employers of some members of the Xen.org security
team to try to get the decision changed, without success.
2012-04-30 23:06 Z (13h before planned disclosure)
The member requesting an embargo contacts the employer
of the discoverer to apply pressure. The discoverer
notifies Xen.org that "on behalf of the organization who
discovered this issue" they request a delay, giving
a putative but unconfirmed new disclosure date of the
2012-05-31.
2012-05-01 08:08 Z (3h52 before planned disclosure)
Xen.org team decides, after internal discussions, that
as the policy gives the discoverer control of the
disclosure date, and the question is not otherwise
addressed, we must agree to this request.
2012-05-10 08:48 Z (3h12 before planned disclosure)
We send an ad-hoc message to the predisclosure list
requesting a disclosure delay for XSA-7 / XSA-8.
We do not include a revised disclosure date as we don't
have one confirmed.
2012-05-01 Backport of XSA-7/XSA-8 patches to 3.4 are prepared.
2012-05-02 We become aware, indirectly, that US-CERT have become
involved.
2012-05-03 US-CERT suggest to us a disclosure date of 2012-06-12.
We make a formal response, saying that we think this is
far too late.
2012-05-03 We receive the first enquiry from a Xen community member
not on the predisclosure list, saying they had heard
somerthing was wrong, and asking about it. This is the
first of several such enquiries.
We quote the number CVE-2012-0217 and say that we expect
it to be disclosed on the 12th of June, but that we
aren't allowed to say more.
2012-05-08 We press the discoverers' employer for a confirmed
disclosure date.
2012-05-08 Given that no disclosure date is forthcoming, we send
XSA-7 v6 and XSA-8 v4, with an indefinite embargo and the
backport patch for Xen 3.4.
2012-05-14 Given that still no firm date is forthcoming, we (the
Xen.org team) point out to the discoverers and to CERT
that our policy requires us to honour reasonable
requests, but that the continued lack of any date is
unreasonable, and demanding that a date be set by the
2012-05-18.
2012-05-14 The discover confirms 2012-06-12 as the disclosure
date.
2012-05-15 We send XSA-7 v7 and XSA-8 v5 with the disclosure date of
2012-06-12.
2012-05-15 We receive a complaint about the delay from a member of
the predisclosure list.
2012-05-15 A member of the predisclosure list discovers what will
become XSA-9. The context was testing the fix for
XSA-7.
2012-05-16 We conclude that this is due to AMD erratum #121, and
involve AMD.
2012-05-16 After discussions we conclude that there is no reasonable
software workaround for Xen and that instead Xen should
refuse to boot on affected systems.
2012-05-18 We decide that releasing this as a separate advisory with
the same embargo date will be less confusing than trying
to fold it into XSA-7.
2012-05-18 While investigating this we come to the conclusion that
our previous patch to fix XSA-7 was incomplete.
2012-05-21 First draft of XSA-9 is available.
2012-05-22 After intense internal discussions we decide that the
original fix for XSA-7 is too fragile and conclude that
it should be replaced by something more obvious.
2012-05-23 We request a CVE for XSA-9 from Mitre and Debian.
2012-05-24 We send XSA-7 v8, now with a cross-reference to XSA-9,
and with updated simpler patch, and XSA-8 v6, and
XSA-9 v1 with its own separate patch, to the
predisclosure list.
2012-05-24 Mitre assign CVE-2012-2934 for XSA-9.
2012-06-11 We send XSA-9 v2, containing the CVE and a clear
statement that no affected processors support SVM.
(The delay to this message was a mistake by the Xen.org
team.)
2012-06-12 Public disclosure.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |