Re: [Xen-devel] Is:livepatch-build-tools.git declare it supported? Was:Re: [PATCH for-4.9] livepatch: Declare live patching as a supported feature

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On Tue, Aug 22, 2017 at 7:37 AM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>>>> On 21.08.17 at 17:28, <george.dunlap@xxxxxxxxxx> wrote:
>> So your argument seems to be:
>>
>> 1. We can only provide security support in situations where we can test
>> all possible combinations in the support matrix.
>>
>> 2. We cannot test the entire matrix of combinations for Xen x livepatch
>> tools x compilers
>>
>> 3. Therefore, we cannot provide security support for livepatching tools.
>>
>> Put this way, I hope you can see what the flaw in the argument is: #1 is
>> false.  Xen has {Xen version} x {Linux version} x {Compiler} x
>> {Hardware}.  Hardware of course includes not only the chip itself, but
>> the BIOS / firmware, and the particular devices (and device firmware).
>> If we wanted we could add in {Python version} for people using pygrub,
>> and {Ocaml compiler version} for people running Ocaml, versions of
>> systemd -- I'm sure with effort I could find more dimensions to add to
>> the matrix.
>>
>> We do not, and never have, *tested* the entire matrix of possible
>> combinations considered "security supported" to make sure they work.
>> Such a matrix is completely impossible to even consider, and even if we
>> did some sort of testing, that could not guarantee that they are bug free.
>>
>> What we do for security support is:
>>
>> 1. Test a *representative sample* of combinations (via osstest, product
>> testing, user testing, &c)
>>
>> 2. Promise to issue XSAs if anyone *happens to discover* a combination
>> in the rest of the support matrix that has a security issue
>>
>> That is the requirment for normal Xen, and it would be the same
>> requirement for livepatch-tools: That between osstest, product, and the
>> community, we get regular testing of *a representative sample* of {Xen,
>> livepatch-tools, compiler}, and (what primarily concerns me) issue an
>> XSA if anyone discovers a security issue somewhere in that matrix.
>>
>> I'm not frustrated, but I am baffled by the fact that this "support
>> matrix" objection is so persistent.  Nearly everyone has brought it up,
>> as though "test every combination" was a necessary requirement, in spite
>> of the fact that 1) there is *no* piece of software for which we test
>> the entire matrix of possible combinations 2) I have said over and over
>> again (in fact, I specifically said a few replies ago -- it's there at
>> the top of this email) that we do not test all possible combinations.
>
> Well, part of it may be that the other components involved in the
> test matrix you suggest are external, i.e. we're just their consumers.

But our response to me, you mentioned distros having patched compilers
as a reason that the matrix is untenably large.

> If we consider just our own portions, the matrix is - as said - one
> dimensional. With the livepatching tools, a dimension is being added.
> Even us issuing Linux XSAs is, with the current upstream status of
> the Xen pieces in there, questionable imo. This is also considering
> the fact that iirc we've never issued an XSA for another Xen guest
> OS, yet it is hard to believe that only Linux would ever had any
> vulnerability.

Well, no -- we have at very least {Linux} x {Xen}, for which we end up
testing *many* different configurations, but certainly not all.

I think guest OS support is actually a pretty good analog.  I can't
imagine not issuing XSAs for bugs in Linux, just as I can't imagine
not issuing XSAs for actual security issues that get found in the
livepatch tools.  If you think we shouldn't give security support for
Linux, it makes sense that you would feel the same way for
livepatch-tools (although I don't really understand why you think that
way about either).

We issue more XSAs for Linux than for other guests, in part because of
the complexity of the code inside Linux compared to other OSes; but
also in part due to the fact that that is the most tested and
looked-at.  There probably *are* more bugs in Linux than in NetBSD or
FreeBSD; but also more of them are found because more people are
testing and looking.

But in any case, in my mind, the promise never was "we test all
versions of Linux with all versions of Xen", much less "We test all
versions of all operating systems with all versions of Xen".  The
promise was, "We test a representative sample of Linux and Xen
combinations, and we promise to report issues if they are found."
That's what I thing is the right thing to do for livepatch-tools as
well.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel