Xen project Mailing List

Re: [Xen-users] Dom0 crashes without logging lately on Debian Stretch with Xen 4.8

From: Andreas Pflug <pgadmin@xxxxxxxxxxxxxxxxx>

Date: Tue, 13 Nov 2018 16:19:29 +0100

Autocrypt: addr=pgadmin@xxxxxxxxxxxxxxxxx; prefer-encrypt=mutual; keydata= xsFNBFmMKQwBEADD1CpyQexCE38SBAF/Dcs5s59mcEyWC5MiM078oAB1f+NA/THkBEX3Jwy2 tskVRP8uJYTcUHLhSIHYbVKGShrPvn6lUQ8WD1B5DuPlNwq3IfGHHLiCfHzvuT6/EfFhNrbT 3doJubXac+thrncW3+HfYl+wnoTKss2IYf3pLPJCVf7o0T60IFrcU8JpHOa9x+5vGEsODAdj W6JX7ki7NuR447Z2J5F1PupR8+0Z+9grDKh4Pc70RMvst+aFyPFBl9WpiR/wyH4FV7n6atAO VuWEuREUJo/8cV2sYrRDuNQCXX0OV+O2twmCwmRosscFBW9xhGXXxMOa6TuzVJtExa1ZjaRk BGjwgV79yfO0RofBYqpdne/7V+bM4/18zvTNPYTFcPgayJgAyMAMneauBeh5INzrKQqFjmaM BsBXSvRlQ/nRwSW0ngijdeWIT+J24mxV0NfL1CdUEdf/E91PXQcrpmtpcce3+E14hFEzgHUK 36miOgl2wFmPiNC7RwKowI+pPdIdB8cVv/tr+gGSz/6G5E1+/RS7Yg1CzKiOhZ2yNFI0sYNd j32dc4sIltcGfwf6J4Pgd+G8be/dileYvVg23T82oo/rLygrnM2IxzOuJDmWIdubhoLKrvqd HAQ7fvgwKYJVdJEFJGeEM+TaWFhtS4zzAG/4uoTjdQs+aGEmuQARAQABzSdBbmRyZWFzIFBm bHVnIDxwZmx1Z0Bwc2UtY29uc3VsdGluZy5kZT7CwXcEEwEKACECGwMFCwkIBwMFFQoJCAsF FgIDAQACHgECF4AFAlmMiDUACgkQy0pg+ikb8EnqqA/+IHuKvm95U32linNotYwUvwkbX2NQ 2lcIOJyFnJHFOxcggw0XFVk2h/7PjyDNUs2GA1Bby8+p3lhlzsD+mKkhlOOYZYoxEXYFKMb+ gEtgNLhhp1HoqN8usvJDELtTNYAljmRxjRcISVCu/iSyJZEiZ0rmnN20TjUviYSw02wPNpFb CXuEVYNhFZ6zpelD7/Fk9MJwGSdnZ6XIt0QQIZ3lB9Gh83jGAkKokxcZdLyLkX8HZMr0hbHW m0hokj4tH6a7v0bGlnAS2/N8OK/tKWx9KyAHrJkP721fff6Lgnq/0+bODmO9mQEYd88UQcze cDp9SiPS3cr7uyPXedN16bhCkWEDV/+yoVnXJ0GjPFgdJ2GH4eplQk+p/9jrxnvOkZq7kmWy PbdNOJk9SX0d9WBKAMaoprdlkskorUBbWsUS5ULStauhtCcVrOH1A2HIQW35AT9+S7FQPhHM sczfBPxZBbVPovC/+ZFf/xnPXt8BMNh89afTygCfRh14pzV3ytaTTEiZT7zTceUQBSUlmnce gBYAhfpartBwlbJDgUtMM2wQX1iOtBYlAswSriFneuwnjG7yNxqz6m6MMaLNz6sKB2JoLVt/ 2t2NsFpl25hI/+NHmmnLUTeHwmhuA7xvzaFI94pBp9Agm2FigxW8HD5RgfhhJR+W6b2GiGau +pjo/jLOwU0EWYwpDAEQANAH0j9o68+5DylPSVdqdVLRGAMbj2mLfmH/CICBf0QrmgWqPBic MqPHxZd3BN/UvPF4meSZIEB5b6P3oSgogdRRWLC3urZpcWmHqhe8NlhQGFh10PMD65sWuR3D Zv1jZaDaNEHWxl4Niz1ydZ1vJVM/2lVceD48hup9GVa8nJkFLGQhcx6/PEx0mspm8ZZg/tLW jqJrLL4YS8RlJrZgowIzh+eJ20jkFWVPQneXldeTvXYZuAyRPG1oPqH+dngh/0UOPLKHfVqH wVTC8NNRCKAyjGZTPZY8WA+lIe4ctJYex8rsiFhO5skgLukXDlPNxM2JifYgYfEHTlEuuF1o uGB+F4XzJd5jTX66BX8I10nhrGx6tbqB/0qo+PElNLDbPU3gWCenUxVQMSOUGVr9989g8enx gG9LrAbS7kEgKmLO23Hcwfjjr+4Cy8msga8JBKHF05FPCs+wsF9JrojRlzlnJ3kEaux+iDqN F03me1dZ3H8Bu7dUFKYI73rnmzsM+mpG7XwDUzIdTlYBNXgr/wyWxK61BGEV9+H6bnf0m7x8 ve0OmeY7bAS/zVi7CeAdUqy6LAN60y3BaTlp1+ztvPHeO2UZqfJxOEEG3mr825clm8652mbJ s//wAJOvU/1kr+eXll8XTYVOXLUK2nU7wBy6rLJT+FAamRn0OgOCrw+vABEBAAHCwWUEGAEK AA8FAlmMKQwCGwwFCQeGH4AACgkQy0pg+ikb8Emfcg//a1IsmCqN+IX+kCkR28D+mJ1xNBSA yPJ7POWrgeQcsh5oWIUJdfDnTzXlujVcySdHT7+nJoofbEaRoc0X0u7fDdm5AJYi/WXcGd7y QbeSuM2vVP2Aiqj+2X/YfsFWzvzG538y47BwrAyxdovD19UZA1nZa5hjUVJdienFsOYqvXq+ 8es1ixWRuMXetiG09355w/Q8+0IIsl4jGLv+B6MMfkDtjjlyhlZSudHsJXBNTRl5dryIPzeC tvg4KDymyD4Kdl/dCYa0kvjvMryMBpbbZ6goW0EtlGgTMnLg9i5mEXAiplY3W67SujHRMd+T 3awPfBRzwMLYzIPL4BNkoe7+mYBJleRbqL/9RXxZ5sF1iBTgU52wUePEyt/btNCYGtkOAYl3 BST7PBkanGpTuM1NYPPNEVmkzCyvsrwc5Mi70Qz2v4zAzLoAWTSa59zX4tzF02uiSWka3NZx grgR1+KmkPPqYdJ5FZUXcH6D9i+VpAPdtak453z5c4TzfaIIHml6jeFjHiyDuX/h+cRtnphr GubIFr8gz4rRw5lmqHJt6I1GqjMsvQ0BH4X4B1iR7GDPZQG5B0P9Ecozi2CYKfrg/vk4l21X EoYjABjZiIqCNYZcikJbio9sZDyz1nRBbZTnBSixFQuJpbRMECrQjiz8qrziiT9zOUtRwkxL WQJuivo=

Delivery-date: Tue, 13 Nov 2018 15:20:45 +0000

List-id: Xen user discussion <xen-users.lists.xenproject.org>

Openpgp: preference=signencrypt

Am 29.10.18 um 12:57 schrieb Roalt Zijlstra | webpower:

Hi there,

Ever since all the Meltdown and Spectre kernel updates and possibly also Xen 4.8 updates, we experience crashes of the Dom0 just out of the blue. Sometimes after 1 day, sometimes after a few days or even 14 days, completely random.

We have two Dell P730 servers and two Dell P720 servers with this behaviour. One thing is that we updated these machine to the latest available firmware, because that is the most secure way. Then we installed Debian Stretch with Xen 4.8 support

We have done serveral installs and 4 servers seem to crash pretty fast and other don't. In the end we think that we can lead it back to the xen-4.8.4-pre version being stable and the xen-4.8.5-pre being unstable. This was kinda independent of the kernel that we were using 4.14 or 4.9.0-8-amd64. This is off course all Debian package numbering.

As last resort we updated on one server all DomU kernels of our Jessie servers on this Dom0 to 4.9.0 from backports instead of the 3.16 kernel. For now that seems to work, but the crashes are random so it could happen any time again. The idea is that these kernels are completely spectre& meltdown unaware and might cause trouble in Xen kernel support. I am not sure if this is true at all, but we are pretty lost what the actual cause is.

We also tested with CentOS and we also had these crashes there with certain combinations of kernel/Xen. The most recent updates seem to be more stable tough. The most frustrating part is the there is absolutely no logs to be found. No kernel oops or what.. the server just resets and boots again.

Are there others experiencing problems like this? Do you see more frequent server/kernel crashes on production servers?

Have you tried netconsole logging to a different server? that might catch that interesting single line of kernel logging that doesn't make it to disk before reboot.

Regards,

Andreas

_______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxxx https://lists.xenproject.org/mailman/listinfo/xen-users