[Xen-devel] xenbits and https redirect (was Re: [xen-unstable-smoke test] 99610: regressions - FAIL)

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

osstest service owner writes ("[xen-unstable-smoke test] 99610: regressions - 
FAIL"):
> flight 99610 xen-unstable-smoke real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/99610/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  build-amd64                   5 xen-build    fail REGR. vs. 97725

fatal: unable to access
'https://xenbits.xen.org/git-http/mini-os.git/': Failed to connect to
xenbits.xen.org port 443: Connection refused

This is because:

1. xen.git specifies the url http://xenbits.xen.org/git-http/mini-os.git/

2. xenbits has/had a redirect to https, as part of our transition to
   using https everywhere.

3. So the xen.git build tries to access
   https://xenbits.xen.org/git-http/mini-os.git/

4. The http_proxy variable set by osstest to try to make http requests
   by build systems go via the cacheing proxy is not effective for
   https

5. The colo firewall intentionally prevents general accesses to the
   global internet by builds (or indeed tests) run by osstest.
   One purpose is to prevent uncached accesses, which are a
   performance and reliability problem.

This is hard to fix because:

6. Using the https_proxy variable would not help.  It would cause
   libcurl (which is what git is using) to use CONNECT.  That is
   not what we need: we need the client to trust the proxy, so
   that we can get cacheing.

7. squid would support proxying GET https:// requests but there is
   AFAICT no way for curl to do this (even in fairly recent curl).
   I found this thread on the subject:
     https://curl.haxx.se/mail/archive-2015-12/0009.html

I think in the medium term the right answer is for osstest to use its
git cacheing proxy, rather than trying to cache the http[s] protocol,
by specifying a .gitconfig involving insteadOf.

For now I have asked Credativ to disable the https redirect on
xenbits.

I would be interested to know whether this redirect caused other
problems.  Should we reenable it when osstest is improved ?

When should we change the http:// references in xen.git to https:// ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel