[Xen-devel] Xen Release 4.12.2 and Python 3: M4 python_devel module and mkheader.py issues

[Kevin Buckley <kevin.buckley.ecs.vuw.ac.nz@xxxxxxxxx>]

Following on from some previous observations on building Xen from
source a Linux From Scratch system, I have some more feedback
when using a more up-to-date Xen and the most recent LFS.

On the back of trying to build Xen 4.12.2 on an LFS 9.0 system,
I've hit an issue in the way that the Xen M4 python_devel module
gets a version number, and then, I think, constructs the shared
library name that it tries to use in 'configure' incorrectly.

The first issue is that Xen's M4 python_devel module tries to
get a version number for "$PYTHON" by doing

> head ../m4/python_devel.m4
AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
ac_previous_cppflags=$CPPFLAGS
ac_previous_ldflags=$LDFLAGS
ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
    print distutils.sysconfig.get_config_var("VERSION")'`
AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
...


That fails as follows

...
checking for unistd.h... yes
  File "<string>", line 1
    import distutils.sysconfig;     print distutils.sysconfig.get_config_var("VE
RSION")

                                                  ^
SyntaxError: invalid syntax
checking for python3-config... /usr/bin/python3-config
..


If we check out the command in an interactive interpeter session,
we can identify the problen here, vis


> python3
Python 3.7.4 (default, Nov 17 2019, 18:19:17)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.sysconfig
>>> distutils.sysconfig.get_config_var("VERSION")
'3.7'
>>>
>>> print distutils.sysconfig.get_config_var("VERSION")'
  File "<stdin>", line 1
    print distutils.sysconfig.get_config_var("VERSION")'
                  ^
SyntaxError: invalid syntax
>>> print(distutils.sysconfig.get_config_var("VERSION"))
3.7
>>> quit()

which suggests that the issue comes from the "print" statement,
not the distutils module.


It is fairly easy to fix the syntax there, by patching Xen's
M4 python_devel module to have parentheses around the argument
to the print command, vis:


ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
    print(distutils.sysconfig.get_config_var("VERSION"))'`


however,  now that the Python interpreter is returning "3.7"
as the "version", Xen's configure script goes on to run some
tests as follows

...
checking for unistd.h... yes
checking for python3-config... /usr/bin/python3-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for PyArg_ParseTuple in -lpython3.7... no
configure: error: Unable to find a suitable python development library
configure: error: ./configure failed for tools

and here's the actual (from tools/config.log) command that's being
run as the test


configure:7456: checking for PyArg_ParseTuple in -lpython3.7
configure:7481: gcc -o conftest -g -O2 -g -O2 \
 -I/usr/include/python3.7m -I/usr/include/python3.7m  \
 -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv \
 -O3 -Wall     -L/usr/lib -lpython3.7m -lcrypt -lpthread \
 -ldl  -lutil -lm  conftest.c -lpython3.7  -lpython3.7m \
 -lcrypt -lpthread -ldl  -lutil -lm   >&5
/usr/bin/ld: cannot find -lpython3.7



So here's the issue:

in LFS 9.0, Section 6.51, Python-3.7.4 we read

  6.51.2. Contents of Python 3

  Installed Programs: 2to3, idle3, pip3, pydoc3, python3, python3-config,
    and pyvenv

  Installed Library: libpython3.7m.so and libpython3.so

  Installed Directories: /usr/include/python3.7m, /usr/lib/python3
    and /usr/share/doc/python-3.7.4

and, on the LFS system, we indeed have

> ls -o /usr/lib/libpython3.*
lrwxrwxrwx 1 python3      20 Nov 17 18:20 /usr/lib/libpython3.7m.so -> libpython
3.7m.so.1.0
-rwxr-xr-x 1 python3 3265256 Nov 17 20:10 /usr/lib/libpython3.7m.so.1.0
-rwxr-xr-x 1 python3   13984 Nov 17 20:10 /usr/lib/libpython3.so


Note that the library names end in 3.7m, not 3.7.

Clearly then, with  the distutils module returning the version
as "3.7", the Xen configure script goes on to look to link against

  -lpython3.7

and not

  -lpython3.7m


It's fairly easy to add a link

   /usr/lib/libpython3.7.so -> libpython3.7m.so.1.0

that will see what Python's distutils module tells Xen it's version
is, match with the library that Xen then decides it should be looking
for but, given that using "python3-config" to get the LDFLAGS,

> /usr/bin/python3-config --ldflags
 -L/usr/lib -lpython3.7m -lcrypt -lpthread -ldl  -lutil -lm
>

would have suppiled the correct name, it might be worth taking
a look at the use of Python's distutils module in the M4 module.


But let's move on.

Having patched Xen's M4 python_devel module to have parentheses
around the argument to the print command, and having created a
link for the name that Xen's configure script will look for, I've
encountered another error within the "make", vis:

> PYTHON=/usr/bin/python3 \
  ./configure --prefix=/usr    \
  --disable-seabios          \
  --disable-qemu-traditional \
  --disable-rombios          \
  --disable-stubdom

> make \
    PYTHON=/usr/bin/python3 \
    EFI_DIR=/usr/lib/efi
...
 make[2]: Leaving directory
'/usr/src/xen/xen-RELEASE-4.12.2/tools/include/xen/lib/x86'
make -C xen-foreign
make[2]: Entering directory
'/usr/src/xen/xen-RELEASE-4.12.2/tools/include/xen-foreign'
/usr/bin/python3 mkheader.py arm32 arm32.h.tmp
/usr/src/xen/xen-RELEASE-4.12.2/tools/include/xen-foreign/../../../xen/include/public/arch-arm.h
/usr/src/xen/xen-RELEASE-4.12.2/tools/include/xen-foreign/../../../xen/include/public/xen.h
Traceback (most recent call last):
  File "mkheader.py", line 209, in <module>
    integers.sort(lambda a, b: cmp(len(b),len(a)));
AttributeError: 'dict_keys' object has no attribute 'sort'
make[2]: *** [Makefile:28: arm32.h] Error 1
make[2]: Leaving directory
'/usr/src/xen/xen-RELEASE-4.12.2/tools/include/xen-foreign'
make[1]: *** [Makefile:13: xen-foreign] Error 2
make[1]: Leaving directory '/usr/src/xen/xen-RELEASE-4.12.2/tools/include'
make: *** [Makefile:44: install-tools-public-headers] Error 2


This one, I'm less able to diagnose, although a "sort" does sound
like something that a 'dict_keys' object should have?


Any clues then, as to whether this is another Python3 hangover for Xen ?

Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel