Re: xen master: xl create hangs

[Anthony PERARD <anthony.perard@xxxxxxxxxx>]

CCing Juergen and xen-devel.

On Mon, Jul 18, 2022 at 06:25:54PM +0000, Mathieu Tarral wrote:
> Using gdb to debug the xl process, I get the following stacktrace:
> 
> (gdb) bt
> #0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, 
> op=265, expected=8652, futex_word=0x7f6debd22a50) at 
> ./nptl/futex-internal.c:57
> #1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, 
> clockid=0, expected=8652, futex_word=0x7f6debd22a50) at 
> ./nptl/futex-internal.c:87
> #2  __GI___futex_abstimed_wait_cancelable64 
> (futex_word=futex_word@entry=0x7f6debd22a50, expected=8652, 
> clockid=clockid@entry=0, abstime=abstime@entry=0x0,
>     private=private@entry=128) at ./nptl/futex-internal.c:139
> #3  0x00007f6deba736a4 in __pthread_clockjoin_ex (threadid=140110084581248, 
> thread_return=thread_return@entry=0x0, clockid=clockid@entry=0,
>     abstime=abstime@entry=0x0, block=block@entry=true) at 
> ./nptl/pthread_join_common.c:105
> #4  0x00007f6deba73543 in ___pthread_join (threadid=<optimized out>, 
> thread_return=thread_return@entry=0x0) at ./nptl/pthread_join.c:24
> #5  0x00007f6deb9a144b in xs_daemon_close (h=0x561db3bc5bc0) at xs.c:366
> #6  0x00007f6deb9a145f in xs_close (xsh=<optimized out>) at xs.c:386
> #7  0x00007f6debc43a36 in libxl_ctx_free (ctx=0x561db3bc52e0) at libxl.c:173
> #8  0x0000561db33bf5a3 in xl_ctx_free () at xl.c:370
> #9  0x00007f6deba22495 in __run_exit_handlers (status=0, listp=0x7f6debbf6838 
> <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true,
>     run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113
> #10 0x00007f6deba22610 in __GI_exit (status=<optimized out>) at 
> ./stdlib/exit.c:143
> #11 0x00007f6deba06d97 in __libc_start_call_main 
> (main=main@entry=0x561db33c0425 <main>, argc=argc@entry=4, 
> argv=argv@entry=0x7ffeb2f263d8)
>     at ../sysdeps/nptl/libc_start_call_main.h:74
> #12 0x00007f6deba06e40 in __libc_start_main_impl (main=0x561db33c0425 <main>, 
> argc=4, argv=0x7ffeb2f263d8, init=<optimized out>, fini=<optimized out>,
>     rtld_fini=<optimized out>, stack_end=0x7ffeb2f263c8) at 
> ../csu/libc-start.c:392
> #13 0x0000561db33bf425 in _start ()
> 
> Colorized version in a Github Gist:
> https://gist.github.com/Wenzel/4da1e0a025954fac13a0ee57147cc44f
> 
> So looks like xs_daemon_close is waiting on a thread to join:
> https://github.com/xen-project/xen/blob/a5fb66f4513c2c2d222dcc3753163b15690bd003/tools/libs/store/xs.c#L366

On Wed, Jul 20, 2022 at 12:53:29PM +0000, Mathieu Tarral wrote:
> > Verify that things work properly at that commit, then use that as the 
> > “good” starting point.
> 
> Turns out that this commit (74a11c43fd7e074b1f77631b446dd2115eacb9e8) was 
> also bad.
> So I used git bisect again, but this time to find the commit which introduced 
> the bug fix
> between 74a11c43fd7e074b1f77631b446dd2115eacb9e8 and RELEASE-4.16.1.
> 
> After a few steps, git bisect identified this commit:
> https://github.com/xen-project/xen/commit/59505f48fabed2e6fa5ad992edaabeb4a1441599
> "Turn off debug by default"
> Surprisingly simple.
> 
> And I confirm that it's one that fixes the issue of xl create hanging.
> 
> I cherry-picked this commit on master:
> https://user-images.githubusercontent.com/964610/179986382-a774c91a-7b68-416b-9dbe-226b8aca0673.png
> 
> recompiled and tested again, my master branch now works as expected, tested 
> with the small config file I already had and the XTF test-pv64-example.
> 
> So it works, but I don't know why this commit fixed it.

$(debug) controls the level of optimisation of the compilation to make
it easier to debug.

So, with debug=y, we have libxenstore having issue with killing the
its reading thread? :-(
Maybe that reading thread is doing something that can't be stopped,
maybe it's waiting for a lock. Could you try to print a back trace of
that thread (or even all thread in `xl`)? ("thread apply all bt full" in gdb)

Thanks,

-- 
Anthony PERARD