Re: [Xen-devel] [RFC] Documentation formats, licenses and file system structure

[Lars Kurth <lars.kurth@xxxxxxxxxx>]

On 11/10/2019, 09:32, "Jan Beulich" <jbeulich@xxxxxxxx> wrote:

    On 10.10.2019 20:30, Lars Kurth wrote:
    > On 10/10/2019, 18:05, "Andrew Cooper" <Andrew.Cooper3@xxxxxxxxxx> wrote:
    >     On 10/10/2019 13:34, Lars Kurth wrote:
    >     > Existing formats and licenses
    >     > * Hypercall ABI Documentation generated from Xen public headers
    >     > Format: kerndoc
    >     > License: typically BSD-3-Clause (documentation is generated from 
public headers)
    >     
    >     Its homegrown markup&perl, superimposed on what used to be doxygen in
    >     the past.
    > 
    > Oh, I forgot
    >     
    >     I wasn't planning on reusing any of the markup, and wasn't expecting 
to
    >     use much of the text either.  I'm still considering the option of
    >     defining that xen/public/* isn't the canonical description of the ABI,
    >     because C is the wrong tool for the job.
    >     
    >     Its fine to provide a C set of headers implementing an ABI, but there 
is
    >     a very deliberate reason why the canonical migration v2 spec is in a
    >     text document.
    > 
    > @Stefano: as you and I believe Brian will be spending time on improving 
the
    > ABI docs, I think we need to build some agreement here on what/how
    > to do it. I was assuming that generally the consensus was to have
    > docs close to the code in source, but this does not seem to be the case.
    
    Well, for migration v2 having the spec in a text file seems sensible
    to me. For the public ABI, however, I think it's more helpful to have
    the doc next to the actual definitions. Considering the possible use
    of languages other than C I can certainly see why separating both
    would be even more clean, but I think here we want to weigh practical
    purposes against cleanliness.

I think that is an area where we need to build some consensus. The problem
falls under what is considered "traceability" in safety speak: in other words, 
for the ABI documentation use-case it must be easy to be able to
keep documentation and code in sync. And ideally, we would be able to
check this automatically somehow, or have a bot provide hints such as 
"You changed XYZ and should have a look and check whether ABC needs
changing also".

I have thought about the problem of "traceability" for some time, which
goes far beyond what we need for this use-case. Typical things that need
to be maintained for a "traceable (safety) documentation set" are

## Keeping key docs and code in sync 
The use-cases here are things such as
- keep man pages and xl sources in sync
- keep ABI docs and headers in sync
- keep documents such as the migration b2 spec in sync with
  actual source
 
This is a problem we already have today and where we do this often
fairly poorly manually (as can be seen on how out-of-date
man pages often are)

Possible solutions for this are
- store docs alongside headers (maybe using the same base
file name) => that would work for ABI docs

- have some tagging or meta-information scheme which links
specific source files to docs files => that would work for most
other docs (albeit not always perfectly - e.g. when functionality
is spread over many files and just portions of them)

For example: tools/xl/xl_cmdtable.c  
is linked to files in docs/man/xl*

This means creating a bot/tool which warns that when you change
foo.c to also look at foo.rst and/or ../../docs/.../bar.rst should be
relatively straightforward. It would require some initial effort
to create initial mappings, but these would never really change,
unless we refactor code significantly.
     
## Keeping dependent documents (or portions of documents) in sync
This is something we have not really faced, because we do not
have a lot of docs.  

In a large documentation set having the right chapter/tree
structure enables this. In waterfall software engineering
models, where you start off with high-level documents/
requirements/specs/etc. documents which become
increasingly detailed this is done through a chapter/tree
structure, with the capability to make separate documents
(or sections thereof) on other documents (or sections
thereof). When you change something, a tool such as DOORS 
forces you review and sign off child documents.

This is conceptually similar to what we would need for
"linking" sources to docs as outlined above, only that
the "linking" is between docs. It would also be easy enough
to check and highlight, what else may have to be looked at.  

## Proving that your tests verify design claims and that *all claims* are tested
This is typically the hardest problem to solve. It requires
for test cases (be it a document or actual code) to
link to claims (in a design, architecture, spec, ...)
and to prove that all of them are tested.

If there is linkage capability, then it is straightforward
to verify automatically that all your branches have
test-case leaves in your documentation tree. But at least
in a safety context you would also have to augment this
with code coverage information. 

But for the purpose of this discussion, this use-case
does not add any extra linkage capability other than
being able to tell whether a piece of doc/code
is a test case.

## In summary
I feel that having some basic linkage mechanism    
between key documents and key documents of
code is generally valuable for the community as a
whole, if we can tie to some checking into it.

For rst docs we already have linkage via the
".. toctree::" directives, which may be entirely
sufficient. But we don't have linkage from source
to docs. 

Building some checking around this should not
be too hard, in particular in light of the CIv2
checking we agreed (but is somewhat stalled).

Enabling the same checks to be run locally (before)
sending a patch, should also not be too hard.

I am just raising ideas at this stage. Any opinions
are highly welcome.

Regards
Lars 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel