[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v3 1/5] xen/scripts: add xen-analysis.py for coverity and eclair analysis



On Wed, 7 Dec 2022, Luca Fancellu wrote:
> Add new script for coverity/eclair analysis tool that will enable
> the procedure to suppress findings when these tool are used.
> The procedure is documented in docs/misra/documenting-violations.rst
> and the script is documented in docs/misra/xen-static-analysis.rst.
> 
> Add in docs/misra/ the files safe.json and
> false-positive-{coverity,eclair}.json that are JSON files containing
> the data structures for the justifications, they are used by the
> analysis script to link the Xen tags to the proprietary tool comment.
> 
> Add docs/misra/documenting-violations.rst to explain how to add
> justifications.
> 
> Add docs/misra/xen-static-analysis.rst to explain how to use the
> script to analyse Xen.
> 
> Add analysis artifacts files to .gitignore.
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@xxxxxxx>

Reviewed-by: Stefano Stabellini <sstabellini@xxxxxxxxxx>


> ---
> v3 changes:
>  - improved help content in the script (Stefano)
>  - used utils.invoke_command from this patch (Stefano)
> v2 changes:
>  - no change
> ---
> ---
>  .gitignore                                   |   1 +
>  docs/misra/documenting-violations.rst        | 191 +++++++++++++++++++
>  docs/misra/false-positive-coverity.json      |  12 ++
>  docs/misra/false-positive-eclair.json        |  12 ++
>  docs/misra/safe.json                         |  11 ++
>  docs/misra/xen-static-analysis.rst           |  54 ++++++
>  xen/scripts/xen-analysis.py                  |  31 +++
>  xen/scripts/xen_analysis/__init__.py         |   0
>  xen/scripts/xen_analysis/generic_analysis.py |  87 +++++++++
>  xen/scripts/xen_analysis/settings.py         | 111 +++++++++++
>  xen/scripts/xen_analysis/tag_database.py     | 109 +++++++++++
>  xen/scripts/xen_analysis/utils.py            |  56 ++++++
>  12 files changed, 675 insertions(+)
>  create mode 100644 docs/misra/documenting-violations.rst
>  create mode 100644 docs/misra/false-positive-coverity.json
>  create mode 100644 docs/misra/false-positive-eclair.json
>  create mode 100644 docs/misra/safe.json
>  create mode 100644 docs/misra/xen-static-analysis.rst
>  create mode 100755 xen/scripts/xen-analysis.py
>  create mode 100644 xen/scripts/xen_analysis/__init__.py
>  create mode 100644 xen/scripts/xen_analysis/generic_analysis.py
>  create mode 100644 xen/scripts/xen_analysis/settings.py
>  create mode 100644 xen/scripts/xen_analysis/tag_database.py
>  create mode 100644 xen/scripts/xen_analysis/utils.py
> 
> diff --git a/.gitignore b/.gitignore
> index ea3243af9dde..f5a66f6194dd 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -10,6 +10,7 @@
>  *.c.cppcheck
>  *.opic
>  *.a
> +*.safparse
>  *.so
>  *.so.[0-9]*
>  *.bin
> diff --git a/docs/misra/documenting-violations.rst 
> b/docs/misra/documenting-violations.rst
> new file mode 100644
> index 000000000000..1d23447556d2
> --- /dev/null
> +++ b/docs/misra/documenting-violations.rst
> @@ -0,0 +1,191 @@
> +.. SPDX-License-Identifier: CC-BY-4.0
> +
> +Documenting violations
> +======================
> +
> +Static analysers are used on the Xen codebase for both static analysis and 
> MISRA
> +compliance.
> +There might be the need to suppress some findings instead of fixing them and
> +many tools permit the usage of in-code comments that suppress findings so 
> that
> +they are not shown in the final report.
> +
> +Xen includes a tool capable of translating a specific comment used in its
> +codebase to the right proprietary in-code comment understandable by the 
> selected
> +analyser that suppress its finding.
> +
> +In the Xen codebase, these tags will be used to document and suppress 
> findings:
> +
> + - SAF-X-safe: This tag means that the next line of code contains a finding, 
> but
> +   the non compliance to the checker is analysed and demonstrated to be safe.
> + - SAF-X-false-positive-<tool>: This tag means that the next line of code
> +   contains a finding, but the finding is a bug of the tool.
> +
> +SAF stands for Static Analyser Finding, the X is a placeholder for a positive
> +number that starts from zero, the number after SAF- shall be incremental and
> +unique, base ten notation and without leading zeros.
> +
> +Entries in the database shall never be removed, even if they are not used
> +anymore in the code (if a patch is removing or modifying the faulty line).
> +This is to make sure that numbers are not reused which could lead to 
> conflicts
> +with old branches or misleading justifications.
> +
> +An entry can be reused in multiple places in the code to suppress a finding 
> if
> +and only if the justification holds for the same non-compliance to the coding
> +standard.
> +
> +An orphan entry, that is an entry who was justifying a finding in the code, 
> but
> +later that code was removed and there is no other use of that entry in the 
> code,
> +can be reused as long as the justification for the finding holds. This is 
> done
> +to avoid the allocation of a new entry with exactly the same justification, 
> that
> +would lead to waste of space and maintenance issues of the database.
> +
> +The files where to store all the justifications are in xen/docs/misra/ and 
> are
> +named as safe.json and false-positive-<tool>.json, they have JSON format, 
> each
> +one has a different justification schema which shares some fields.
> +
> +Here is an example to add a new justification in safe.json::
> +
> +|{
> +|    "version": "1.0",
> +|    "content": [
> +|        {
> +|            "id": "SAF-0-safe",
> +|            "analyser": {
> +|                "coverity": "misra_c_2012_rule_20_7_violation",
> +|                "eclair": "MC3R1.R20.7"
> +|            },
> +|            "name": "R20.7 C macro parameters not used as expression",
> +|            "text": "The macro parameters used in this [...]"
> +|        },
> +|        {
> +|            "id": "SAF-1-safe",
> +|            "analyser": {},
> +|            "name": "Sentinel",
> +|            "text": "Next ID to be used"
> +|        }
> +|    ]
> +|}
> +
> +To document a finding in safe.json, just add another block {[...]} before the
> +sentinel block, using the id contained in the sentinel block and increment by
> +one the number contained in the id of the sentinel block.
> +
> +Here is an explanation of the fields inside an object of the "content" array:
> + - id: it is a unique string that is used to refer to the finding, many 
> finding
> +   can be tagged with the same id, if the justification holds for any applied
> +   case.
> +   It tells the tool to substitute a Xen in-code comment having this 
> structure:
> +   /* SAF-0-safe [...] \*/
> + - analyser: it is an object containing pair of key-value strings, the key is
> +   the analyser, so it can be coverity or eclair, the value is the 
> proprietary
> +   id corresponding on the finding, for example when coverity is used as
> +   analyser, the tool will translate the Xen in-code coment in this way:
> +   /* SAF-0-safe [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] 
> \*/
> +   if the object doesn't have a key-value, then the corresponding in-code
> +   comment won't be translated.
> + - name: a simple name for the finding
> + - text: a proper justification to turn off the finding.
> +
> +
> +Here is an example to add a new justification in false-positive-<tool>.json::
> +
> +|{
> +|    "version": "1.0",
> +|    "content": [
> +|        {
> +|            "id": "SAF-0-false-positive-<tool>",
> +|            "violation-id": "<proprietary-id>",
> +|            "tool-version": "<version>",
> +|            "name": "R20.7 [...]",
> +|            "text": "[...]"
> +|        },
> +|        {
> +|            "id": "SAF-1-false-positive-<tool>",
> +|            "violation-id": "",
> +|            "tool-version": "",
> +|            "name": "Sentinel",
> +|            "text": "Next ID to be used"
> +|        }
> +|    ]
> +|}
> +
> +To document a finding in false-positive-<tool>.json, just add another block
> +{[...]} before the sentinel block, using the id contained in the sentinel 
> block
> +and increment by one the number contained in the id of the sentinel block.
> +
> +Here is an explanation of the fields inside an object of the "content" array:
> + - id: it has the same meaning as in the "safe" justification schema.
> +   It tells the tool to substitute a Xen in-code comment having this 
> structure:
> +   /* SAF-0-false-positive-<tool> [...] \*/
> + - violation-id: its value is a string containing the proprietary id
> +   corresponding to the finding, for example when <tool> is coverity, the Xen
> +   tool will translate the Xen in-code coment in this way:
> +   /* SAF-0-false-positive-coverity [...] \*/ -> /* 
> coverity[misra_c_2012_rule_20_7_violation] \*/
> +   if the object doesn't have a value, then the corresponding in-code comment
> +   won't be translated.
> + - tool-version: the version of the tool affected by the false positive, if 
> it
> +   is discovered in more than one version, this string can be a range
> +   (eg. 2.7 - 3.0)
> + - name, text: they have the same meaning as in the "safe" justification 
> schema.
> +
> +
> +Justification example
> +---------------------
> +
> +Here an example of the usage of the in-code comment tags to suppress a 
> finding
> +for the Rule 8.6:
> +
> +Eclair reports it in its web report, file xen/include/xen/kernel.h, line 68:
> +
> +| MC3R1.R8.6 for program 'xen/xen-syms', variable '_start' has no definition
> +
> +Also coverity reports it, here is an extract of the finding:
> +
> +| xen/include/xen/kernel.h:68:
> +| 1. misra_c_2012_rule_8_6_violation: Function "_start" is declared but never
> + defined.
> +
> +The analysers are complaining because we have this in 
> xen/include/xen/kernel.h
> +at line 68::
> +
> +| extern char _start[], _end[], start[];
> +
> +Those are symbols exported by the linker, hence we will need to have a proper
> +deviation for this finding.
> +
> +We will prepare our entry in the safe.json database::
> +
> +|{
> +|    "version": "1.0",
> +|    "content": [
> +|        {
> +|        [...]
> +|        },
> +|        {
> +|            "id": "SAF-1-safe",
> +|            "analyser": {
> +|                "eclair": "MC3R1.R8.6",
> +|                "coverity": "misra_c_2012_rule_8_6_violation"
> +|            },
> +|            "name": "Rule 8.6: linker script defined symbols",
> +|            "text": "It is safe to declare this symbol because it is 
> defined in the linker script."
> +|        },
> +|        {
> +|            "id": "SAF-2-safe",
> +|            "analyser": {},
> +|            "name": "Sentinel",
> +|            "text": "Next ID to be used"
> +|        }
> +|    ]
> +|}
> +
> +And we will use the proper tag above the violation line::
> +
> +| /* SAF-1-safe R8.6 linker defined symbols */
> +| extern char _start[], _end[], start[];
> +
> +This entry will fix also the violation on _end and start, because they are on
> +the same line and the same "violation ID".
> +
> +Also, the same tag can be used on other symbols from the linker that are
> +declared in the codebase, because the justification holds for them too.
> diff --git a/docs/misra/false-positive-coverity.json 
> b/docs/misra/false-positive-coverity.json
> new file mode 100644
> index 000000000000..462448414f80
> --- /dev/null
> +++ b/docs/misra/false-positive-coverity.json
> @@ -0,0 +1,12 @@
> +{
> +    "version": "1.0",
> +    "content": [
> +        {
> +            "id": "SAF-0-false-positive-coverity",
> +            "violation-id": "",
> +            "tool-version": "",
> +            "name": "Sentinel",
> +            "text": "Next ID to be used"
> +        }
> +    ]
> +}
> diff --git a/docs/misra/false-positive-eclair.json 
> b/docs/misra/false-positive-eclair.json
> new file mode 100644
> index 000000000000..1d6ea5d7f045
> --- /dev/null
> +++ b/docs/misra/false-positive-eclair.json
> @@ -0,0 +1,12 @@
> +{
> +    "version": "1.0",
> +    "content": [
> +        {
> +            "id": "SAF-0-false-positive-eclair",
> +            "violation-id": "",
> +            "tool-version": "",
> +            "name": "Sentinel",
> +            "text": "Next ID to be used"
> +        }
> +    ]
> +}
> diff --git a/docs/misra/safe.json b/docs/misra/safe.json
> new file mode 100644
> index 000000000000..e079d3038120
> --- /dev/null
> +++ b/docs/misra/safe.json
> @@ -0,0 +1,11 @@
> +{
> +    "version": "1.0",
> +    "content": [
> +        {
> +            "id": "SAF-0-safe",
> +            "analyser": {},
> +            "name": "Sentinel",
> +            "text": "Next ID to be used"
> +        }
> +    ]
> +}
> diff --git a/docs/misra/xen-static-analysis.rst 
> b/docs/misra/xen-static-analysis.rst
> new file mode 100644
> index 000000000000..5b886474d4a0
> --- /dev/null
> +++ b/docs/misra/xen-static-analysis.rst
> @@ -0,0 +1,54 @@
> +.. SPDX-License-Identifier: CC-BY-4.0
> +
> +Xen static analysis
> +===================
> +
> +The Xen codebase integrates some scripts and tools that helps the developer 
> to
> +perform static analysis of the code, currently Xen supports three analysis 
> tool
> +that are eclair, coverity and cppcheck.
> +The Xen tree has a script (xen-analysis.py) available to ease the analysis
> +process and it integrates a way to suppress findings on these tools (only 
> Eclair
> +and Coverity are currently supported by the script), please check the
> +documenting-violation.rst document to know more about it.
> +
> +Analyse Xen with Coverity or Eclair
> +-----------------------------------
> +
> +The xen-analysis.py script has two arguments to select which tool is used for
> +the analysis:
> +
> + - xen-analysis.py --run-coverity -- [optional make arguments]
> + - xen-analysis.py --run-eclair -- [optional make arguments]
> +
> +For example when using Coverity to analyse a Xen build obtained by passing 
> these
> +arguments to the make system: XEN_TARGET_ARCH=arm64
> +CROSS_COMPILE=aarch64-linux-gnu-, the optional make arguments passed to
> +xen-analysis.py must be the same and the command below should be passed to
> +Coverity in its build phase:
> +
> + - xen-analysis.py --run-coverity -- XEN_TARGET_ARCH=arm64
> +   CROSS_COMPILE=aarch64-linux-gnu-
> +
> +Which tells to the script to prepare the codebase for an analysis by Coverity
> +and forwards the make arguments to the make build invocation.
> +
> +When invoking the script, the procedure below will be followed:
> +
> + 1. Find which files among \*.c and \*.h has any in-code comment as
> +    /* SAF-X-[...] \*/, the meaning of these comments is explained in
> +    documenting-violation.rst.
> +    Save the files obtained as <file>.safparse and generate <file> files 
> where
> +    the special in-code comments above are substituted with the proprietary
> +    in-code comment used by the selected analysis tool. The safe.json and
> +    false-positive-<tool>.json text file database are used to link each Xen 
> tag
> +    to the right proprietary in-code comment.
> + 2. Now Xen compilation starts using every <additional make parameters> 
> supplied
> +    at the script invocation. Coverity and Eclair are capable of intercepting
> +    the compiler running from make to perform their analysis without
> +    instrumenting the makefile.
> + 3. As final step every <file>.safparse file are reverted back as <file> and
> +    every artifact related to the analysis will be cleaned.
> +    This step is performed even in case any of the previous step fail, to 
> skip
> +    this step, call the script adding the --no-clean argument, but before
> +    running again the script, call it with the --clean-only argument, that 
> will
> +    execute only this cleaning step.
> diff --git a/xen/scripts/xen-analysis.py b/xen/scripts/xen-analysis.py
> new file mode 100755
> index 000000000000..b5d9ef1862c9
> --- /dev/null
> +++ b/xen/scripts/xen-analysis.py
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +
> +import sys
> +from xen_analysis import settings, generic_analysis
> +from xen_analysis.generic_analysis import *
> +
> +
> +def main(argv):
> +    ret_code = 0
> +    settings.parse_commandline(argv)
> +    try:
> +        if settings.step_parse_tags:
> +            generic_analysis.parse_xen_tags()
> +        if settings.step_build_xen:
> +            generic_analysis.build_xen()
> +    except (ParseTagPhaseError, BuildPhaseError) as e:
> +        print("ERROR: {}".format(e))
> +        if hasattr(e, "errorcode"):
> +            ret_code = e.errorcode
> +    finally:
> +        if settings.step_clean_analysis:
> +            e = generic_analysis.clean_analysis_artifacts()
> +            if e:
> +                print("ERROR: {}".format(e))
> +                ret_code = 1
> +
> +    sys.exit(ret_code)
> +
> +
> +if __name__ == "__main__":
> +    main(sys.argv[1:])
> diff --git a/xen/scripts/xen_analysis/__init__.py 
> b/xen/scripts/xen_analysis/__init__.py
> new file mode 100644
> index 000000000000..e69de29bb2d1
> diff --git a/xen/scripts/xen_analysis/generic_analysis.py 
> b/xen/scripts/xen_analysis/generic_analysis.py
> new file mode 100644
> index 000000000000..372ff3566442
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/generic_analysis.py
> @@ -0,0 +1,87 @@
> +#!/usr/bin/env python3
> +
> +import os, subprocess
> +from . import settings, utils, tag_database
> +
> +class ParseTagPhaseError(Exception):
> +    pass
> +
> +class BuildPhaseError(Exception):
> +    pass
> +
> +class CleanPhaseError(Exception):
> +    pass
> +
> +
> +def parse_xen_tags():
> +    # Load the database for the Xen tags
> +    subs_list = tag_database.load_tag_database(
> +        settings.analysis_tool,
> +        [settings.repo_dir + "/docs/misra/safe.json"]
> +    )
> +    subs_list = tag_database.load_tag_database(
> +        settings.analysis_tool,
> +        [settings.repo_dir + "/docs/misra/false-positive-{}.json"
> +                                .format(settings.analysis_tool)],
> +        subs_list,
> +        "false-positive"
> +    )
> +
> +    # Create outdir if it doesn't exists
> +    os.makedirs(settings.outdir, exist_ok=True)
> +
> +    # The following lambda function will return a file if it contains lines 
> with
> +    # a comment containing "SAF-<number>-{safe|false-positive-<tool>}" on a
> +    # single line.
> +    grep_action = lambda x: utils.grep(x,
> +                                    tag_database.get_xen_tag_comment_regex(
> +                                                        
> settings.analysis_tool)
> +    )
> +    # Look for a list of .h/.c files that matches the condition above
> +    parse_file_list = utils.recursive_find_file(settings.xen_dir, 
> r'.*\.[ch]$',
> +                                                grep_action)
> +
> +    for entry in parse_file_list:
> +        file = entry["file"]
> +        bkp_file = file + ".safparse"
> +        if os.path.isfile(bkp_file):
> +            raise ParseTagPhaseError(
> +                "Found {}, please check the integrity of {}"
> +                    .format(bkp_file,file)
> +                )
> +        os.rename(file, bkp_file)
> +        time_bkp_file = os.stat(bkp_file)
> +        # Create <file> from <file>.safparse but with the Xen tag parsed
> +        tag_database.substitute_tags(settings.analysis_tool, bkp_file, entry,
> +                                     subs_list)
> +        # Set timestamp for file equal to bkp_file, so that if the file is
> +        # modified during the process by the user, we can catch it
> +        os.utime(file, (time_bkp_file.st_atime, time_bkp_file.st_mtime))
> +
> +
> +def build_xen():
> +    utils.invoke_command(
> +            "make -C {} {} build"
> +                .format(settings.xen_dir, settings.make_forward_args),
> +            False, BuildPhaseError,
> +            "Build error occured when running:\n{}"
> +        )
> +
> +
> +def clean_analysis_artifacts():
> +    safparse_files = utils.recursive_find_file(settings.xen_dir,
> +                                               r'.*.safparse$')
> +    for original_file in safparse_files:
> +        # This commands strips the .safparse extension, leaving <file>
> +        parsed_file_path = os.path.splitext(original_file)[0]
> +        mtime_original_file = os.stat(original_file).st_mtime
> +        mtime_parsed_file = os.stat(parsed_file_path).st_mtime
> +        if mtime_original_file != mtime_parsed_file:
> +            return CleanPhaseError(
> +                    "The file {} was modified during the analysis "
> +                    "procedure, it is impossible now to restore from the "
> +                    "content of {}, please handle it manually"
> +                    .format(parsed_file_path, original_file)
> +                )
> +        # Replace <file>.safparse to <file>
> +        os.replace(original_file, parsed_file_path)
> diff --git a/xen/scripts/xen_analysis/settings.py 
> b/xen/scripts/xen_analysis/settings.py
> new file mode 100644
> index 000000000000..e8d474eaf378
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/settings.py
> @@ -0,0 +1,111 @@
> +#!/usr/bin/env python3
> +
> +import sys, re, os
> +
> +module_dir = os.path.dirname(os.path.realpath(__file__))
> +xen_dir = os.path.realpath(module_dir + "/../..")
> +repo_dir = os.path.realpath(xen_dir + "/..")
> +tools_dir = os.path.realpath(xen_dir + "/tools")
> +
> +step_parse_tags = True
> +step_build_xen = True
> +step_clean_analysis = True
> +
> +target_build = False
> +target_clean = False
> +
> +analysis_tool = ""
> +make_forward_args = ""
> +outdir = xen_dir
> +
> +
> +def help():
> +    msg="""
> +Usage: {} [OPTION] ... [-- [make arguments]]
> +
> +This script runs the analysis on the Xen codebase.
> +
> +The phases for the analysis are <tags>, <build>, <clean>
> +
> +Depending on the options below, only some phases will run:
> +
> +<no options>: tags, build, clean
> +--build-only: build
> +--clean-only: clean
> +--no-build:   tags, clean
> +--no-clean:   tags, build
> +
> +--no-build/--no-clean can be passed together to avoid both clean and build
> +phases.
> +Tags and build phases need to specify --run-coverity or --run-eclair.
> +
> +Options:
> +  --build-only    Run only the commands to build Xen with the optional make
> +                  arguments passed to the script
> +  --clean-only    Run only the commands to clean the analysis artifacts
> +  -h, --help      Print this help
> +  --no-build      Skip the build Xen phase
> +  --no-clean      Don\'t clean the analysis artifacts on exit
> +  --run-coverity  Run the analysis for the Coverity tool
> +  --run-eclair    Run the analysis for the Eclair tool
> +"""
> +    print(msg.format(sys.argv[0]))
> +
> +
> +def parse_commandline(argv):
> +    global analysis_tool
> +    global make_forward_args
> +    global outdir
> +    global step_parse_tags
> +    global step_build_xen
> +    global step_clean_analysis
> +    global target_build
> +    global target_clean
> +    forward_to_make = False
> +    for option in argv:
> +        if forward_to_make:
> +            # Intercept outdir
> +            outdir_regex = re.match("^O=(.*)$", option)
> +            if outdir_regex:
> +                outdir = outdir_regex.group(1)
> +            # Forward any make arguments
> +            make_forward_args = make_forward_args + " " + option
> +        elif option == "--build-only":
> +            target_build = True
> +        elif option == "--clean-only":
> +            target_clean = True
> +        elif (option == "--help") or (option == "-h"):
> +            help()
> +            sys.exit(0)
> +        elif option == "--no-build":
> +            step_build_xen = False
> +        elif option == "--no-clean":
> +            step_clean_analysis = False
> +        elif (option == "--run-coverity") or (option == "--run-eclair"):
> +            analysis_tool = option[6:]
> +        elif option == "--":
> +            forward_to_make = True
> +        else:
> +            print("Invalid option: {}".format(option))
> +            help()
> +            sys.exit(1)
> +
> +    if target_build and target_clean:
> +        print("--build-only is not compatible with --clean-only argument.")
> +        sys.exit(1)
> +
> +    if target_clean:
> +        step_parse_tags = False
> +        step_build_xen = False
> +        step_clean_analysis = True
> +        return
> +
> +    if analysis_tool == "":
> +        print("Please specify one analysis tool.")
> +        help()
> +        sys.exit(1)
> +
> +    if target_build:
> +        step_parse_tags = False
> +        step_build_xen = True
> +        step_clean_analysis = False
> diff --git a/xen/scripts/xen_analysis/tag_database.py 
> b/xen/scripts/xen_analysis/tag_database.py
> new file mode 100644
> index 000000000000..ca374bbb62dd
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/tag_database.py
> @@ -0,0 +1,109 @@
> +#!/usr/bin/env python3
> +
> +import re, json
> +
> +class TagDatabaseError(Exception):
> +    pass
> +
> +# This is the dictionary for the rules that translates to proprietary 
> comments:
> +#  - cppcheck: /* cppcheck-suppress[id] */
> +#  - coverity: /* coverity[id] */
> +#  - eclair:   /* -E> hide id 1 "" */
> +# Add entries to support more analyzers
> +tool_syntax = {
> +    "cppcheck":"cppcheck-suppress[VID]",
> +    "coverity":"coverity[VID]",
> +    "eclair":"-E> hide VID 1 \"\""
> +}
> +
> +
> +def get_xen_tag_index_type_regex(tool):
> +    return r'^SAF-(\d+)-(safe|false-positive-' + tool + ')$'
> +
> +
> +def get_xen_tag_comment_regex(tool):
> +    return r'^[ \t]*/\* +(SAF-\d+-(?:safe|false-positive-' + tool + 
> ')).*\*/$'
> +
> +
> +# Returns a data structure containing dictionaries for safe and 
> false-positive-*
> +# Xen tags, the key is the unique index of the tag and the content is the
> +# proprietary in-code comment to be used when the tag is found in the 
> codebase
> +def load_tag_database(tool, input_files, data_struct = None, schema = 
> "safe"):
> +    ret = data_struct if data_struct is not None else {
> +        "safe": {},
> +        "false-positive-" + tool: {}
> +    }
> +    database = []
> +
> +    # Open all input files
> +    for file in input_files:
> +        try:
> +            with open(file, "rt") as handle:
> +                content = json.load(handle)
> +                database = database + content['content']
> +        except json.JSONDecodeError as e:
> +            raise TagDatabaseError("JSON decoding error in file {}: {}"
> +                                    .format(file, e))
> +        except Exception as e:
> +            raise TagDatabaseError("Can't open file {}: {}"
> +                                    .format(file, e))
> +
> +    for entry in database:
> +        # If the false-positive schema is used, check the proprietary id in 
> the
> +        # 'violation-id' field, otherwise rely on the "safe" schema.
> +        if schema == "false-positive":
> +            proprietary_id = entry['violation-id']
> +        elif tool in entry['analyser']:
> +            proprietary_id = entry['analyser'][tool]
> +        else:
> +            proprietary_id = ""
> +        if proprietary_id != "":
> +            comment=tool_syntax[tool].replace("VID",proprietary_id)
> +            # Regex to capture the index of the Xen tag and the schema
> +            xen_tag = re.compile(get_xen_tag_index_type_regex(tool))\
> +                            .match(entry["id"])
> +            if xen_tag and xen_tag.group(1) and xen_tag.group(2):
> +                # Save in safe or false-positive-* the key {#id: "comment"}
> +                id_number = int(xen_tag.group(1))
> +                key = xen_tag.group(2)
> +                ret[key][id_number] = "/* {} */\n".format(comment)
> +            else:
> +                raise TagDatabaseError(
> +                        "Error in database file, entry {} has unexpected "
> +                        "format.".format(entry["id"])
> +                    )
> +
> +    return ret
> +
> +
> +def substitute_tags(tool, input_file, grep_struct, subs_rules):
> +    try:
> +        with open(grep_struct["file"], "wt") as outfile:
> +
> +            try:
> +                with open(input_file, "rt") as infile:
> +                    parsed_content = infile.readlines()
> +            except Exception as e:
> +                raise TagDatabaseError("Issue with reading file {}: {}"
> +                                       .format(input_file, e))
> +
> +            # grep_struct contains the line number where the comments are, 
> the
> +            # line number starts from 1 but in the array the first line is 
> zero.
> +            # For every line where there is a Xen tag comment, get the Xen 
> tag
> +            # that is in the capture group zero, extract from the Xen tag the
> +            # unique index and the type (safe, false-positive-*) and with 
> those
> +            # information access the subs_rules dictionary to see if there is
> +            # a match
> +            for line_number in grep_struct["matches"]:
> +                xen_tag = grep_struct["matches"][line_number][0]
> +                xen_tag_regex_obj = re.compile(
> +                            
> get_xen_tag_index_type_regex(tool)).match(xen_tag)
> +                id_number = int(xen_tag_regex_obj.group(1))
> +                key = xen_tag_regex_obj.group(2)
> +                if id_number in subs_rules[key]:
> +                    parsed_content[line_number-1] = 
> subs_rules[key][id_number]
> +
> +            outfile.writelines(parsed_content)
> +    except Exception as e:
> +        raise TagDatabaseError("Issue with writing file {}: {}"
> +                               .format(grep_struct["file"], e))
> diff --git a/xen/scripts/xen_analysis/utils.py 
> b/xen/scripts/xen_analysis/utils.py
> new file mode 100644
> index 000000000000..1193e3f4631e
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/utils.py
> @@ -0,0 +1,56 @@
> +#!/usr/bin/env python3
> +
> +import os, re, subprocess
> +
> +
> +def grep(filepath, regex):
> +    regObj = re.compile(regex)
> +    res = { "file": filepath, "matches": {} }
> +    try:
> +        with open(filepath, "rt") as f:
> +            line_number = 1
> +            for line in f:
> +                match = regObj.match(line)
> +                if match:
> +                    res["matches"][line_number] = match.groups()
> +                line_number = line_number + 1
> +    except Exception as e:
> +        print("WARNING: Can't open {}: {}".format(filepath, e))
> +
> +    # Return filename and line matches if there are
> +    return res if res["matches"] else {}
> +
> +
> +def recursive_find_file(path, filename_regex, action = None):
> +    filename_reg_obj = re.compile(filename_regex)
> +    res = []
> +    for root, dirs, fnames in os.walk(path):
> +        for fname in fnames:
> +            if filename_reg_obj.match(fname):
> +                if action is None:
> +                    res.append(os.path.join(root, fname))
> +                else:
> +                    out = action(os.path.join(root, fname))
> +                    if out:
> +                        res.append(out)
> +
> +    return res
> +
> +
> +def invoke_command(command, needs_output, exeption_type = Exception,
> +                   exeption_msg = ""):
> +    try:
> +        pipe_stdout = subprocess.PIPE if (needs_output == True) else None
> +        output = subprocess.run(command, shell=True, check=True,
> +                                stdout=pipe_stdout, stderr=subprocess.STDOUT,
> +                                encoding='utf8')
> +    except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
> +        if needs_output == True:
> +            exeption_msg = exeption_msg.format(e.cmd, output.stdout)
> +        else:
> +            exeption_msg = exeption_msg.format(e.cmd)
> +        excp = exeption_type(exeption_msg)
> +        excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
> +        raise excp
> +
> +    return output.stdout
> -- 
> 2.17.1
> 



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.