Source code for xdress.autodescribe

"""This module creates descriptions of C/C++ classes, functions, and variables
from source code, by using external parsers (GCC-XML, Clang AST) and the type system.

This module is available as an xdress plugin by the name ``xdress.autodescribe``.

:author: Anthony Scopatz <scopatz@gmail.com>

Descriptions
============
A key component of API wrapper generation is having a a top-level, abstract
representation of the software that is being wrapped.  In C++ there are three
basic constructs which may be wrapped: variables, functions, and classes.

The abstract representation of a C++ class is known as a **description** (abbr.
*desc*).  This description is simply a Python dictionary with a specific structure.
This structure makes heavy use of the type system to declare the types of all needed
parameters.

The Name Key
------------
The *name* key is a dictionary that represents the API name of the element
being described.  This contains exactly the same keys that the utils.apiname()
type has fields.  While apiname is used for user input and validation, the values
here must truly describe the API element.  The following keys -- and only the
following keys -- are allowed in the name dictionary.

:srcname: str or tuple, the element's API name in the original source code,
    eg. MyClass.
:srcfiles: tuple of str, this is a sequence of unique strings which represents
    the file paths where the API element *may* be defined. For example, ('myfile.c',
    'myfile.h').  If the element is defined outside of these files, then the automatic
    discovery or description may fail. Since these files are parsed they must
    actually exist on the filesystem.
:tarbase: str, the base portion of all automatically generated (target) files.
    This does not include the directory or the file extension.  For example, if you
    wanted cythongen to create a file name 'mynewfile.pyx' then the value here would
    be simply 'mynewfile'.
:tarname: str or tuple, the element's API name in the automatically generated
    (target) files, e.g. MyNewClass.
:incfiles: tuple of str, this is a sequence of all files which must be #include'd
    to access the srcname at compile time.  This should be as minimal of a set as
    possible, preferably only one file.  For example, 'hdf5.h'.
:sidecars: tuple of str, this is a sequence of all sidecar files to use for
    this API element. Like srcfiles, these files must exist for xdress to run.
    For example, 'myfile.py'.
:language: str, flag for the language that the srcfiles are implemented in. Valid
    options are: 'c', 'c++', 'f', 'fortran', 'f77', 'f90', 'python', and 'cython'.

Variable Description Top-Level Keys
------------------------------------
The following are valid top-level keys in a variable description dictionary:
name, namespace, type, docstring, and extra.

:name: dict, the variable name, see above
:namespace: str or None, the namespace or module the variable lives in.
:type: str or tuple, the type of the variable
:docstring: str, optional, this is a documentation string for the variable.
:extra: dict, optional, this stores arbitrary metadata that may be used with
    different backends. It is not added by any auto-describe routine but may be
    inserted later if needed.  One example use case is that the Cython generation
    looks for the pyx, pxd, and cpppxd keys for strings of supplemental Cython
    code to insert directly into the wrapper.

Function Description Top-Level Keys
------------------------------------
The following are valid top-level keys in a function description dictionary:
name, namespace, signatures, docstring, and extra.

:name: dict, the function name, see above
:namespace: str or None, the namespace or module the function lives in.
:signatures: dict or dict-like, the keys of this dictionary are function call
    signatures and the values are dicts of non-signature information.
    The signatures themselves are tuples. The first element of these tuples is the
    method name. The remaining elements (if any) are the function arguments.
    Arguments are themselves length-2 tuples whose first elements are the argument
    names and the second element is the argument type. The values are themselves
    dicts with the following keys:

        :return: the return type of this function. Unlike class constuctors
            and destructors, the return type may not be None (only 'void' values
            are allowed).
        :defaults: a length-N tuple of length-2 tuples of the default argument
            kinds and values. N must be the number of arguments in the signature.
            In the length-2 tuples, the first element must be a member of the
            utils.Arg enum and the second element is the associated default value.
            If no default argument exists use utils.Args.NONE as the kind and
            by convention set the value to None, though this should be ignored in all
            cases.

:docstring: str, optional, this is a documentation string for the function.
:extra: dict, optional, this stores arbitrary metadata that may be used with
    different backends. It is not added by any auto-describe routine but may be
    inserted later if needed.  One example use case is that the Cython generation
    looks for the pyx, pxd, and cpppxd keys for strings of supplemental Cython
    code to insert directly into the wrapper.

Class Description Top-Level Keys
---------------------------------
The following are valid top-level keys in a class description dictionary:
name, parents, namespace, attrs, methods, docstrings, and extra.

:name: dict, the class name, see above
:parents: possibly empty list of strings, the immediate parents of the class
    (not grandparents).
:namespace: str or None, the namespace or module the class lives in.
:attrs: dict or dict-like, the names of the attributes (member variables) of the
    class mapped to their types, given in the format of the type system.
:methods: dict or dict-like, similar to the attrs except that the keys are now
    function signatures and the values are dicts of non-signature information.
    The signatures themselves are tuples. The first element of these tuples is the
    method name. The remaining elements (if any) are the function arguments.
    Arguments are themselves length-2 tuples whose first elements are the argument
    names and the second element is the argument type. The values are themselves
    dicts with the following keys:

        :return: the return type of this function. If the return type is None
            (as opposed to 'void'), then this method is assumed to be a constructor
            or destructor.
        :defaults: a length-N tuple of length-2 tuples of the default argument
            kinds and values. N must be the number of arguments in the signature.
            In the length-2 tuples, the first element must be a member of the
            utils.Arg enum and the second element is the associated default value.
            If no default argument exists use utils.Args.NONE as the kind and
            by convention set the value to None, though this should be ignored in all
            cases.

:construct: str, optional, this is a flag for how the class is implemented.
    Accepted values are 'class' and 'struct'.  If this is not present, then 'class'
    is assumed.  This is most useful from wrapping C structs as Python classes.
:docstrings: dict, optional, this dictionary is meant for storing documentation
    strings.  All values are thus either strings or dictionaries of strings.
    Valid keys include: class, attrs, and methods.  The attrs and methods
    keys are dictionaries which may include keys that mirror the top-level keys of
    the same name.
:extra: dict, optional, this stores arbitrary metadata that may be used with
    different backends. It is not added by any auto-describe routine but may be
    inserted later if needed.  One example use case is that the Cython generation
    looks for the pyx, pxd, and cpppxd keys for strings of supplemental Cython
    code to insert directly into the wrapper.

Toaster Example
---------------
Suppose we have a C++ class called Toaster that takes bread and makes delicious
toast.  A valid description dictionary for this class would be as follows::

    class_desc = {
        'name': {
            'language': 'c++',
            'incfiles': ('toaster.h',),
            'srcfiles': ('src/toaster.h', 'src/toaster.cpp'),
            'srcname': 'Toaster',
            'sidecars': ('src/toaster.py',),
            'tarbase': 'toaster',
            'tarname': 'Toaster',
            },
        'parents': ['FCComp'],
        'namespace': 'bright',
        'construct': 'class',
        'attrs': {
            'n_slices': 'int32',
            'rate': 'float64',
            'toastiness': 'str',
            },
        'methods': {
            ('Toaster',): {'return': None, 'defaults': ()},
            ('Toaster', ('name', 'str')): {'return': None,
                'defaults': ((Args.LIT, ""),)},
            ('Toaster', ('paramtrack', ('set', 'str')), ('name', 'str', '""')): {
                'return': None,
                'defaults': ((Args.NONE, None), (Args.LIT, ""))},
            ('~Toaster',): {'return': None, 'defaults': ()},
            ('tostring',): {'return': 'str', 'defaults': ()},
            ('calc',): {'return': 'Material', 'defaults': ()},
            ('calc', ('incomp', ('map', 'int32', 'float64'))): {
                'return': 'Material',
                'defaults': ((Args.NONE, None),)},
            ('calc', ('mat', 'Material')): {
                'return': 'Material',
                'defaults': ((Args.NONE, None),)},
            ('write', ('filename', 'str')): {
                'return': 'void',
                'defaults': ((Args.LIT, "toaster.txt"),)},
            ('write', ('filename', ('char' '*'), '"toaster.txt"')): {
                'return': 'void',
                'defaults': ((Args.LIT, "toaster.txt"),)},
            },
        'docstrings': {
            'class': "I am a toaster!",
            'attrs': {
                'n_slices': 'the number of slices',
                'rate': 'the toast rate',
                'toastiness': 'the toastiness level',
                },
            'methods': {
                'Toaster': "Make me a toaster!",
                '~Toaster': "Noooooo",
                'tostring': "string representation of the toaster",
                'calc': "actually makes the toast.",
                'write': "persists the toaster state."
                },
            },
        'extra': {
            'pyx': 'toaster = Toaster()  # make toaster singleton'
            },
        }

Automatic Description Generation
--------------------------------
The purpose of this module is to create description dictionaries like those
above by automatically parsing C++ classes.  In theory this parsing step may
be handled by visiting any syntax tree of C++ code.  Two options were pursued here:
GCC-XML and the Python bindings to the Clang AST.  Unfortunately, the Clang AST
bindings lack exposure for template argument types.  These are needed to use any
standard library containers.  Thus while the Clang method was pursued to a mostly
working state, the GCC-XML version is the only fully functional automatic describer
for the moment.

Automatic Descriptions API
==========================
"""
from __future__ import print_function
import os
import io
import re
import sys
import ast
from copy import deepcopy
import linecache
import subprocess
import itertools
import functools
import pickle
import collections
from hashlib import md5
from numbers import Number
from pprint import pprint, pformat
from warnings import warn
from functools import reduce

if os.name == 'nt':
    import ntpath
    import posixpath

# pycparser conditional imports
try:
    import pycparser
    PycparserNodeVisitor = pycparser.c_ast.NodeVisitor
except ImportError:
    pycparser = None
    PycparserNodeVisitor = object  # fake this for class definitions

from . import utils
from .utils import exec_file, RunControl, NotSpecified, Arg, merge_descriptions, \
    find_source, FORBIDDEN_NAMES, find_filenames, warn_forbidden_name, apiname, \
    ensure_apiname, c_literal, extra_filenames, newoverwrite, _lang_exts
from . import astparsers
from .typesystem import TypeSystem

try:
    from . import clang
    from .clang.cindex import CursorKind, TypeKind, AccessKind
except ImportError:
    clang = None

if sys.version_info[0] >= 3:
    basestring = str

# d = int64, u = uint64
_GCCXML_LITERAL_INTS = re.compile('(\d+)([du]?)')
_GCCXML_LITERAL_ENUMS = re.compile('\(\w+::(\w+)\)(\d+)')

_none_arg = Arg.NONE, None
_none_return = {'return': None, 'defaults': ()}

[docs]def clearmemo(): """Clears all function memoizations for autodescribers.""" for x in globals().values(): if callable(x) and hasattr(x, 'cache'): x.cache.clear()
def _make_c_to_xdress(): numpy_to_c = { 'void': 'void', 'bool': 'bool', 'byte': ('char','signed char'), # TODO: Don't assume char is signed in C 'ubyte': 'unsigned char', 'short': ('short','short int'), 'ushort': ('unsigned short','short unsigned int'), 'intc': 'int', 'uintc': 'unsigned int', 'int': ('long','long int'), 'uint': ('unsigned long','long unsigned int'), 'longlong': 'long long', 'ulonglong': ('unsigned long long','long long unsigned int'), 'int16': 'int16_t', 'int32': 'int32_t', 'int64': 'int64_t', 'float32': 'float', 'float64': 'double', 'complex64': 'float _Complex', 'complex128': 'double _Complex', 'longdouble': 'long double', } # TODO: Revisit as part of https://github.com/xdress/xdress/issues/109. # The above table is likely fine (except for the signedness of char), # but we need to make sure the platform-agnostic information isn't # standardized away. c_to_xdress = {} for n,cs in numpy_to_c.items(): for c in cs if isinstance(cs,tuple) else (cs,): c_to_xdress[c] = n return c_to_xdress _c_to_xdress = _make_c_to_xdress() _integer_types = frozenset(s+i for i in 'int16 int32 int64 short intc int longlong'.split() for s in ('','u')) _float_types = frozenset('float double float32 float64'.split()) if clang is not None: _clang_base_types = { TypeKind.VOID : 'void', TypeKind.BOOL : 'bool', TypeKind.CHAR_U : 'ubyte', TypeKind.UCHAR : 'ubyte', TypeKind.USHORT : 'ushort', TypeKind.UINT : 'uintc', TypeKind.ULONG : 'uint', TypeKind.ULONGLONG : 'ulonglong', TypeKind.CHAR_S : 'byte', TypeKind.SCHAR : 'byte', TypeKind.SHORT : 'short', TypeKind.INT : 'intc', TypeKind.LONG : 'int', TypeKind.LONGLONG : 'longlong', TypeKind.FLOAT : 'float32', TypeKind.DOUBLE : 'float64', TypeKind.LONGDOUBLE : 'longdouble', } if 1: # TODO: This step uses platform dependent details, and will need to be cleaned # if we want to generate platform independent .pyx files. from numpy import dtype def hack_type(t): t = dtype(t).name return t[:-4]+'char' if t.endswith('int8') else t _c_to_xdress = dict((k,hack_type(v)) for k,v in _c_to_xdress.items()) if clang is not None: _clang_base_types = dict((k,hack_type(v)) for k,v in _clang_base_types.items()) # Hard code knowledge of certain templated classes, so that allocator arguments # can be stripped. TODO: This should be replaced by a more flexible mechanism hack_template_args = { 'array': ('value_type',), 'deque': ('value_type',), 'forward_list': ('value_type',), 'list': ('value_type',), 'map': ('key_type', 'mapped_type'), 'multimap': ('key_type', 'mapped_type'), 'set': ('key_type',), 'multiset': ('key_type',), 'unordered_map': ('key_type', 'mapped_type'), 'unordered_multimap': ('key_type', 'mapped_type'), 'unordered_set': ('key_type',), 'unordered_multiset': ('key_type',), 'vector': ('value_type',), } # # GCC-XML Describers #
[docs]def gccxml_describe(filename, name, kind, includes=(), defines=('XDRESS',), undefines=(), extra_parser_args=(), ts=None, verbose=False, debug=False, builddir='build', onlyin=None, language='c++', clang_includes=()): """Use GCC-XML to describe the class. Parameters ---------- filename : str The path to the file. name : str The name to describe. kind : str The kind of type to describe, valid flags are 'class', 'func', and 'var'. includes: list of str, optional The list of extra include directories to search for header files. defines: list of str, optional The list of extra macro definitions to apply. undefines: list of str, optional The list of extra macro undefinitions to apply. extra_parser_args : list of str, optional Further command line arguments to pass to the parser. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to diplay extra information while describing the class. debug : bool, optional Flag to enable/disable debug mode. builddir : str, optional Location of -- often temporary -- build files. onlyin: set of str The paths to the files that the definition is allowed to exist in. language : str Valid language flag. clang_includes : ignored Returns ------- desc : dict A dictionary describing the class which may be used to generate API bindings. """ # GCC-XML and/or Cygwin wants posix paths on Windows. posixfilename = posixpath.join(*ntpath.split(filename)) if os.name == 'nt' \ else filename root = astparsers.gccxml_parse(posixfilename, includes=includes, defines=defines, undefines=undefines, extra_parser_args=extra_parser_args, verbose=verbose, debug=debug, builddir=builddir) if onlyin is None: onlyin = set([filename]) describers = {'class': GccxmlClassDescriber, 'func': GccxmlFuncDescriber, 'var': GccxmlVarDescriber} describer = describers[kind](name, root, onlyin=onlyin, ts=ts, verbose=verbose) describer.visit() return describer.desc
[docs]class GccxmlBaseDescriber(object): """Base class used to generate descriptions via GCC-XML output. Sub-classes need only implement a visit() method and optionally a constructor. The default visitor methods are valid for classes.""" _funckey = None _describes = None def __init__(self, name, root=None, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The name to describe. root : element tree node, optional The root element node. onlyin : str, optional Filename the class or struct described must live in. Prevents finding elements of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting. """ self.desc = {'name': name, 'namespace': None} self.name = name self.ts = ts or TypeSystem() self.verbose = verbose self._root = root origonlyin = onlyin onlyin = [onlyin] if isinstance(onlyin, basestring) else onlyin onlyin = set() if onlyin is None else set(onlyin) self.onlyin = set() self._filemap = {} for fnode in root.iterfind("File"): fid = fnode.attrib['id'] fname = fnode.attrib['name'] self._filemap[fid] = fname for fname in onlyin: fnode = root.find("File[@name='{0}']".format(fname)) if fnode is None: fnode = root.find("File[@name='./{0}']".format(fname)) if fnode is None: continue fid = fnode.attrib['id'] self.onlyin.add(fid) if 0 == len(self.onlyin): msg = "{0!r} is not present in {1!r}; autodescribing will probably fail." msg = msg.format(name, origonlyin) warn(msg, RuntimeWarning) self._currfunc = [] # this must be a stack to handle nested functions self._currfuncsig = None self._currargkind = None self._currclass = [] # this must be a stack to handle nested classes self._level = -1 self._template_args = hack_template_args #self._template_args.update(ts.template_types) def __str__(self): return pformat(self.desc) def __del__(self): linecache.clearcache() def _pprint(self, node): if self.verbose: print("{0}{1} {2}: {3}".format(self._level * " ", node.tag, node.attrib.get('id', ''), node.attrib.get('name', None))) def _template_literal_enum_val(self, targ): """Finds the node associated with an enum value in a template""" m = _GCCXML_LITERAL_ENUMS.match(targ) if m is None: return None enumname, val = m.groups() query = ".//*[@name='{0}']" node = self._root.find(query.format(enumname)) if node is None: return None for child in node.iterfind('EnumValue'): if child.attrib['init'] == val: return child return None def _visit_template_function(self, node): nodename = node.attrib['name'] demangled = node.attrib.get('demangled', None) if demangled is None: return (nodename,) # we know we have a template now demangled = demangled[demangled.index(nodename):] template_args = utils.split_template_args(demangled) inst = [nodename] self._level += 1 targ_nodes = [] targ_islit = [] # gross but string parsing of node name is needed. query = ".//*[@name='{0}']" for targ in template_args: targ_node = self._root.find(query.format(targ)) if targ_node is None: try: targ_node = c_literal(targ) targ_islit.append(True) except ValueError: targ_node = self._template_literal_enum_val(targ) if targ_node is None: continue targ_islit.append(False) else: targ_islit.append(False) targ_nodes.append(targ_node) argkinds = [] # just in case it is needed for targ_node, targ_lit in zip(targ_nodes, targ_islit): if targ_lit: targ_kind, targ_value = Arg.LIT, targ_node elif 'id' in targ_node.attrib: targ_kind, targ_value = Arg.TYPE, self.type(targ_node.attrib['id']) else: targ_kind, targ_value = Arg.VAR, targ_node.attrib['name'] argkinds.append(targ_kind) inst.append(targ_value) self._level -= 1 #inst.append(0) This doesn't apply to top-level functions, only function types inst = tuple(inst) return inst def _visit_template_class(self, node): name = node.attrib['name'] members = node.attrib.get('members', '').strip().split() if 0 < len(members): children = [child for m in members for child in \ self._root.iterfind(".//*[@id='{0}']".format(m))] tags = [child.tag for child in children] template_name = children[tags.index('Constructor')].attrib['name'] # 'map' else: template_name = name.split('<', 1)[0] if template_name == 'basic_string': return 'str' inst = [template_name] self._level += 1 targ_nodes = [] targ_islit = [] if template_name in self._template_args: for targ in self._template_args[template_name]: possible_targ_nodes = [c for c in children if c.attrib['name'] == targ] targ_nodes.append(possible_targ_nodes[0]) targ_islit.append(False) else: # gross but string parsing of node name is needed. targs = utils.split_template_args(name) query = ".//*[@name='{0}']" for targ in targs: targ_node = self._root.find(query.format(targ)) if targ_node is None: targ_node = c_literal(targ) targ_islit.append(True) else: targ_islit.append(False) targ_nodes.append(targ_node) argkinds = [] for targ_node, targ_lit in zip(targ_nodes, targ_islit): if targ_lit: targ_kind, targ_value = Arg.LIT, targ_node elif 'id' in targ_node.attrib: targ_kind, targ_value = Arg.TYPE, self.type(targ_node.attrib['id']) else: targ_kind, targ_value = Arg.VAR, targ_node.attrib['name'] argkinds.append(targ_kind) inst.append(targ_value) self._level -= 1 inst.append(0) inst = tuple(inst) self.ts.register_argument_kinds(inst, tuple(argkinds)) return inst
[docs] def visit_class(self, node): """visits a class or struct.""" self._pprint(node) name = node.attrib['name'] self._currclass.append(name) if self._describes == 'class' and (name == self.ts.gccxml_type(self.name) or name == self._name): if 'bases' not in node.attrib: msg = ("The type {0!r} is used as part of an API element but no " "declarations were made with it. Please declare a variable " "of type {0!r} somewhere in the source or header.") raise NotImplementedError(msg.format(name)) bases = node.attrib['bases'].split() # TODO: Record whether bases are public, private, or protected bases = [self.type(b.replace('private:','')) for b in bases] self.desc['parents'] = bases ns = self.context(node.attrib['context']) if ns is not None and ns != "::": self.desc['namespace'] = ns if '<' in name and name.endswith('>'): name = self._visit_template_class(node) self._currclass.pop() return name
visit_struct = visit_class
[docs] def visit_base(self, node): """visits a base class.""" self._pprint(node) self.visit(node) # Walk farther down the tree
def _visit_func(self, node): name = node.attrib['name'] if name.startswith('_') or name in FORBIDDEN_NAMES: warn_forbidden_name(name, self.name) return demangled = node.attrib.get('demangled', "") demangled = demangled if name + '<' in demangled \ and '>' in demangled else None if demangled is None: # normal function self._currfunc.append(name) else: # template function self._currfunc.append(self._visit_template_function(node)) self._currfuncsig = [] self._currargkind = [] self._level += 1 for child in node.iterfind('Argument'): self.visit_argument(child) self._level -= 1 if node.tag == 'Constructor': rtntype = None elif node.tag == 'Destructor': rtntype = None if demangled is None: self._currfunc[-1] = '~' + self._currfunc[-1] else: self._currfunc[-1] = ('~' + self._currfunc[-1][0],) + \ self._currfunc[-1][1:] else: rtntype = self.type(node.attrib['returns']) funcname = self._currfunc.pop() if self._currfuncsig is None: return key = (funcname,) + tuple(self._currfuncsig) self.desc[self._funckey][key] = {'return': rtntype, 'defaults': tuple(self._currargkind)} self._currfuncsig = None self._currargkind = None
[docs] def visit_constructor(self, node): """visits a class constructor.""" self._pprint(node) self._visit_func(node)
[docs] def visit_destructor(self, node): """visits a class destructor.""" self._pprint(node) self._visit_func(node)
[docs] def visit_method(self, node): """visits a member function.""" self._pprint(node) self._visit_func(node)
[docs] def visit_function(self, node): """visits a non-member function.""" self._pprint(node) self._visit_func(node) ns = self.context(node.attrib['context']) if ns is not None and ns != "::": self.desc['namespace'] = ns
[docs] def visit_argument(self, node): """visits a constructor, destructor, or method argument.""" self._pprint(node) name = node.attrib.get('name', None) if name is None: self._currfuncsig = None self._currargkind = None return if name in FORBIDDEN_NAMES: rename = name + '__' warn_forbidden_name(name, self.name, rename) name = rename tid = node.attrib['type'] t = self.type(tid) default = node.attrib.get('default', None) arg = (name, t) if default is None: argkind = _none_arg else: try: default = c_literal(default) islit = True except ValueError: islit = False # Leave default as is argkind = (Arg.LIT if islit else Arg.VAR, default) self._currfuncsig.append(arg) self._currargkind.append(argkind)
[docs] def visit_field(self, node): """visits a member variable.""" self._pprint(node) context = self._root.find(".//*[@id='{0}']".format(node.attrib['context'])) if context.attrib['name'] == self.name: # assert this field is member of the class we are trying to parse name = node.attrib['name'] if name in FORBIDDEN_NAMES: warn_forbidden_name(name, self.name) return t = self.type(node.attrib['type']) self.desc['attrs'][name] = t
[docs] def visit_typedef(self, node): """visits a type definition anywhere.""" self._pprint(node) name = node.attrib.get('name', None) if name == 'string': return 'str' else: return self.type(node.attrib['type'])
def visit_enumeration(self, node): self._pprint(node) currenum = [] for child in node.iterfind('EnumValue'): currenum.append((child.attrib['name'], child.attrib['init'])) return ('enum', node.attrib['name'], tuple(currenum))
[docs] def visit_fundamentaltype(self, node): """visits a base C++ type, mapping it to the approriate type in the type system.""" self._pprint(node) tname = node.attrib['name'] t = _c_to_xdress.get(tname, None) return t
_predicates = frozenset(['*', '&', 'const', 'volatile', 'restrict']) def _add_predicate(self, baset, pred): if isinstance(baset, basestring): return (baset, pred) last = baset[-1] if last in self._predicates or isinstance(last, int): return (baset, pred) else: return tuple(baset) + (pred,)
[docs] def visit_arraytype(self, node): """visits an array type and maps it to a '*' refinement type.""" self._pprint(node) baset = self.type(node.attrib['type']) # FIXME something involving the min, max, and/or size # attribs needs to also go here. t = self._add_predicate(baset, '*') return t
[docs] def visit_functiontype(self, node): """visits an function type and returns a 'function' dependent refinement type.""" self._pprint(node) t = ['function'] args = [] for i, child in enumerate(node.iterfind('Argument')): argt = self.type(child.attrib['type']) args.append(('_{0}'.format(i), argt)) t.append(tuple(args)) rtnt = self.type(node.attrib['returns']) t.append(rtnt) return tuple(t)
[docs] def visit_referencetype(self, node): """visits a reference and maps it to a '&' refinement type.""" self._pprint(node) baset = self.type(node.attrib['type']) t = self._add_predicate(baset, '&') return t
[docs] def visit_pointertype(self, node): """visits a pointer and maps it to a '*' refinement type.""" self._pprint(node) baset = self.type(node.attrib['type']) if baset[0] == 'function': t = ('function_pointer',) + baset[1:] else: t = self._add_predicate(baset, '*') return t
[docs] def visit_cvqualifiedtype(self, node): """visits constant, volatile, and restricted types and maps them to 'const', 'volatile', and 'restrict' refinement types.""" self._pprint(node) t = self.type(node.attrib['type']) if int(node.attrib.get('const', 0)): t = self._add_predicate(t, 'const') if int(node.attrib.get('volatile', 0)): t = self._add_predicate(t, 'volatile') if int(node.attrib.get('restrict', 0)): t = self._add_predicate(t, 'restrict') return t
[docs] def type(self, id): """Resolves the type from its id and information in the root element tree.""" node = self._root.find(".//*[@id='{0}']".format(id)) tag = node.tag.lower() meth_name = 'visit_' + tag meth = getattr(self, meth_name, None) t = None if meth is not None: self._level += 1 t = meth(node) self._level -= 1 return t
[docs] def visit_namespace(self, node): """visits the namespace that a node is defined in.""" self._pprint(node) name = node.attrib['name'] return name
[docs] def context(self, id): """Resolves the context from its id and information in the element tree.""" node = self._root.find(".//*[@id='{0}']".format(id)) tag = node.tag.lower() meth_name = 'visit_' + tag meth = getattr(self, meth_name, None) c = None if meth is not None: self._level += 1 c = meth(node) self._level -= 1 return c
[docs]class GccxmlClassDescriber(GccxmlBaseDescriber): """Class used to generate class descriptions via GCC-XML output.""" _funckey = 'methods' _describes = 'class' _constructvalue = 'class' def __init__(self, name, root=None, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The class name, this may not have a None value. root : element tree node, optional The root element node of the class or struct to describe. onlyin : str, optional Filename the class or struct described must live in. Prevents finding classes of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the class. """ super(GccxmlClassDescriber, self).__init__(name, root=root, onlyin=onlyin, ts=ts, verbose=verbose) self.desc['attrs'] = {} self.desc[self._funckey] = {} self.desc['construct'] = self._constructvalue self.desc['type'] = ts.canon(name) # Gross, but it solves the problem that for uint valued template parameters # (ie MyClass<3>) gccxml names this MyClass<3u> or MyClass<3d> but # the type system has no way of distinguising this from MyClass<3> as an int. # maybe it should, but I think this will get us pretty far. self._name = None def _find_class_node(self): basename = self.name[0] namet = self.desc['type'] query = "Class" for node in self._root.iterfind(query): if node.attrib['file'] not in self.onlyin: continue nodename = node.attrib['name'] if not nodename.startswith(basename): continue if '<' not in nodename or not nodename.endswith('>'): continue nodet = self._visit_template_class(node) if nodet == namet: self._name = nodename # gross break else: node = None return node
[docs] def visit(self, node=None): """Visits the class node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level class node is found and visited. """ if node is None: if not isinstance(self.name, basestring) and self.name not in self.ts.argument_kinds: node = self._find_class_node() if node is None: query = "Class[@name='{0}']".format(self.ts.gccxml_type(self.name)) node = self._root.find(query) if node is None: query = "Struct[@name='{0}']".format(self.ts.gccxml_type(self.name)) node = self._root.find(query) if node is None and not isinstance(self.name, basestring): # Must be a template with some wacky argument values node = self._find_class_node() if node is None: raise RuntimeError("could not find class {0!r}".format(self.name)) if node.attrib['file'] not in self.onlyin: msg = ("{0} autodescribing failed: found class in {1!r} ({2!r}) but " "expected it in {3}.") fid = node.attrib['file'] ois = ", ".join(["{0!r} ({1!r})".format(self._filemap[v], v) \ for v in sorted(self.onlyin)]) print("ONLYIN =", self.onlyin) msg = msg.format(self.name, self._filemap[fid], fid, ois) raise RuntimeError(msg) self.desc['construct'] = node.tag.lower() self.visit_class(node) members = node.attrib.get('members', '').strip().split() children = [self._root.find(".//*[@id='{0}']".format(m)) for m in members] children = [c for c in children if c.attrib['access'] == 'public'] self._level += 1 for child in children: tag = child.tag.lower() meth_name = 'visit_' + tag meth = getattr(self, meth_name, None) if meth is not None: meth(child) self._level -= 1
[docs]class GccxmlVarDescriber(GccxmlBaseDescriber): """Class used to generate variable descriptions via GCC-XML output.""" _describes = 'var' def __init__(self, name, root=None, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The function name, this may not have a None value. root : element tree node, optional The root element node of the function to describe. onlyin : str, optional Filename the function described must live in. Prevents finding functions of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the function. """ super(GccxmlVarDescriber, self).__init__(name, root=root, onlyin=onlyin, ts=ts, verbose=verbose)
[docs] def visit(self, node=None): """Visits the variable node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level class node is found and visited. """ root = node or self._root for n in root.iterfind("Variable[@name='{0}']".format(self.name)): if n.attrib['file'] in self.onlyin: ns = self.context(n.attrib['context']) if ns is not None and ns != "::": self.desc['namespace'] = ns self.desc['type'] = self.type(n.attrib['type']) break else: msg = ("{0} autodescribing failed: found variable in {1!r} but " "expected it in {2!r}.") msg = msg.format(self.name, node.attrib['file'], self.onlyin) raise RuntimeError(msg) # Variables can also be enums for n in root.iterfind("Enumeration[@name='{0}']".format(self.name)): if n.attrib['file'] in self.onlyin: ns = self.context(n.attrib['context']) if ns is not None and ns != "::": self.desc['namespace'] = ns # Grab the type and put it in self.desc['type'] = self.visit_enumeration(n) break else: msg = ("{0} autodescribing failed: found variable in {1!r} but " "expected it in {2!r}.") msg = msg.format(self.name, node.attrib['file'], self.onlyin) raise RuntimeError(msg)
[docs]class GccxmlFuncDescriber(GccxmlBaseDescriber): """Class used to generate function descriptions via GCC-XML output.""" _funckey = 'signatures' _describes = 'func' def __init__(self, name, root=None, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The function name, this may not have a None value. root : element tree node, optional The root element node of the function to describe. onlyin : str, optional Filename the function described must live in. Prevents finding functions of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the function. """ super(GccxmlFuncDescriber, self).__init__(name, root=root, onlyin=onlyin, ts=ts, verbose=verbose) self.desc[self._funckey] = {}
[docs] def visit(self, node=None): """Visits the function node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level class node is found and visited. """ root = node or self._root name = self.name ts = self.ts if isinstance(name, basestring): basename = name namet = (name,) else: # Must be a template function basename = name[0] namet = [basename] for x in name[1:]: if isinstance(x, Number): pass else: x = ts.canon(x) namet.append(x) namet = tuple(namet) if not isinstance(name, basestring): pattern = re.compile(r'(?: |::)'+basename+'.*>') for n in root.iterfind("Function[@name='{0}']".format(basename)): if not isinstance(name, basestring): # Must be a template function if n.attrib['file'] not in self.onlyin: continue nodename = n.attrib.get('demangled', '') if not pattern.search(nodename): continue nodet = self._visit_template_function(n) if nodet != namet: continue if n.attrib['file'] not in self.onlyin: msg = ("{0} autodescribing failed: found function in {1!r} but " "expected it in {2!r}.") msg = msg.format(name, node.attrib['file'], self.onlyin) raise RuntimeError(msg) self.visit_function(n) # # Clang Describers #
[docs]def clang_describe(filename, name, kind, includes=(), defines=('XDRESS',), undefines=(), extra_parser_args=(), ts=None, verbose=False, debug=False, builddir=None, onlyin=None, language='c++', clang_includes=()): """Use Clang to describe the class. Parameters ---------- filename : str The path to the file. name : str The name to describe. kind : str The kind of type to describe, valid flags are 'class', 'func', and 'var'. includes: list of str, optional The list of extra include directories to search for header files. defines: list of str, optional The list of extra macro definitions to apply. undefines: list of str, optional The list of extra macro undefinitions to apply. extra_parser_args : list of str, optional Further command line arguments to pass to the parser. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to diplay extra information while describing the class. debug : bool, optional Flag to enable/disable debug mode. Currently ignored. builddir : str, optional Ignored. Exists only for compatibility with gccxml_describe. onlyin : set of str, optional The paths to the files that the definition is allowed to exist in. language : str Valid language flag. Returns ------- desc : dict A dictionary describing the class which may be used to generate API bindings. """ tu = astparsers.clang_parse(filename, includes=includes, defines=defines, undefines=undefines, extra_parser_args=extra_parser_args, verbose=verbose, debug=debug, language=language, clang_includes=clang_includes) ts = ts or TypeSystem() if onlyin is None: onlyin = None if filename is None else frozenset([filename]) onlyin = clang_fix_onlyin(onlyin) if kind == 'class': cls = clang_find_class(tu, name, ts=ts, filename=filename, onlyin=onlyin) desc = clang_describe_class(cls) elif kind == 'func': fns = clang_find_function(tu, name, ts=ts, filename=filename, onlyin=onlyin) desc = clang_describe_functions(fns) elif kind == 'var': var = clang_find_var(tu, name, ts=ts, filename=filename, onlyin=onlyin) desc = clang_describe_var(var) else: raise ValueError('bad description kind {0}, name {1}'.format(kind,name)) linecache.clearcache() # Clean up results of clang_range_str return desc
[docs]def clang_fix_onlyin(onlyin): '''Make sure onlyin is a set and add ./path versions for each relative path''' if onlyin is not None: onlyin = set(onlyin) for f in tuple(onlyin): if not os.path.isabs(f): onlyin.add('./'+f) if os.sep != '/': # I'm not sure if clang lists paths with / or \ on windows onlyin.add(os.path.join('.',f)) onlyin = frozenset(onlyin) return onlyin
[docs]def clang_range_str(source_range): """Get the text present on a source range.""" start = source_range.start stop = source_range.end filename = start.file.name if filename != stop.file.name: msg = 'range spans multiple files: {0!r} & {1!r}' msg = msg.format(filename, stop.file.name) raise ValueError(msg) lines = [linecache.getline(filename, n) for n in range(start.line, stop.line+1)] lines[-1] = lines[-1][:stop.column-1] # stop slice must come first for lines[0] = lines[0][start.column-1:] # len(lines) == 1 s = "".join(lines) return s
[docs]def clang_find_scopes(tu, onlyin, namespace=None): """Find all 'toplevel' scopes, optionally restricting to a given namespace""" namespace_kind = CursorKind.NAMESPACE if namespace is None: def all_namespaces(node): for n in node.get_children(): if n.kind == namespace_kind: if onlyin is None or n.location.file.name in onlyin: yield n for c in all_namespaces(n): yield c return (tu.cursor,)+tuple(all_namespaces(tu.cursor)) else: scopes = [] for n in tu.cursor.get_children(): if n.kind == namespace_kind and n.spelling == namespace: if onlyin is None or n.location.file.name in onlyin: scopes.append(n) return scopes
[docs]def clang_find_decls(tu, name, kinds, onlyin, namespace=None): """Find all declarations of the given name and kind in the given scopes.""" scopes = clang_find_scopes(tu, onlyin, namespace=namespace) decls = [] for s in scopes[::-1]: for c in s.get_children(): if c.kind in kinds and c.spelling == name: if onlyin is None or c.location.file.name in onlyin: decls.append(c) return decls # TODO: This functionality belongs in TypeSystem
def canon_template_arg(ts, kind, arg): if kind == Arg.TYPE: return ts.canon(arg) return arg def clang_where(namespace, filename): where = '' if namespace is not None: where += " in namespace {0}".format(namespace) if filename is not None: where += " in file {0}".format(filename) return where
[docs]def clang_find_class(tu, name, ts, namespace=None, filename=None, onlyin=None): """Find the node for a given class in the given translation unit.""" templated = isinstance(name, tuple) if templated: basename = name[0] if name[-1] != 0: raise NotImplementedError('no predicate support in clang class description') args = name[1:-1] kinds = CursorKind.CLASS_TEMPLATE, else: basename = name kinds = CursorKind.CLASS_DECL, CursorKind.STRUCT_DECL decls = clang_find_decls(tu, basename, kinds=kinds, onlyin=onlyin, namespace=namespace) decls = frozenset(c.get_definition() or c for c in decls) # Use definitions if available if len(decls)==1: decl, = decls if not templated: # No templates, so we're done return decl else: # Search for the desired template specialization kinds = clang_template_param_kinds(decl) args = tuple(canon_template_arg(ts,k,a) for k,a in zip(kinds, args)) args = clang_expand_template_args(decl, args) for spec in decl.get_specializations(): if args == tuple(canon_template_arg(ts,k,a) for k,a in zip(kinds, clang_describe_template_args(spec))): return spec # Nothing found, time to complain where = clang_where(namespace, filename) if not decls: raise ValueError("class '{0}' could not be found{1}".format(name, where)) elif len(decls)>1: raise ValueError("class '{0}' found more than once ({2} times) {1}".format(name, len(decls), where)) else: raise ValueError("class '{0}' found, but specialization {1} not found{2}".format(basename, name, where))
[docs]def clang_find_function(tu, name, ts, namespace=None, filename=None, onlyin=None): """Find all nodes corresponding to a given function. If there is a separate declaration and definition, they will be returned as separate nodes, in the order given in the file.""" templated = isinstance(name, tuple) if templated: basename = name[0] args = name[1:] kinds = CursorKind.FUNCTION_TEMPLATE, else: basename = name kinds = CursorKind.FUNCTION_DECL, decls = clang_find_decls(tu, basename, kinds=kinds, onlyin=onlyin, namespace=namespace) if decls: if not templated: # No templates, so we're done return decls else: # Search for the desired function specialization decl, = decls # TODO: Support multiple decl case kinds = clang_template_param_kinds(decl) args = tuple(canon_template_arg(ts,k,a) for k,a in zip(kinds, args)) args = clang_expand_template_args(decl, args) for spec in decl.get_specializations(): if args == tuple(canon_template_arg(ts,k,a) for k,a in zip(kinds, clang_describe_template_args(spec))): return [spec] # Nothing found, time to complain where = clang_where(namespace, filename) if not decls: raise ValueError("function '{0}' could not be found{1}".format(name, where)) elif len(decls)>1: raise ValueError("function '{0}' found more than once ({2} times) {1}".format(name, len(decls), where)) else: raise ValueError("function '{0}' found, but specialization {1} not found{1}".format(basename, name, where))
[docs]def clang_find_var(tu, name, ts, namespace=None, filename=None, onlyin=None): """Find the node for a given var.""" assert isinstance(name, basestring) kinds = CursorKind.ENUM_DECL, decls = clang_find_decls(tu, name, kinds=kinds, onlyin=onlyin, namespace=namespace) decls = list(set(c.get_definition() or c for c in decls)) # Use definitions if available if len(decls)==1: return decls[0] # Nothing found, time to complain where = clang_where(namespace, filename) if not decls: raise ValueError("var '{0}' could not be found{1}".format(name, where)) else: raise ValueError("var '{0}' found more than once ({2} times) {1}".format(name, len(decls), where))
def clang_dump(node, indent=0, onlyin=None, file=sys.stdout): try: spelling = node.spelling except AttributeError: spelling = '<unknown-spelling>' s = '%*s%s %s'%(indent,'',node.kind.name,spelling) if node.kind == CursorKind.CXX_ACCESS_SPEC_DECL: s += ' '+node.access.name r = node.extent if r.start.line == r.end.line: try: s += ' : '+clang_range_str(r) except AttributeError: pass print(s,file=file) if onlyin is None: for c in node.get_children(): clang_dump(c,indent+2,file=file) else: for c in node.get_children(): f = c.extent.start.file if f and f.name in onlyin: clang_dump(c,indent+2,file=file) def clang_parent_namespace(node): if node.semantic_parent.kind == CursorKind.NAMESPACE: return node.semantic_parent.spelling # Otherwise, return none _operator_pattern = re.compile(r'^operator\W')
[docs]def clang_describe_class(cls): """Describe the class at the given clang AST node""" if cls.get_definition() is None: raise ValueError("can't describe undefined class '{0}' at {1}" .format(cls.spelling, clang_str_location(cls.location))) parents = [] attrs = {} methods = {} if cls.kind == CursorKind.CLASS_DECL: construct = 'class' elif cls.kind == CursorKind.STRUCT_DECL: construct = 'struct' else: raise ValueError('bad class kind {0}'.format(cls.kind.name)) typ = cls.spelling dest = '~'+typ templated = cls.has_template_args() if templated: cons = (typ,) + clang_describe_template_args(cls) dest = (dest,) + cons[1:] typ = cons + (0,) else: cons = typ for kid in cls.get_children(all_spec_bodies=1): kind = kid.kind if kind == CursorKind.CXX_BASE_SPECIFIER: parents.append(clang_describe_type(kid.type, kid.location)) elif kid.access == AccessKind.PUBLIC: if kind == CursorKind.CXX_METHOD: # TODO: For now, we ignore operators if not _operator_pattern.match(kid.spelling): sig, defaults = clang_describe_args(kid) methods[sig] = {'return': clang_describe_type(kid.result_type, kid.location), 'defaults': defaults} elif kind == CursorKind.CONSTRUCTOR: sig, defaults = clang_describe_args(kid) methods[(cons,)+sig[1:]] = {'return': None, 'defaults': defaults} elif kind == CursorKind.DESTRUCTOR: methods[(dest,)] = _none_return elif kind == CursorKind.FIELD_DECL: attrs[kid.spelling] = clang_describe_type(kid.type, kid.location) # Make sure defaulted methods are described if cls.has_default_constructor(): # Check if any user defined constructors act as a default constructor for sig, info in methods.items(): if sig[0] == cons: for _,default in info['defaults']: if default is None: break else: # All arguments have defaults, so no need to generate a default manually break else: methods[(cons,)] = _none_return if cls.has_simple_destructor(): methods[(dest,)] = _none_return # Put everything together return {'name': typ, 'type': typ, 'namespace': clang_parent_namespace(cls), 'parents': parents, 'attrs': attrs, 'methods': methods, 'construct': construct}
[docs]def clang_describe_var(var): """Describe the var at the given clang AST node""" if var.kind == CursorKind.ENUM_DECL: return {'name': var.spelling, 'namespace': clang_parent_namespace(var), 'type': clang_describe_enum(var)} else: raise NotImplementedError('var kind {0}: {1}'.format(var.kind, var.spelling))
def clang_str_location(loc): s = '%d:%d'%(loc.line,loc.column) return '%s:%s'%(loc.file.name,s) if loc.file else s
[docs]def clang_describe_functions(funcs): """Describe the function at the given clang AST nodes. If more than one node is given, we verify that they match and find argument names where we can.""" descs = tuple(map(clang_describe_function,funcs)) if len(descs)==1: return descs[0] def merge(d0,d1): """Merge two descriptions, checking that they describe the same function except possibly for argument name differences. If argument names conflict, the name from d0 is kept.""" def check(name,v0,v1): if v0 != v1: raise ValueError("{0} mismatch: {1} != {2}".format(name,v0,v1)) for s in 'name','namespace': check(s,d0[s],d1[s]) (args0,r0), = d0['signatures'].items() (args1,r1), = d1['signatures'].items() check('return type',r0,r1) check('name',args0[0],args1[0]) check('arity',len(args0)-1,len(args1)-1) args = [args0[0]] for i,(a0,a1) in enumerate(zip(args0[1:],args1[1:])): check('argument %d type'%i,a0[1],a1[1]) a = [a0[0] or a1[0],a0[1]] if len(a0)>2: if len(a1)>2: check('argument %d default'%i,a0[2],a1[2]) a.append(a0[2]) elif len(a1)>2: a.append(a1[2]) args.append(tuple(a)) return {'name':d0['name'],'namespace':d0['namespace'],'signatures':{tuple(args):r0}} try: return reduce(merge,descs) except ValueError: for j in xrange(len(funcs)): for i in xrange(j): try: merge(descs[i],descs[j]) except ValueError as e: pprint(descs[i]) pprint(descs[j]) raise ValueError("mismatch between declarations of '{0}' at {1} and {2}: {3}" .format(descs[0]['name'],clang_str_location(funcs[i].location), clang_str_location(funcs[j].location),e)) raise
[docs]def clang_describe_function(func): """Describe the function at the given clang AST node.""" assert func.kind == CursorKind.FUNCTION_DECL sig, defaults = clang_describe_args(func) signatures = {sig: {'return': clang_describe_type(func.result_type, func.location), 'defaults': defaults}} name = next(iter(signatures))[0] return {'name': name, 'namespace': clang_parent_namespace(func), 'signatures': signatures}
def clang_describe_args(func): if func.has_template_args(): descs = [(func.spelling,) + clang_describe_template_args(func)] else: descs = [func.spelling] defaults = [] for arg in func.get_arguments(): descs.append((arg.spelling, clang_describe_type(arg.type, arg.location))) default = arg.default_argument defaults.append(_none_arg if default is None else clang_describe_expression(default)) return tuple(descs), tuple(defaults)
[docs]def clang_describe_type(typ, loc): """Describe the type reference at the given cursor""" typ = typ.get_canonical() kind = typ.kind try: desc = _clang_base_types[kind] except KeyError: if kind == TypeKind.RECORD: decl = typ.get_declaration() cls = decl.spelling if cls == 'basic_string': desc = 'str' elif decl.has_template_args(): desc = (cls,) + clang_describe_template_args(decl) + (0,) else: desc = cls elif kind == TypeKind.LVALUEREFERENCE: desc = (clang_describe_type(typ.get_pointee(), loc), '&') elif kind == TypeKind.POINTER: p = typ.get_pointee() if p.kind == TypeKind.FUNCTIONPROTO: desc = ('function_pointer', tuple(('_{0}'.format(i),clang_describe_type(arg, loc)) for i,arg in enumerate(p.argument_types())), clang_describe_type(p.get_result(), loc)) else: desc = (clang_describe_type(p, loc), '*') elif kind == TypeKind.FUNCTIONPROTO: desc = ('function', tuple(('_{0}'.format(i),clang_describe_type(arg, loc)) for i,arg in enumerate(typ.argument_types())), clang_describe_type(typ.get_result(), loc)) elif kind == TypeKind.ENUM: return clang_describe_enum(typ.get_declaration()) else: raise NotImplementedError('type kind {0}: {1} at {2}' .format(typ.kind, typ.spelling, clang_str_location(loc))) if typ.is_const_qualified(): return (desc, 'const') else: return desc
def clang_describe_enum(decl): options = tuple((v.spelling, str(v.enum_value)) for v in decl.get_children()) return ('enum', decl.spelling, options) if 0: # TODO: Automatic support for default template arguments doesn't work yet def clang_template_arg_info(node): count = 0 defaults = [] kinds = CursorKind.TEMPLATE_TYPE_PARAMETER, CursorKind.TEMPLATE_NON_TYPE_PARAMETER for kid in node.get_children(): if kid.kind in kinds: count += 1 default = kid.default_argument if default is not None: defaults.append(default) else: break return count,tuple(defaults)
[docs]def clang_template_param_kinds(node): '''Find the Arg kind of each template argument of node''' kinds = [] for kid in node.get_children(): if kid.kind == CursorKind.TEMPLATE_TYPE_PARAMETER: kinds.append(Arg.TYPE) elif kid.kind == CursorKind.TEMPLATE_NON_TYPE_PARAMETER: typ = clang_describe_type(kid.type, kid.location) kinds.append(Arg.VAR if isinstance(typ, tuple) and typ[0]=='enum' else Arg.LIT) else: # Template arguments come first, so we're done break return kinds
[docs]def clang_describe_template_args(node): """TODO: Broken version handling defaults automatically:: _, defaults = clang_template_arg_info(node.specialized_template) args = [clang_describe_template_arg(a) for a in node.get_template_args()] for i in xrange(len(defaults)): if defaults[-1-i] == args[-1]: args.pop() return tuple(args) TODO: Needs a better docstring. """ loc = node.location args = tuple(clang_describe_template_arg(a, loc) for a in node.get_template_args()) if node.spelling in hack_template_args: return args[:len(hack_template_args[node.spelling])] else: return args
[docs]def clang_expand_template_args(node, args): """TODO: Broken version handling defaults automatically:: count,defaults = clang_template_arg_info(node) print('SP %s, COUNT %s, %s, %s'%(node.spelling,count,defaults,args)) if len(args) < count: return tuple(args) + defaults[(count-len(args)):] return tuple(args)+defaults[count-len(args)] TODO: Needs a better docstring. """ return args
[docs]def clang_describe_template_arg(arg, loc): '''Describe a template argument''' kind = arg.kind if kind == CursorKind.TYPE_TEMPLATE_ARG: return clang_describe_type(arg.type, loc) try: s = arg.spelling.strip() lit = c_literal(s) except: pass else: if kind == CursorKind.INTEGRAL_TEMPLATE_ARG: typ = clang_describe_type(arg.type, loc) if isinstance(typ, tuple) and typ[0]=='enum': # Convert integers to enum names lit = str(lit) for n,v in typ[2]: if v == lit: return n else: raise RuntimeError('template argument {0} is invalid, expected one of {1} at {2}' .format(lit, ', '.join('%s=%s'%(n,v) for n,v in typ[2]), clang_str_location(loc))) return lit if kind == CursorKind.EXPRESSION_TEMPLATE_ARG: exp, = arg.get_children() if exp.referenced: exp = exp.referenced if exp.kind == CursorKind.ENUM_CONSTANT_DECL: return s # Nothing worked, so bail raise NotImplementedError('template argument {0}, kind {1} at {2}' .format(s, kind.name, clang_str_location(loc)))
def clang_describe_expression(exp): # For now, we just use clang_range_str to pull the expression out of the file. # This is because clang doesn't seem to have any mechanism for printing expressions. s = clang_range_str(exp.extent) try: return Arg.LIT, c_literal(s) except: pass if exp.referenced: exp = exp.referenced if exp.kind == CursorKind.ENUM_CONSTANT_DECL: return Arg.VAR, s.strip() # Nothing worked, so bail kind = exp.kind.name raise NotImplementedError('unhandled expression "{0}" of kind {1} at {2}' .format(s, kind, clang_str_location(exp.location))) # # pycparser Describers # class PycparserBaseDescriber(PycparserNodeVisitor): _funckey = None def __init__(self, name, root, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The name to describe. root : pycparser AST The root of the abstract syntax tree. onlyin : str, optional Filename the class or struct described must live in. Prevents finding classes of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the class. """ super(PycparserBaseDescriber, self).__init__() self.desc = {'name': name, 'namespace': None} self.name = name self.ts = ts or TypeSystem() self.verbose = verbose self._root = root self._currfunc = [] # this must be a stack to handle nested functions self._currfuncsig = None self._currargkind = None self._currclass = [] # this must be a stack to handle nested classes self._level = -1 self._currtype = None self._currenum = None self._loading_bases = False self._basetypes = { 'char': 'char', 'signed char': 'char', 'unsigned char': 'uchar', 'short': 'int16', 'short int': 'int16', 'signed short': 'int16', 'signed short int': 'int16', 'int': 'int32', 'signed int': 'int32', 'long' : 'int32', 'long int' : 'int32', 'signed long' : 'int32', 'signed long int' : 'int32', 'long long' : 'int64', 'long long int' : 'int64', 'signed long long' : 'int64', 'signed long long int' : 'int64', 'unsigned short': 'uint16', 'unsigned short int': 'uint16', 'unsigned': 'uint32', 'unsigned int': 'uint32', 'unsigned long': 'uint32', 'unsigned long int': 'uint32', 'long unsigned int': 'uint32', 'unsigned long long' : 'uint64', 'unsigned long long int' : 'uint64', 'float': 'float32', 'double': 'float64', 'long double': 'float128', 'void': 'void', } def _pprint(self, node): if self.verbose: node.show() def load_basetypes(self): self._loading_bases = True for child_name, child in self._root.children(): if isinstance(child, pycparser.c_ast.Typedef): self._basetypes[child.name] = self.type(child) if self.verbose: print("Base type mapping = ") pprint(self._basetypes) self._loading_bases = False def visit_FuncDef(self, node): self._pprint(node) name = node.decl.name ftype = node.decl.type if name.startswith('_') or name in FORBIDDEN_NAMES: warn_forbidden_name(name, self.name) return self._currfunc.append(name) self._currfuncsig = [] self._currargkind = [] self._level += 1 children = () if ftype.args is None else ftype.args.children() for _, child in children: if isinstance(child, pycparser.c_ast.EllipsisParam): continue arg = (child.name, self.type(child)) if arg == (None, 'void'): # skip foo(void) cases, since no arg name is given continue if arg[0] in FORBIDDEN_NAMES: rename = arg[0] + '__' warn_forbidden_name(arg[0], self.name, rename) arg = (rename, arg[1]) self._currfuncsig.append(arg) self._currargkind.append(_none_arg) self._level -= 1 rtntype = self.type(ftype.type) funcname = self._currfunc.pop() if self._currfuncsig is None: self._currargkind = None return key = (funcname,) + tuple(self._currfuncsig) self.desc[self._funckey][key] = {'return': rtntype, 'defaults': tuple(self._currargkind)} self._currfuncsig = None self._currargkind = None def visit_IdentifierType(self, node): self._pprint(node) t = " ".join(node.names) t = self._basetypes.get(t, t) self._currtype = t def visit_Decl(self, node): self._pprint(node) if isinstance(node.type, pycparser.c_ast.FuncDecl): self.visit(node.type) key = (node.name,) + self._currtype[1] self.desc[self._funckey][key] = {'return': self._currtype[2], 'defaults': (_none_arg,) * len(self._currtype[1])} self._currtype = None else: self.visit(node.type) def visit_TypeDecl(self, node): self._pprint(node) if hasattr(node.type, 'children'): self.visit(node.type) def _enumint(self, value): base = 10 if value.startswith('0x') or value.startswith('-0x') or \ value.startswith('+0x'): base = 16 return int(value, base) def visit_Enumerator(self, node): self._pprint(node) if node.value is None: if len(self._currenum) == 0: value = 0 else: value = self._currenum[-1][-1] + 1 elif isinstance(node.value, pycparser.c_ast.Constant): value = self._enumint(node.value.value) elif isinstance(node.value, pycparser.c_ast.UnaryOp): if not isinstance(node.value.expr, pycparser.c_ast.Constant): raise ValueError("non-contant enum values not yet supported") value = self._enumint(node.value.op + node.value.expr.value) else: value = node.value self._currenum.append((node.name, value)) def visit_Enum(self, node): self._pprint(node) self._currenum = [] for _, child in node.children(): self.visit(child) self._currtype = ('enum', node.name, tuple(self._currenum)) self._currenum = None def visit_PtrDecl(self, node): self._pprint(node) self.visit(node.type) if self._currtype is not None and self._currtype[0] == 'function': self._currtype = ('function_pointer',) + self._currtype[1:] else: self._currtype = (self._currtype, '*') def visit_ArrayDecl(self, node): self._pprint(node) self.visit(node.type) predicate = '*' if node.dim is None else int(node.dim.value) self._currtype = (self._currtype, predicate) def visit_FuncDecl(self, node): self._pprint(node) args = [] params = () if node.args is None else node.args.params for i, arg in enumerate(params): if isinstance(arg, pycparser.c_ast.EllipsisParam): continue argname = arg.name or '_{0}'.format(i) argtype = self.type(arg, safe=True) args.append((argname, argtype)) rtntype = self.type(node.type, safe=True) self._currtype = ('function', tuple(args), rtntype) def visit_Struct(self, node): self._pprint(node) name = node.name if name is None: name = "<name-not-found>" self._currtype = name def visit_Typedef(self, node): self._pprint(node) self._currtype = None if not hasattr(node.type, 'children'): return self.visit(node.type) name = self._currtype self._currtype = None if name is None: return if name == "<name-not-found>": name = node.name self._currtype = name def type(self, node, safe=False): self._pprint(node) if safe: stashtype = self._currtype self._currtype = None self.visit(node) t = self._currtype self._currtype = None if safe: self._currtype = stashtype return t def visit_members(self, node): self._pprint(node) for _, child in node.children(): name = child.name if name.startswith('_') or name in FORBIDDEN_NAMES: warn_forbidden_name(name, self.name) continue t = self.type(child) if t == "<name-not-found>": msg = ("autodescribe: warning: anonymous struct members not " "yet supported, found {0}.{1}") print(msg.format(self.name, name)) continue elif t == ("<name-not-found>", '*'): msg = ("autodescribe: warning: anonymous struct pointer members " "not yet supported, found {0}.{1}") print(msg.format(self.name, name)) continue self.desc['attrs'][name] = t class PycparserVarDescriber(PycparserBaseDescriber): _type_error_msg = "{0} is a {1}, use {2} instead." def __init__(self, name, root, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The variable name. root : pycparser AST The root of the abstract syntax tree. onlyin : str, optional Filename the variable described must live in. Prevents finding variables of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the class. """ super(PycparserVarDescriber, self).__init__(name, root, onlyin=onlyin, ts=ts, verbose=verbose) def visit(self, node=None): """Visits the variable definition node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level variable node is found and visited. """ if node is None: self.load_basetypes() for child_name, child in self._root.children(): if getattr(child, 'name', None) == self.name: if isinstance(child, pycparser.c_ast.FuncDef): raise TypeError(self._type_error_msg.format( self.name, 'function', 'PycparserFuncDescriber')) if isinstance(child, pycparser.c_ast.Struct): raise TypeError(self._type_error_msg.format( self.name, 'struct', 'PycparserClassDescriber')) self.desc['type'] = self.type(child) break else: super(PycparserVarDescriber, self).visit(node) class PycparserFuncDescriber(PycparserBaseDescriber): _funckey = 'signatures' def __init__(self, name, root, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The function name. root : pycparser AST The root of the abstract syntax tree. onlyin : str, optional Filename the class or struct described must live in. Prevents finding classes of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the class. """ super(PycparserFuncDescriber, self).__init__(name, root, onlyin=onlyin, ts=ts, verbose=verbose) self.desc[self._funckey] = {} def visit(self, node=None): """Visits the function node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level class node is found and visited. """ if node is None: self.load_basetypes() for child_name, child in self._root.children(): if isinstance(child, pycparser.c_ast.FuncDef) and \ child.decl.name == self.name: self.visit(child) elif isinstance(child, pycparser.c_ast.Decl) and \ child.name == self.name: self.visit(child) else: super(PycparserFuncDescriber, self).visit(node) class PycparserClassDescriber(PycparserBaseDescriber): _funckey = 'methods' _constructvalue = 'struct' def __init__(self, name, root, onlyin=None, ts=None, verbose=False): """Parameters ------------- name : str The name to describe. root : pycparser AST The root of the abstract syntax tree. onlyin : str, optional Filename the class or struct described must live in. Prevents finding classes of the same name coming from other libraries. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to display extra information while visiting the class. Notes ----- It is impossible for C structs to have true member functions, only function pointers. """ super(PycparserClassDescriber, self).__init__(name, root, onlyin=onlyin, ts=ts, verbose=verbose) self.desc['attrs'] = {} self.desc[self._funckey] = {} self.desc['parents'] = [] self.desc['construct'] = self._constructvalue self.desc['type'] = ts.canon(name) def visit(self, node=None): """Visits the struct (class) node and all sub-nodes, generating the description dictionary as it goes. Parameters ---------- node : element tree node, optional The element tree node to start from. If this is None, then the top-level struct (class) node is found and visited. """ if node is None: self.load_basetypes() for child_name, child in self._root.children(): if isinstance(child, pycparser.c_ast.Typedef) and \ isinstance(child.type, pycparser.c_ast.TypeDecl) and \ isinstance(child.type.type, pycparser.c_ast.Struct): child = child.type.type if not isinstance(child, pycparser.c_ast.Struct): continue if child.name != self.name: continue self.visit_members(child) else: super(PycparserClassDescriber, self).visit(node) _pycparser_describers = { 'var': PycparserVarDescriber, 'func': PycparserFuncDescriber, 'class': PycparserClassDescriber, }
[docs]def pycparser_describe(filename, name, kind, includes=(), defines=('XDRESS',), undefines=(), extra_parser_args=(), ts=None, verbose=False, debug=False, builddir='build', onlyin=None, language='c', clang_includes=()): """Use pycparser to describe the fucntion or struct (class). Parameters ---------- filename : str The path to the file. name : str The name to describe. kind : str The kind of type to describe, valid flags are 'class', 'func', and 'var'. includes: list of str, optional The list of extra include directories to search for header files. defines: list of str, optional The list of extra macro definitions to apply. undefines: list of str, optional The list of extra macro undefinitions to apply. extra_parser_args : list of str, optional Further command line arguments to pass to the parser. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to diplay extra information while describing the class. debug : bool, optional Flag to enable/disable debug mode. builddir : str, optional Location of -- often temporary -- build files. onlyin : set of str The paths to the files that the definition is allowed to exist in. language : str Must be 'c'. clang_includes : ignored Returns ------- desc : dict A dictionary describing the class which may be used to generate API bindings. """ assert language=='c' root = astparsers.pycparser_parse(filename, includes=includes, defines=defines, undefines=undefines, extra_parser_args=extra_parser_args, verbose=verbose, debug=debug, builddir=builddir) if onlyin is None: onlyin = set([filename]) describer = _pycparser_describers[kind](name, root, onlyin=onlyin, ts=ts, verbose=verbose) describer.visit() return describer.desc # # General utilities #
def _make_includer(filenames, builddir, language, verbose=False): """Creates a source file made up of #include pre-processor statements for all of the files in filenames. Returns the path to the newly made file. """ newfile = "" newnames = [] for filename in filenames: newnames.append(filename.replace(os.path.sep, '_')) newfile += '#include "{0}"\n'.format(filename) newnames = "-".join(newnames) if len(newnames) > 250: # this is needed to prevent 'IOError: [Errno 36] File name too long' newnames = md5(newnames).hexdigest() newname = os.path.join(builddir, newnames + '.' + _lang_exts[language]) newoverwrite(newfile, newname, verbose=verbose) return newname _describers = { 'clang': clang_describe, 'gccxml': gccxml_describe, 'pycparser': pycparser_describe, }
[docs]def describe(filename, name=None, kind='class', includes=(), defines=('XDRESS',), undefines=(), extra_parser_args=(), parsers='gccxml', ts=None, verbose=False, debug=False, builddir='build', language='c++', clang_includes=()): """Automatically describes an API element in a file. This is the main entry point. Parameters ---------- filename : str or container of strs The path to the file or a list of file paths. If this is a list to many files, a temporary file will be created that #includes all of the files in this list in order. This temporary file is the one which will be parsed. name : str The name to describe. kind : str, optional The kind of type to describe, valid flags are 'class', 'func', and 'var'. includes: list of str, optional The list of extra include directories to search for header files. defines: list of str, optional The list of extra macro definitions to apply. undefines: list of str, optional The list of extra macro undefinitions to apply. extra_parser_args : list of str, optional Further command line arguments to pass to the parser. parsers : str, list, or dict, optional The parser / AST to use to use for the file. Currently 'clang', 'gccxml', and 'pycparser' are supported, though others may be implemented in the future. If this is a string, then this parser is used. If this is a list, this specifies the parser order to use based on availability. If this is a dictionary, it specifies the order to use parser based on language, i.e. ``{'c' ['pycparser', 'gccxml'], 'c++': ['gccxml', 'pycparser']}``. ts : TypeSystem, optional A type system instance. verbose : bool, optional Flag to diplay extra information while describing the class. debug : bool, optional Flag to enable/disable debug mode. builddir : str, optional Location of -- often temporary -- build files. language : str Valid language flag. clang_includes : list of str, optional clang-specific include paths. Returns ------- desc : dict A dictionary describing the class which may be used to generate API bindings. """ if isinstance(filename, basestring): onlyin = set([filename]) else: onlyin = set(filename) filename = filename[0] if len(filename) == 0 \ else _make_includer(filename, builddir, language, verbose=verbose) if name is None: name = os.path.split(filename)[-1].rsplit('.', 1)[0].capitalize() parser = astparsers.pick_parser(language, parsers) describer = _describers[parser] desc = describer(filename, name, kind, includes=includes, defines=defines, undefines=undefines, extra_parser_args=extra_parser_args, ts=ts, verbose=verbose, debug=debug, builddir=builddir, onlyin=onlyin, language=language, clang_includes=clang_includes) return desc # # Plugin #
[docs]class XDressPlugin(astparsers.ParserPlugin): """This plugin creates automatic description dictionaries of all souce and target files.""" def __init__(self): super(XDressPlugin, self).__init__() self.pysrcenv = {}
[docs] def defaultrc(self): """This plugin adds the env dictionary to the rc.""" rc = RunControl() rc._update(super(XDressPlugin, self).defaultrc) # target enviroment made up of module dicts made up of descriptions rc.env = {} return rc
[docs] def rcdocs(self): """This plugin adds the env dictionary to the rc.""" docs = {} docs.update(super(XDressPlugin, self).rcdocs) docs['env'] = "The target environment computed by the autodescriber." return docs
[docs] def setup(self, rc): """Expands variables, functions, and classes in the rc based on copying src filenames to tar filename.""" super(XDressPlugin, self).setup(rc) for i, var in enumerate(rc.variables): rc.variables[i] = ensure_apiname(var) for i, fnc in enumerate(rc.functions): rc.functions[i] = ensure_apiname(fnc) for i, cls in enumerate(rc.classes): rc.classes[i] = cls = ensure_apiname(cls) if not isinstance(cls.srcname, basestring) and cls.srcname[-1] is not 0: # ensure the predicate is a scalar for template specializations rc.classes[i] = cls = cls._replace(srcname=tuple(cls.srcname) + (0,)) if not isinstance(cls.tarname, basestring) and cls.tarname[-1] is not 0: # ensure the predicate is a scalar for template specializations rc.classes[i] = cls = cls._replace(tarname=tuple(cls.tarname) + (0,)) if 'make_dtypes' not in rc: rc.make_dtypes = False self.register_classes(rc)
def execute(self, rc): print("autodescribe: scraping C/C++ APIs from source") self.load_sidecars(rc) self.compute_classes(rc) self.compute_functions(rc) self.compute_variables(rc) def report_debug(self, rc): super(XDressPlugin, self).report_debug(rc) # Helper methods below
[docs] def register_classes(self, rc): """Registers classes with the type system. This can and should be done trying to describe the class.""" ts = rc.ts for i, cls in enumerate(rc.classes): print("autodescribe: registering {0}".format(cls.srcname)) fnames = extra_filenames(cls) ts.register_classname(cls.srcname, rc.package, fnames['pxd_base'], fnames['cpppxd_base'], make_dtypes=rc.make_dtypes) if cls.srcname != cls.tarname: ts.register_classname(cls.tarname, rc.package, fnames['pxd_base'], fnames['cpppxd_base'], cpp_classname=cls.srcname, make_dtypes=rc.make_dtypes)
[docs] def load_pysrcmod(self, sidecar, rc): """Loads a module dictionary from a sidecar file into the pysrcenv cache.""" if sidecar in self.pysrcenv: return if os.path.isfile(sidecar): glbs = globals() locs = {} exec_file(sidecar, glbs, locs) if 'mod' not in locs: pymod = {} elif callable(locs['mod']): pymod = eval('mod()', glbs, locs) else: pymod = locs['mod'] if 'ts' in locs: rc.ts.update(locs['ts']) elif 'type_system' in locs: rc.ts.update(locs['type_system']) else: pymod = {} self.pysrcenv[sidecar] = pymod
[docs] def load_sidecars(self, rc): """Loads all sidecar files.""" sidecars = set() for x in rc.variables: sidecars.update(x.sidecars) for x in rc.functions: sidecars.update(x.sidecars) for x in rc.classes: sidecars.update(x.sidecars) for sidecar in sidecars: self.load_pysrcmod(sidecar, rc)
[docs] def compute_desc(self, name, kind, rc): """Returns a description dictionary for a class or function implemented in a source file and bound into a target file. Parameters ---------- name : apiname API element name to describe. kind : str The kind of type to describe, valid flags are 'class', 'func', and 'var'. rc : xdress.utils.RunControl Run contoler for this xdress execution. Returns ------- desc : dict Description dictionary. """ cache = rc._cache if cache.isvalid(name, kind): srcdesc = cache[name, kind] else: srcdesc = describe(name.srcfiles, name=name.srcname, kind=kind, includes=rc.includes, defines=rc.defines, undefines=rc.undefines, extra_parser_args=rc.extra_parser_args, parsers=rc.parsers, ts=rc.ts, verbose=rc.verbose, debug=rc.debug, builddir=rc.builddir, language=name.language, clang_includes=rc.clang_includes) srcdesc['name'] = dict(zip(name._fields, name)) cache[name, kind] = srcdesc descs = [srcdesc] descs += [self.pysrcenv[s].get(name.srcname, {}) for s in name.sidecars] descs.append({'extra': extra_filenames(name)}) desc = merge_descriptions(descs) return desc
_extrajoinkeys = ['pxd_header', 'pxd_footer', 'pyx_header', 'pyx_footer', 'cpppxd_header', 'cpppxd_footer']
[docs] def adddesc2env(self, desc, env, name): """Adds a description to environment.""" # Add to target environment # docstrings overwrite, extras accrete docs = [self.pysrcenv[s].get(name.srcname, {}).get('docstring', '') \ for s in name.sidecars] docs = "\n\n".join([d for d in docs if len(d) > 0]) mod = {name.tarname: desc, 'docstring': docs, 'srcpxd_filename': desc['extra']['srcpxd_filename'], 'pxd_filename': desc['extra']['pxd_filename'], 'pyx_filename': desc['extra']['pyx_filename'], 'language': name.language, } tarbase = name.tarbase extrajoinkeys = self._extrajoinkeys if tarbase not in env: env[tarbase] = mod env[tarbase]["name"] = tarbase env[tarbase]['extra'] = modextra = dict(zip(extrajoinkeys, ['']*len(extrajoinkeys))) else: #env[tarbase].update(mod) env[tarbase][name.tarname] = desc modextra = env[tarbase]['extra'] for sidecar in name.sidecars: pyextra = self.pysrcenv[sidecar].get(name.srcname, {}).get('extra', {}) for key in extrajoinkeys: modextra[key] += pyextra.get(key, '')
[docs] def compute_variables(self, rc): """Computes variables descriptions and loads them into the environment.""" ts = rc.ts env = rc.env cache = rc._cache for i, var in enumerate(rc.variables): print("autodescribe: describing {0}".format(var.srcname)) desc = self.compute_desc(var, 'var', rc) if rc.verbose: pprint(desc) cache.dump() self.adddesc2env(desc, env, var) ts.register_variable_namespace(desc['name']['srcname'], desc['namespace'], desc['type']) if 0 == i%rc.clear_parser_cache_period: astparsers.clearmemo()
[docs] def compute_functions(self, rc): """Computes function descriptions and loads them into the environment.""" env = rc.env cache = rc._cache for i, fnc in enumerate(rc.functions): print("autodescribe: describing {0}".format(fnc.srcname)) desc = self.compute_desc(fnc, 'func', rc) if rc.verbose: pprint(desc) cache.dump() self.adddesc2env(desc, env, fnc) if 0 == i%rc.clear_parser_cache_period: astparsers.clearmemo()
[docs] def compute_classes(self, rc): """Computes class descriptions and loads them into the environment.""" # compute all class descriptions first cache = rc._cache env = rc.env # target environment, not source one for i, cls in enumerate(rc.classes): print("autodescribe: describing {0}".format(cls.srcname)) desc = self.compute_desc(cls, 'class', rc) cache.dump() if rc.verbose: pprint(desc) self.adddesc2env(desc, env, cls) if 0 == i%rc.clear_parser_cache_period: astparsers.clearmemo()