Python Module Internal Representation and Introspection

Internal

• Python Introspection
• Python Language Modularization

The module Class

All module and package instances are represented internally as instances of the module class.

Checking whether an Object Instance is a Module

import types
import mymodule

assert isinstance(mymodule, types.ModuleType)
assert not isinstance('something', types.ModuleType)

Attributes

All object instances declared in the module (variables, functions, classes, etc.) become attributes of the module instance and they are accessible with inspect.getmembers().

inspect.getmembers()

Additionally, the following special attributes are present:

__name__

The __name__ special variable contains the name of the module, when it was imported, as string.

import mymodule
assert mymodule.__name__ == 'mymodule'

When the module is executed directly with python mymodule.py, it is never imported, so __name__ is set to the "__main__" string.

__file__

Once imported, the file associated with the module can be determined using the module object's __file__ attribute, as string:

import mymodule
[...]
print(mymodule.__file__)

The directory portion of __file__ should be one of the directories in sys.path.

__doc__

The content of the module docstring, if declared, otherwise None.

__package__

An empty string for a top-level module, the name of the package for a package or for a module that was loaded as part of a package.

__path__

The __path__ attribute exists only for module instances that represent packages, not for those instances that represent ordinary modules.

__path__ contains a list of the package root directories, where the component modules, subpackages, __init__.py and __main__.py live.

To check whether __path__ exists, use hasattr().

Dynamic Module Tree Traversal and Class Loading

def find_class(module_or_package: types.ModuleType, predicate) -> type:
    """
    If given a module, look for a class whose name, as string, satisfies the predicate and return the first match.
    If given a package, recursively load the modules while descending in the package structure, look for a class whose name, as string, satisfies
    the predicate and return the first match.
    :param module_or_package: the module or the package instance. Must be imported by the calling layer.
    :param predicate: a function that examines the class name, as string, and returns True if the class is acceptable, False otherwise.
    :return: the class instance, or None if no such class exists
    """
    if not isinstance(module_or_package, types.ModuleType):
        raise TypeError(f'invalid module: {module_or_package}')
    if not isinstance(predicate, types.FunctionType):
        raise TypeError(f'invalid predicate: {predicate}')
    if not hasattr(module_or_package, '__path__'):
        # module
        module = module_or_package
        for name, value in inspect.getmembers(module):
            if isinstance(value, type):
                # a class, apply the predicate
                if predicate(name):
                    return value
    else:
        # package
        package = module_or_package
        paths = package.__path__
        for p in paths:
            path = Path(p)
            if not path.is_dir():
                raise IllegalStateError(f'package {package.__name__} path not a directory: {p}')
            file_names = []
            dir_names = []
            for f in path.iterdir():
                # module or sub-package. Import in the local namespace and proceed recursively.
                if f.name.startswith('__'):
                    continue
                name = f.name.replace('.py', '')
                if f.is_file():
                    file_names.append(name)
                else:
                    dir_names.append(name)
            # Process modules first, to favor classes declared closest from the top
            all_names = file_names
            all_names.extend(dir_names)
            for name in all_names:
                module_or_package = importlib.import_module(f'.{name}', package.__name__)
                cls = find_class(module_or_package, predicate)
                if cls:
                    return cls

Usage:

def predicate(class_name: str) -> bool:
    return class_name == 'SomeClass'

import test_package
cls = find_class(test_package, predicate)