Python Language Modularization

External

• https://realpython.com/python-modules-packages

Internal

• Python Language

Overview

Python code is organized in units like standalone programs, modules and packages. When a module or a package is published, people refer to it as a library. In this context, the term library is simply a generic term for a bunch of code that was designed with the aim of being reused by many applications. It provides some generic functionality that can be used by specific applications.

Standalone Program

A standalone program consists of one or more files of code that is read by the Python interpreter and executed. A typical way to interact with a Python program is command line arguments. For more details on handling command line arguments, see:

Command Line Argument Processing in Python

Python Script

A script is a module whose aim is to be executed. It has the same meaning as "program", standalone program, or "application", but it is usually used to describe simple and small program. It contains a stored set of instructions that can be handed over to the Python interpreter:

python3 ./my-script.py

Python scripts have .py extensions.

The python code can be specified in-line with a here-doc:

python3 <<EOF
print('hello')
print('hello2')
EOF

The same approach can be taken when Python code needs to be executed from within a bash script, for more details see:

Calling Python from bash | Inline Python Code

Module

A module is an organizational unit of Python code. It consists of one file. The module can be imported inside another Python program or executed on its own. The module can define variables, functions and classes. If intended to run on its own, the module will also include runnable code. The file name consists of the module name with the suffix .py appended. Modules are loaded into Python by the process of importing, where the code in one module is made available to Python code in another module. The files of Python modules and packages are managed by a Python package manager like pip, Conda, Pipenv and Poetry.

Modules have a namespace containing arbitrary Python objects.

Module Name

Naming conventions are documented here: https://www.python.org/dev/peps/pep-0008/#id36: Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. The name of the module cannot contain dashes ('-'). The name of the directory the module is stored in can contain dashes.

Package

A package is Python code stored into multiple files, organized into a file hierarchy. A package contains multiple modules, each stored in its own file. The structural element that tells Python a file hierarchy is a regular package is a file named __init__.py This file can be empty, but Python needs it to tread the directory containing it as a package. The package directory can recursively contain sub-packages. Packages allow for a hierarchical structuring of the module namespace using dot notation. In the same way that modules avoid collisions between global variable names, packages avoid collision between module names. For example, the urllib package contains several modules: urllib.request, urllib.error, etc. Modules and packages are managed by a Python package manager like pip, Conda, Pipenv and Poetry.

Regular Package

A traditional package, such as a directory containing an __init__.py file.

__init__.py

https://docs.python.org/3/reference/import.html#regular-packages

__main__.py

The file contains the code of the "main" module, which will be imported automatically when the package is imported.

Namespace Package

A PEP 420 package which serves only as a container for subpackages. Namespace packages may have no physical representation, and have no __init__.py file.

Package Metadata

Name: pulumi
Version: 2.11.2
Summary: Pulumi's Python SDK
Home-page: https://github.com/pulumi/pulumi
Author:
Author-email:
License: Apache 2.0
Location: /Users/ovidiu/Library/Python/3.8/lib/python/site-packages
Requires: dill, grpcio, protobuf
Required-by: pulumi-aws, pulumi-kubernetes, pulumi-random, pulumi-tls

Python Standard Library

https://docs.python.org/3/library/

If the published module or package is is available in the Python Module Index, it is referred to as a "standard library". The standard library modules are included with Python when it is installed.

Notable Modules and Packages

These modules are available in the Python Standard Library, hence are referred to as "standard libraries".

Python Module Index

https://docs.python.org/3/py-modindex.html

Python Package Index PyPI

https://pypi.org

It can be searched with pip search.

Organizatorium

• Each installation of Python may have different modules installed. Python determines the path to its modules by examining the location of the python3 executable.
• Clarify the difference between importing a module and a package.
• The import system: https://docs.python.org/3/reference/import.html
• Technically, a package is a Python module with a __path__ attribute.
• Not clear yet how to define working packages. I've defined a package with an __init__.py and a __main__.py with a function defined inside __main__.py, and after import I get:

ImportError: cannot import name 'my_test_function' from 'my_package' (/Users/ovidiu/.../main/python/my_package/__init__.py)

• Packages can be run as if they were scripts if the package provides a top-level script __main__.py.
  • https://docs.python.org/3/library/__main__.html
  • Understand setup.py - it defines the entry point.