Python Project Layout

Internal

• Python
• Calling Python from bash

Overview

This article documents a typical Python project layout, and the step-by-step project bootstrap procedure.

Project Layout

A typical Python project layout, which allows for code written in other programming languages as well, is similar to:

.
├─ .gitignore
├─ requirements.txt
├─ pyproject.toml
├─ run
├─ initialize
├─ src
│   └─ somepkg
│       ├─  __init__.py
│       ├─  __main__.py
│       ├─ some_module_1.py
│       ├─ some_module_2.py
│       ├─ ...
│       └─ VERSION
├─ tests
│   └─ somepkg
│       ├─  
│           
├─ .venv # created automatically upon virtual environment initialization
│    ├─ bin
│    ...
│           
└─ dist # created automatically upon publishing the project
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl

Start with an empty requirements.txt file, it can be expanded incrementally.

Project Bootstrap

Initialize the Virtual Environment

Initialize the virtual environment following the manual procedure described here:

Python Virtual Environment | Virtual Environment Creation

Update pip and, if requirements.txt has declared dependencies, install them:

Python Virtual Environment | Virtual Environments and pip

Add .gitignore

venv/
__pycache__/
idea/
.idea/

Setup your Package and Modules

Python code is organized in modules, which are grouped together in packages. A package may contain multiple modules.

To start with, you can pick the name of the package, and create a directory with that name under src. The package and module names have restrictions, documented here:

Python Module Names

Python Package Names

.
├─ src
│   └─ somepkg
│       ├─ some_module_1.py 
│       ├─ some_module_2.py 
│      ...
...

The modules from under src/somepkg will be available to the interpreter as long as the .../src/somepkg directory is listed in the PYTHONPATH environment variable. For more details see:

Locating Module Files

To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:

import somepkg.some_module_1 as some_module_1
...

More details in:

Importing

Add __main__.py

Add the initial __main__.py:

.
├─ src
│   └─ mypackage
│       └─ __main__.py 
...

For an example of idiomatic __main__.py and more details on __main__.py see:

Modularization | __main__.py

Add the run script

#!/usr/bin/env bash

PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m mypackage "$@"

chmod a+x ./run

Expand requirements.txt

Add your dependency to requirements.txt:

# ...
PyGithub == 1.58.2

and run:

.venv/bin/pip install -r requirements.txt

Also see:

pip | requirements.txt

initialize: Initialization and Dependency Maintenance Script

Do I really need this?

#!/usr/bin/env bash
# shellcheck disable=SC2086

python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt

if ! trust_root.sh --help >/dev/null 2>&1; then
  cat 1>&2 <<EOF

trust_root.sh not found on your system.

Install it with:

 [...]
EOF
else
  trust_root.sh "$(dirname $0)/venv"
fi

Alternative (needs refactoring):

# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}

Set the PyCharm Project

A project set up this way will be compatible with PyCharm. To complete the PyCharm setup:

• Set the Python interpreter. Use the interpreter from .venv/bin/python. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
• Designate the src as the source root directory: Right Click → Mark Directory as → Sources Root
• Designate the tests as the test sources root directory: Right Click → Mark Directory as → Test Sources Root

Setup PyCharm Debugging

To setup main script debugging:

Edit Configurations → The + sign → Python

Name: "__main__.py arg1 arg2"

Script path: Click on the folder icon and navigate. The final result is similar to: /Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py.

Parameters: ...

Environment variables:

PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1

A Build-System Independent Format for Source Trees

Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/