Python Project Layout

Internal

• Python
• Calling Python from bash

Overview

This article documents a typical Python project layout, and the step-by-step project bootstrap procedure.

Project Layout

A typical Python project layout, which allows for code written in other programming languages as well, is similar to:

.
├─ .gitignore
├─ requirements.txt
├─ pyproject.toml
├─ run
├─ initialize
├─ src
│   └─ somepkg
│       ├─  __init__.py
│       ├─  __main__.py
│       ├─ somemodule_1.py
│       ├─ somemodule_2.py
│       ├─ ...
│       └─ VERSION
├─ tests
│   └─ somepkg
│       ├─  
│           
├─ .venv # created automatically upon virtual environment initialization
│    ├─ bin
│    ...
│           
└─ dist # created automatically upon publishing the project
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl

Start with an empty requirements.txt file, it can be expanded incrementally.

Project Bootstrap

Initialize the Virtual Environment

Initialize the virtual environment following the manual procedure described here:

Python Virtual Environment | Virtual Environment Creation

Update pip and, if requirements.txt has declared dependencies, install them:

Python Virtual Environment | Virtual Environments and pip

Add .gitignore

venv/
__pycache__/
idea/
.idea/

Pick your Package and Modules Names

Python code is organized in modules, which are grouped together in packages. A package may contain multiple modules.

To start with, you can pick the name of the package, and create a directory with that name under src. The package name has restrictions, documented here:

Python Package Names

Add __main__.py

Add the initial __main__.py:

.
├─ src
│   └─ mypackage
│       └─ __main__.py 
...

For an example of idiomatic __main__.py and more details on __main__.py see:

Modularization | __main__.py

Add the run script

#!/usr/bin/env bash

PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m mypackage "$@"

chmod a+x ./run

Expand requirements.txt

Add your dependency to requirements.txt:

# ...
PyGithub == 1.58.2

and run:

.venv/bin/pip install -r requirements.txt

Also see:

pip | requirements.txt

initialize: Initialization and Dependency Maintenance Script

Do I really need this?

#!/usr/bin/env bash
# shellcheck disable=SC2086

python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt

if ! trust_root.sh --help >/dev/null 2>&1; then
  cat 1>&2 <<EOF

trust_root.sh not found on your system.

Install it with:

 [...]
EOF
else
  trust_root.sh "$(dirname $0)/venv"
fi

Alternative (needs refactoring):

# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}

Set the PyCharm Project

A project set up this way will be compatible with PyCharm. To complete the PyCharm setup:

• Set the Python interpreter. Use the interpreter from .venv/bin/python. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
• Designate the src as the source root directory: Right Click → Mark Directory as → Sources Root
• Designate the tests as the test sources root directory: Right Click → Mark Directory as → Test Sources Root

Setup PyCharm Debugging

To setup main script debugging:

Edit Configurations → The + sign → Python

Name: "__main__.py arg1 arg2"

Script path: Click on the folder icon and navigate. The final result is similar to: /Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py.

Parameters: ...

Environment variables:

PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1

A Build-System Independent Format for Source Trees

Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/