Python Project Layout: Difference between revisions
(89 intermediate revisions by the same user not shown) | |||
Line 4: | Line 4: | ||
=Overview= | =Overview= | ||
This article documents a typical Python [[#Project_Layout|project layout]], and the [[#Project_Bootstrap|step-by-step project bootstrap]] procedure. | |||
=Project Layout= | |||
A typical Python project layout, which allows for code written in other programming languages as well, is similar to: | A typical Python project layout, which allows for code written in other programming languages as well, is similar to: | ||
<font size=-1> | <font size=-1> | ||
. | . | ||
├─ .gitignore | ├─[[#Add_.gitignore| .gitignore]] | ||
├─ requirements.txt | ├─ [[#Expand_requirements.txt|requirements.txt]] | ||
├─ run | ├─ [[Pyproject.toml#Overview|pyproject.toml]] | ||
├─ [[#Add_the_run_script|run]] | |||
├─ [[#initialize|initialize]] | |||
├─ src | ├─ src | ||
│ └─ | │ └─ somepkg | ||
│ ├─ __init__.py | │ ├─ [[Python_Language_Modularization#init_.py|__init__.py]] | ||
│ ├─ __main__.py | │ ├─ [[#Add_main_.py|__main__.py]] | ||
│ ├─ some_module_1.py | |||
│ ├─ some_module_2.py | |||
│ ├─ ... | |||
│ └─ VERSION | │ └─ VERSION | ||
├─ tests | ├─ tests | ||
│ └─ | │ └─ somepkg | ||
│ ├─ | │ ├─ | ||
│ | │ | ||
├─ .venv <font color=teal># created automatically upon virtual environment initialization</font> | |||
│ ├─ bin | |||
│ ... | |||
│ | |||
└─ <span id='dist'></span>dist <font color=teal># created automatically upon [[Publishing_a_Python_Distribution_Package_in_a_Repository#Create_the_Distribution_Archives|publishing]] the project</font> | |||
├─ somepkg-0.1.0.tar.gz | |||
└─ somepkg-0.1.0-py3-none-any.whl | |||
</font> | </font> | ||
Start with an empty <code>requirements.txt</code> file, it can be expanded incrementally. | Start with an empty <code>requirements.txt</code> file, it can be expanded incrementally. | ||
=Initialize the Virtual Environment= | |||
=Project Bootstrap= | |||
==Initialize the Virtual Environment== | |||
Initialize the virtual environment following the manual procedure described here: | Initialize the virtual environment following the manual procedure described here: | ||
{{Internal|Python_Virtual_Environment#Virtual_Environment_Creation|Python Virtual Environment | Virtual Environment Creation}} | {{Internal|Python_Virtual_Environment#Virtual_Environment_Creation|Python Virtual Environment | Virtual Environment Creation}} | ||
Update <code>pip</code> and, if <code>requirements.txt</code> has declared dependencies, install them: | |||
{{Internal|Python_Virtual_Environment#Virtual_Environments_and_pip|Python Virtual Environment | Virtual Environments and <tt>pip</tt>}} | |||
==Add <tt>.gitignore</tt>== | |||
<syntaxhighlight lang='text'> | |||
venv/ | |||
__pycache__/ | |||
idea/ | |||
.idea/ | |||
</syntaxhighlight> | |||
==Setup your Package and Modules== | |||
Python code is organized in [[Python_Language_Modularization#Modules|modules]], which are grouped together in [[Python_Language_Modularization#Package_(Import_Package)|packages]]. A package may contain multiple modules. | |||
To start with, you can pick the name of the package, and create a directory with that name under <code>src</code>. The package and module names have restrictions, documented here: {{Internal|Python_Language_Modularization#Module_Name|Python Module Names}} | |||
{{Internal|Python_Language_Modularization#Package_Name|Python Package Names}} | |||
<font size=-1.5> | |||
. | |||
├─ src | |||
│ └─ somepkg | |||
│ ├─ some_module_1.py | |||
│ ├─ some_module_2.py | |||
│ ... | |||
... | |||
</font> | |||
The modules from under <code>src/somepkg</code> will be available to the interpreter as long as the <code>.../src/somepkg</code> directory is listed in the <code>PYTHONPATH</code> environment variable. For more details see: {{Internal|Python_Language_Modularization#Locating_Module_Files_-_Module_Search_Path|Locating Module Files}} | |||
To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots: | |||
<syntaxhighlight lang='py'> | |||
import somepkg.some_module_1 as some_module_1 | |||
... | |||
</syntaxhighlight> | |||
More details in: {{Internal|Python_Language_Modularization#Importing|Importing}} | |||
= | ==Add <tt>__main__.py</tt>== | ||
Add the initial <code>__main__.py</code>: | |||
<font size=-1.5> | |||
. | |||
├─ src | |||
│ └─ somepkg | |||
│ ├─ __main__.py | |||
│ ... | |||
... | |||
</font> | |||
For an example of idiomatic <code>__main__.py</code> and more details on <code>__main__.py</code> see: {{Internal|Python_Language_Modularization#main_.py|Modularization | <tt>__main__.py</tt>}} | |||
==Add the run script== | |||
<syntaxhighlight lang='bash'> | |||
#!/usr/bin/env bash | |||
PYTHONPATH="$(dirname "$0")/src" | |||
export PYTHONPATH | |||
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@" | |||
</syntaxhighlight> | |||
<syntaxhighlight lang='bash'> | |||
chmod a+x ./run | |||
</syntaxhighlight> | |||
==Expand <tt>requirements.txt</tt>== | |||
Add your dependency to <code>requirements.txt</code>: | |||
<font size=-1> | <font size=-1> | ||
# ... | |||
PyGithub == 1.58.2 | |||
</font> | </font> | ||
and run: | |||
<syntaxhighlight lang='bash'> | |||
<syntaxhighlight lang=' | .venv/bin/pip install -r requirements.txt | ||
venv/ | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Also see: {{Internal|Pip#requirements.txt_File|<tt>pip</tt> | <tt>requirements.txt</tt>}} | |||
===<span id='initialize'></span><tt>initialize</tt>: Initialization and Dependency Maintenance Script=== | |||
<font color=darkkhaki>Do I really need this?</font> | |||
<syntaxhighlight lang='bash'> | |||
#!/usr/bin/env bash | |||
# shellcheck disable=SC2086 | |||
python -m venv "$(dirname $0)/venv" | |||
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip | |||
"$(dirname $0)/venv/bin/pip" install -r requirements.txt | |||
if ! trust_root.sh --help >/dev/null 2>&1; then | |||
cat 1>&2 <<EOF | |||
trust_root.sh not found on your system. | |||
Install it with: | |||
[...] | |||
# If the virtual environment does not exist, create it based on requirements.txt. | EOF | ||
else | |||
trust_root.sh "$(dirname $0)/venv" | |||
fi | |||
</syntaxhighlight> | |||
Alternative (needs refactoring): | |||
<syntaxhighlight lang='bash'> | |||
# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options. | |||
function manage_venv() { | function manage_venv() { | ||
force_init=false | force_init=false | ||
Line 95: | Line 163: | ||
"$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt" | "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt" | ||
} | } | ||
</syntaxhighlight> | |||
=Set the PyCharm Project= | |||
A project set up this way will be compatible with [[PyCharm_Concepts#Overview|PyCharm]]. To complete the PyCharm setup: | |||
* Set the Python interpreter. Use the interpreter from <code>.venv/bin/python</code>. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter. | |||
* Designate the <code>src</code> as the source root directory: Right Click → Mark Directory as → Sources Root | |||
* Designate the <code>tests</code> as the test sources root directory: Right Click → Mark Directory as → Test Sources Root | |||
==Setup PyCharm Debugging== | |||
To setup main script debugging: | |||
Edit Configurations → The + sign → Python | |||
Name: "__main__.py arg1 arg2" | |||
Script path: Click on the folder icon and navigate. The final result is similar to: <code>/Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py</code>. | |||
Parameters: ... | |||
Environment variables: | |||
<syntaxhighlight lang='text'> | |||
PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1 | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=A Build-System Independent Format for Source Trees= | |||
<font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font> | <font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font> |
Latest revision as of 21:31, 19 September 2024
Internal
Overview
This article documents a typical Python project layout, and the step-by-step project bootstrap procedure.
Project Layout
A typical Python project layout, which allows for code written in other programming languages as well, is similar to:
. ├─ .gitignore ├─ requirements.txt ├─ pyproject.toml ├─ run ├─ initialize ├─ src │ └─ somepkg │ ├─ __init__.py │ ├─ __main__.py │ ├─ some_module_1.py │ ├─ some_module_2.py │ ├─ ... │ └─ VERSION ├─ tests │ └─ somepkg │ ├─ │ ├─ .venv # created automatically upon virtual environment initialization │ ├─ bin │ ... │ └─ dist # created automatically upon publishing the project ├─ somepkg-0.1.0.tar.gz └─ somepkg-0.1.0-py3-none-any.whl
Start with an empty requirements.txt
file, it can be expanded incrementally.
Project Bootstrap
Initialize the Virtual Environment
Initialize the virtual environment following the manual procedure described here:
Update pip
and, if requirements.txt
has declared dependencies, install them:
Add .gitignore
venv/
__pycache__/
idea/
.idea/
Setup your Package and Modules
Python code is organized in modules, which are grouped together in packages. A package may contain multiple modules.
To start with, you can pick the name of the package, and create a directory with that name under src
. The package and module names have restrictions, documented here:
. ├─ src │ └─ somepkg │ ├─ some_module_1.py │ ├─ some_module_2.py │ ... ...
The modules from under src/somepkg
will be available to the interpreter as long as the .../src/somepkg
directory is listed in the PYTHONPATH
environment variable. For more details see:
To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:
import somepkg.some_module_1 as some_module_1
...
More details in:
Add __main__.py
Add the initial __main__.py
:
. ├─ src │ └─ somepkg │ ├─ __main__.py │ ... ...
For an example of idiomatic __main__.py
and more details on __main__.py
see:
Add the run script
#!/usr/bin/env bash
PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@"
chmod a+x ./run
Expand requirements.txt
Add your dependency to requirements.txt
:
# ... PyGithub == 1.58.2
and run:
.venv/bin/pip install -r requirements.txt
Also see:
initialize: Initialization and Dependency Maintenance Script
Do I really need this?
#!/usr/bin/env bash
# shellcheck disable=SC2086
python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt
if ! trust_root.sh --help >/dev/null 2>&1; then
cat 1>&2 <<EOF
trust_root.sh not found on your system.
Install it with:
[...]
EOF
else
trust_root.sh "$(dirname $0)/venv"
fi
Alternative (needs refactoring):
# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
force_init=false
while [[ -n $1 ]]; do
if [[ $1 == '--force-init' ]]; then
force_init=true
fi
shift
done
if [[ -d $(dirname $0)/venv ]]; then
if ${force_init}; then
rm -rf "$(dirname $0)/venv"
else
return 0
fi
fi
echo "initializing venv ..."
python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
python3 -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}
Set the PyCharm Project
A project set up this way will be compatible with PyCharm. To complete the PyCharm setup:
- Set the Python interpreter. Use the interpreter from
.venv/bin/python
. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter. - Designate the
src
as the source root directory: Right Click → Mark Directory as → Sources Root - Designate the
tests
as the test sources root directory: Right Click → Mark Directory as → Test Sources Root
Setup PyCharm Debugging
To setup main script debugging:
Edit Configurations → The + sign → Python
Name: "__main__.py arg1 arg2"
Script path: Click on the folder icon and navigate. The final result is similar to: /Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py
.
Parameters: ...
Environment variables:
PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1
A Build-System Independent Format for Source Trees
Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/