Python Project Layout: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(93 intermediate revisions by the same user not shown)
Line 4: Line 4:


=Overview=
=Overview=
This article documents a typical Python [[#Project_Layout|project layout]], and the [[#Project_Bootstrap|step-by-step project bootstrap]] procedure.


=Project Layout=
A typical Python project layout, which allows for code written in other programming languages as well, is similar to:
A typical Python project layout, which allows for code written in other programming languages as well, is similar to:
<font size=-1>
<font size=-1>
  .
  .
  ├─ .gitignore
  ├─[[#Add_.gitignore| .gitignore]]
  ├─ requirements.txt
  ├─ [[#Expand_requirements.txt|requirements.txt]]
  ├─ run
├─ [[Pyproject.toml#Overview|pyproject.toml]]
  ├─ [[#Add_the_run_script|run]]
├─ [[#initialize|initialize]]
  ├─ src
  ├─ src
  │  └─ mypackage
  │  └─ somepkg
  │      ├─  __init__.py
  │      ├─  [[Python_Language_Modularization#init_.py|__init__.py]]
  │      ├─  __main__.py  
  │      ├─  [[#Add_main_.py|__main__.py]]
│      ├─ some_module_1.py
│      ├─ some_module_2.py
│      ├─ ...
  │      └─ VERSION
  │      └─ VERSION
  ├─ tests
  ├─ tests
  │  └─ mypackage
  │  └─ somepkg
  │      ├─   
  │      ├─   
  │           
  │           
  └─ venv <font color=teal># created automatically upon virtual environment initialization</font>
  ├─ .venv <font color=teal># created automatically upon virtual environment initialization</font>
      ├─ bin
│    ├─ bin
 
│    ...
 
│         
└─ <span id='dist'></span>dist <font color=teal># created automatically upon [[Publishing_a_Python_Distribution_Package_in_a_Repository#Create_the_Distribution_Archives|publishing]] the project</font>
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl
</font>
</font>


=Setting a Manual Project=
Start with an empty <code>requirements.txt</code> file, it can be expanded incrementally.


A project set up this way will be compatible with [[PyCharm_Concepts#Overview|PyCharm]]. Also see: {{Internal|PyCharm_Operations#Create_a_Simple.2C_Standalone_Project|Simple PyCharm Project}}
=Project Bootstrap=


Add the dependencies to <code>requirements.txt</code>. For more details, see: {{Internal|Pip#requirements.txt|requirements.txt}}
==Initialize the Virtual Environment==
Initialize the virtual environment following the manual procedure described here:
{{Internal|Python_Virtual_Environment#Virtual_Environment_Creation|Python Virtual Environment &#124; Virtual Environment Creation}}
Update <code>pip</code> and, if <code>requirements.txt</code> has declared dependencies, install them:
{{Internal|Python_Virtual_Environment#Virtual_Environments_and_pip|Python Virtual Environment &#124; Virtual Environments and <tt>pip</tt>}}


Initialize the virtual environment:
==Add <tt>.gitignore</tt>==
<syntaxhighlight lang='text'>
venv/
__pycache__/
idea/
.idea/
</syntaxhighlight>
==Setup your Package and Modules==
Python code is organized in [[Python_Language_Modularization#Modules|modules]], which are grouped together in [[Python_Language_Modularization#Package_(Import_Package)|packages]]. A package may contain multiple modules.


<font size=-1>
To start with, you can pick the name of the package, and create a directory with that name under <code>src</code>. The package and module names have restrictions, documented here: {{Internal|Python_Language_Modularization#Module_Name|Python Module Names}}
  cd ${PROJECT_HOME}
{{Internal|Python_Language_Modularization#Package_Name|Python Package Names}}
  python3 -m venv venv
 
  venv/bin/pip install -r ./requirements.txt
<font size=-1.5>
  .
├─ src
│  └─ somepkg
│      ├─ some_module_1.py
│      ├─ some_module_2.py
  │      ...
  ...
</font>
 
The modules from under <code>src/somepkg</code> will be available to the interpreter as long as the <code>.../src/somepkg</code> directory is listed in the <code>PYTHONPATH</code> environment variable. For more details see: {{Internal|Python_Language_Modularization#Locating_Module_Files_-_Module_Search_Path|Locating Module Files}}
 
To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:
 
<syntaxhighlight lang='py'>
import somepkg.some_module_1 as some_module_1
...
</syntaxhighlight>
More details in: {{Internal|Python_Language_Modularization#Importing|Importing}}
 
==Add <tt>__main__.py</tt>==
Add the initial <code>__main__.py</code>:
<font size=-1.5>
.
├─ src
│  └─ somepkg
│      ├─ __main__.py
│      ...
...
</font>
</font>
For an example of idiomatic <code>__main__.py</code> and more details on <code>__main__.py</code> see: {{Internal|Python_Language_Modularization#main_.py|Modularization &#124; <tt>__main__.py</tt>}}
==Add the run script==
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash


For more details on virtual environments, see: {{Internal|Python_Language#Virtual_Environment|Virtual Environment}}
PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@"
</syntaxhighlight>


==<tt>.gitignore</tt>==
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='text'>
chmod a+x ./run
venv/
__pycache__/
idea/
</syntaxhighlight>
</syntaxhighlight>


==Bash Wrapper==
==Expand <tt>requirements.txt</tt>==
Add your dependency to <code>requirements.txt</code>:
<font size=-1>
# ...
PyGithub == 1.58.2
</font>


<font color=darkkhaki>TODO: reconcile with [[Calling_Python_from_bash#Running_a_Python_Program_with_a_Bash_Wrapper|Calling Python from bash &#124; Running_a_Python_Program_with_a_Bash_Wrapper]]</font>.
and run:


<syntaxhighlight lang='bash'>
.venv/bin/pip install -r requirements.txt
</syntaxhighlight>
Also see: {{Internal|Pip#requirements.txt_File|<tt>pip</tt> &#124; <tt>requirements.txt</tt>}}
===<span id='initialize'></span><tt>initialize</tt>: Initialization and Dependency Maintenance Script===
<font color=darkkhaki>Do I really need this?</font>
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash
#!/usr/bin/env bash
# shellcheck disable=SC2086
python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt
if ! trust_root.sh --help >/dev/null 2>&1; then
  cat 1>&2 <<EOF
trust_root.sh not found on your system.


function main() {
Install it with:
  manage_venv "$@"
  PYTHONPATH="$(dirname $0)/src"
  export PYTHONPATH
  "$(dirname $0)/venv/bin/python3" -m gpm "$@"
}


#
[...]
# If the virtual environment does not exist, create it based on requirements.txt.
EOF
# If it does exist, recreate if '--force-init' is present among options.
else
#
  trust_root.sh "$(dirname $0)/venv"
fi
</syntaxhighlight>
Alternative (needs refactoring):
<syntaxhighlight lang='bash'>
# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
function manage_venv() {
   force_init=false
   force_init=false
Line 87: Line 163:
   "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
   "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}
}
</syntaxhighlight>
=Set the PyCharm Project=
A project set up this way will be compatible with [[PyCharm_Concepts#Overview|PyCharm]]. To complete the PyCharm setup:
* Set the Python interpreter. Use the interpreter from <code>.venv/bin/python</code>. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
* Designate the <code>src</code> as the source root directory: Right Click → Mark Directory as → Sources Root
* Designate the <code>tests</code> as the test sources root directory: Right Click → Mark Directory as → Test Sources Root
==Setup PyCharm Debugging==
To setup main script debugging:
Edit Configurations  → The + sign  → Python
Name: "__main__.py arg1 arg2"
Script path: Click on the folder icon and navigate. The final result is similar to: <code>/Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py</code>.


main "$@"
Parameters: ...
</syntaxhighlight>


==<tt>__main__.py</tt>==
Environment variables:  
<syntaxhighlight lang='py'>
def main():
    print('.x')


main()
<syntaxhighlight lang='text'>
PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1
</syntaxhighlight>
</syntaxhighlight>
=TODO=


=A Build-System Independent Format for Source Trees=
<font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font>
<font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font>

Latest revision as of 21:31, 19 September 2024

Internal

Overview

This article documents a typical Python project layout, and the step-by-step project bootstrap procedure.

Project Layout

A typical Python project layout, which allows for code written in other programming languages as well, is similar to:

.
├─ .gitignore
├─ requirements.txt
├─ pyproject.toml
├─ run
├─ initialize
├─ src
│   └─ somepkg
│       ├─  __init__.py
│       ├─  __main__.py
│       ├─ some_module_1.py
│       ├─ some_module_2.py
│       ├─ ...
│       └─ VERSION
├─ tests
│   └─ somepkg
│       ├─  
│           
├─ .venv # created automatically upon virtual environment initialization
│    ├─ bin
│    ...
│           
└─ dist # created automatically upon publishing the project
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl

Start with an empty requirements.txt file, it can be expanded incrementally.

Project Bootstrap

Initialize the Virtual Environment

Initialize the virtual environment following the manual procedure described here:

Python Virtual Environment | Virtual Environment Creation

Update pip and, if requirements.txt has declared dependencies, install them:

Python Virtual Environment | Virtual Environments and pip

Add .gitignore

venv/
__pycache__/
idea/
.idea/

Setup your Package and Modules

Python code is organized in modules, which are grouped together in packages. A package may contain multiple modules.

To start with, you can pick the name of the package, and create a directory with that name under src. The package and module names have restrictions, documented here:

Python Module Names
Python Package Names

.
├─ src
│   └─ somepkg
│       ├─ some_module_1.py 
│       ├─ some_module_2.py 
│      ...
...

The modules from under src/somepkg will be available to the interpreter as long as the .../src/somepkg directory is listed in the PYTHONPATH environment variable. For more details see:

Locating Module Files

To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:

import somepkg.some_module_1 as some_module_1
...

More details in:

Importing

Add __main__.py

Add the initial __main__.py:

.
├─ src
│   └─ somepkg
│       ├─ __main__.py 
│       ...
...

For an example of idiomatic __main__.py and more details on __main__.py see:

Modularization | __main__.py

Add the run script

#!/usr/bin/env bash

PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@"
chmod a+x ./run

Expand requirements.txt

Add your dependency to requirements.txt:

# ...
PyGithub == 1.58.2

and run:

.venv/bin/pip install -r requirements.txt

Also see:

pip | requirements.txt

initialize: Initialization and Dependency Maintenance Script

Do I really need this?

#!/usr/bin/env bash
# shellcheck disable=SC2086

python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt

if ! trust_root.sh --help >/dev/null 2>&1; then
  cat 1>&2 <<EOF

trust_root.sh not found on your system.

Install it with:

 [...]
EOF
else
  trust_root.sh "$(dirname $0)/venv"
fi

Alternative (needs refactoring):

# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}

Set the PyCharm Project

A project set up this way will be compatible with PyCharm. To complete the PyCharm setup:

  • Set the Python interpreter. Use the interpreter from .venv/bin/python. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
  • Designate the src as the source root directory: Right Click → Mark Directory as → Sources Root
  • Designate the tests as the test sources root directory: Right Click → Mark Directory as → Test Sources Root

Setup PyCharm Debugging

To setup main script debugging:

Edit Configurations → The + sign → Python

Name: "__main__.py arg1 arg2"

Script path: Click on the folder icon and navigate. The final result is similar to: /Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py.

Parameters: ...

Environment variables:

PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1

A Build-System Independent Format for Source Trees

Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/