Python Project Layout: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(43 intermediate revisions by the same user not shown)
Line 4: Line 4:


=Overview=
=Overview=
This article documents a typical Python [[#Project_Layout|project layout]], and the [[#Project_Bootstrap|step-by-step project bootstrap]] procedure.
=Project Layout=
A typical Python project layout, which allows for code written in other programming languages as well, is similar to:
A typical Python project layout, which allows for code written in other programming languages as well, is similar to:
<font size=-1>
<font size=-1>
Line 9: Line 12:
  ├─[[#Add_.gitignore| .gitignore]]
  ├─[[#Add_.gitignore| .gitignore]]
  ├─ [[#Expand_requirements.txt|requirements.txt]]
  ├─ [[#Expand_requirements.txt|requirements.txt]]
├─ [[Pyproject.toml#Overview|pyproject.toml]]
  ├─ [[#Add_the_run_script|run]]
  ├─ [[#Add_the_run_script|run]]
  ├─ [[#initialize|initialize]]
  ├─ [[#initialize|initialize]]
  ├─ src
  ├─ src
  │  └─ mypackage
  │  └─ somepkg
  │      ├─  [[Python_Language_Modularization#init_.py|__init__.py]]
  │      ├─  [[Python_Language_Modularization#init_.py|__init__.py]]
  │      ├─  [[#Add_main_.py|__main__.py]]
  │      ├─  [[#Add_main_.py|__main__.py]]
  │      ├─ mypackage-part-1.py
  │      ├─ some_module_1.py
  │      ├─ mypackage-part-2.py
  │      ├─ some_module_2.py
  │      ├─ ...
  │      ├─ ...
  │      └─ VERSION
  │      └─ VERSION
  ├─ tests
  ├─ tests
  │  └─ mypackage
  │  └─ somepkg
  │      ├─   
  │      ├─   
  │           
  │           
  └─ venv <font color=teal># created automatically upon virtual environment initialization</font>
  ├─ .venv <font color=teal># created automatically upon virtual environment initialization</font>
      ├─ bin
│    ├─ bin
      ...
│    ...
│         
└─ <span id='dist'></span>dist <font color=teal># created automatically upon [[Publishing_a_Python_Distribution_Package_in_a_Repository#Create_the_Distribution_Archives|publishing]] the project</font>
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl
</font>
</font>


Start with an empty <code>requirements.txt</code> file, it can be expanded incrementally.
Start with an empty <code>requirements.txt</code> file, it can be expanded incrementally.


=Initialize the Virtual Environment=
=Project Bootstrap=
 
==Initialize the Virtual Environment==
Initialize the virtual environment following the manual procedure described here:
Initialize the virtual environment following the manual procedure described here:
{{Internal|Python_Virtual_Environment#Virtual_Environment_Creation|Python Virtual Environment &#124; Virtual Environment Creation}}
{{Internal|Python_Virtual_Environment#Virtual_Environment_Creation|Python Virtual Environment &#124; Virtual Environment Creation}}
Update <code>pip</code> and, if <code>requirements.txt</code> has declared dependencies, install them:
Update <code>pip</code> and, if <code>requirements.txt</code> has declared dependencies, install them:
{{Internal|Python_Virtual_Environment#Virtual_Environments_and_pip|Python Virtual Environment &#124; Virtual Environments and <tt>pip</tt>}}
{{Internal|Python_Virtual_Environment#Virtual_Environments_and_pip|Python Virtual Environment &#124; Virtual Environments and <tt>pip</tt>}}
=Add <tt>.gitignore</tt>=
 
==Add <tt>.gitignore</tt>==
<syntaxhighlight lang='text'>
<syntaxhighlight lang='text'>
venv/
venv/
Line 42: Line 53:
.idea/
.idea/
</syntaxhighlight>
</syntaxhighlight>
==Setup your Package and Modules==
Python code is organized in [[Python_Language_Modularization#Modules|modules]], which are grouped together in [[Python_Language_Modularization#Package_(Import_Package)|packages]]. A package may contain multiple modules.
To start with, you can pick the name of the package, and create a directory with that name under <code>src</code>. The package and module names have restrictions, documented here: {{Internal|Python_Language_Modularization#Module_Name|Python Module Names}}
{{Internal|Python_Language_Modularization#Package_Name|Python Package Names}}


=Add <tt>__main__.py</tt>=
<font size=-1.5>
Add the initial <code>__main__.py</code>:
<font size=-1>
  .
  .
  ├─ src
  ├─ src
  │  └─ mypackage
  │  └─ somepkg
  │      └─ __main__.py  
  │      ├─ some_module_1.py  
│      ├─ some_module_2.py
│      ...
  ...
  ...
</font>
</font>


<syntaxhighlight lang='py'>
The modules from under <code>src/somepkg</code> will be available to the interpreter as long as the <code>.../src/somepkg</code> directory is listed in the <code>PYTHONPATH</code> environment variable. For more details see: {{Internal|Python_Language_Modularization#Locating_Module_Files_-_Module_Search_Path|Locating Module Files}}
def main():
    print('.')


To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:


main()
<syntaxhighlight lang='py'>
import somepkg.some_module_1 as some_module_1
...
</syntaxhighlight>
More details in: {{Internal|Python_Language_Modularization#Importing|Importing}}


</syntaxhighlight>
==Add <tt>__main__.py</tt>==
Also see: {{Internal|Python_Language_Modularization#main_.py|Modularization &#124; <tt>main_.py</tt>}}
Add the initial <code>__main__.py</code>:
<font size=-1.5>
.
├─ src
│  └─ somepkg
│      ├─ __main__.py
│      ...
...
</font>
For an example of idiomatic <code>__main__.py</code> and more details on <code>__main__.py</code> see: {{Internal|Python_Language_Modularization#main_.py|Modularization &#124; <tt>__main__.py</tt>}}


=Add the run script=
==Add the run script==
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash
#!/usr/bin/env bash
# shellcheck disable=SC2086


PYTHONPATH="$(dirname $0)/src"
PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
export PYTHONPATH
"$(dirname $0)/venv/bin/python" -m mypackage "$@"
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@"
</syntaxhighlight>
</syntaxhighlight>


Line 77: Line 104:
</syntaxhighlight>
</syntaxhighlight>


=Expand <tt>requirements.txt</tt>=
==Expand <tt>requirements.txt</tt>==
Add your dependency to <code>requirements.txt</code>:
Add your dependency to <code>requirements.txt</code>:
<font size=-1>
<font size=-1>
Line 87: Line 114:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
venv/bin/pip install -r requirements.txt
.venv/bin/pip install -r requirements.txt
</syntaxhighlight>
</syntaxhighlight>
Also see: {{Internal|Pip#requirements.txt_File|<tt>pip</tt> &#124; <tt>requirements.txt</tt>}}
Also see: {{Internal|Pip#requirements.txt_File|<tt>pip</tt> &#124; <tt>requirements.txt</tt>}}
==<span id='initialize'></span><tt>initialize</tt>: Initialization and Dependency Maintenance Script==
===<span id='initialize'></span><tt>initialize</tt>: Initialization and Dependency Maintenance Script===
<font color=darkkhaki>Do I really need this?</font>
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash
#!/usr/bin/env bash
Line 111: Line 139:
   trust_root.sh "$(dirname $0)/venv"
   trust_root.sh "$(dirname $0)/venv"
fi
fi
</syntaxhighlight>
Alternative (needs refactoring):
<syntaxhighlight lang='bash'>
# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}
</syntaxhighlight>
</syntaxhighlight>


Line 116: Line 168:
A project set up this way will be compatible with [[PyCharm_Concepts#Overview|PyCharm]]. To complete the PyCharm setup:
A project set up this way will be compatible with [[PyCharm_Concepts#Overview|PyCharm]]. To complete the PyCharm setup:


* Set the Python interpreter. Use the interpreter from <code>venv/bin/python</code>.
* Set the Python interpreter. Use the interpreter from <code>.venv/bin/python</code>. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
* Designate the <code>src</code> as the source root directory: Right Click → Mark Directory as → Sources Root
* Designate the <code>src</code> as the source root directory: Right Click → Mark Directory as → Sources Root
* Designate the <code>tests</code> as the test sources root directory: Right Click → Mark Directory as → Test Sources Root
* Designate the <code>tests</code> as the test sources root directory: Right Click → Mark Directory as → Test Sources Root
Line 138: Line 190:
</syntaxhighlight>
</syntaxhighlight>


=TO CONTINUE=
=A Build-System Independent Format for Source Trees=
 
==Setting a Manual Project==
 
 
===Bash Wrapper===
 
<font color=darkkhaki>TODO: reconcile with [[Calling_Python_from_bash#Running_a_Python_Program_with_a_Bash_Wrapper|Calling Python from bash &#124; Running_a_Python_Program_with_a_Bash_Wrapper]]</font>.
 
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash
 
function main() {
  manage_venv "$@"
  PYTHONPATH="$(dirname $0)/src"
  export PYTHONPATH
  "$(dirname $0)/venv/bin/python3" -m gpm "$@"
}
 
#
# If the virtual environment does not exist, create it based on requirements.txt.
# If it does exist, recreate if '--force-init' is present among options.
#
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}
 
main "$@"
</syntaxhighlight>
 
==TODO==
 
<font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font>
<font color=darkkhaki>Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/</font>

Latest revision as of 21:31, 19 September 2024

Internal

Overview

This article documents a typical Python project layout, and the step-by-step project bootstrap procedure.

Project Layout

A typical Python project layout, which allows for code written in other programming languages as well, is similar to:

.
├─ .gitignore
├─ requirements.txt
├─ pyproject.toml
├─ run
├─ initialize
├─ src
│   └─ somepkg
│       ├─  __init__.py
│       ├─  __main__.py
│       ├─ some_module_1.py
│       ├─ some_module_2.py
│       ├─ ...
│       └─ VERSION
├─ tests
│   └─ somepkg
│       ├─  
│           
├─ .venv # created automatically upon virtual environment initialization
│    ├─ bin
│    ...
│           
└─ dist # created automatically upon publishing the project
    ├─ somepkg-0.1.0.tar.gz
    └─ somepkg-0.1.0-py3-none-any.whl

Start with an empty requirements.txt file, it can be expanded incrementally.

Project Bootstrap

Initialize the Virtual Environment

Initialize the virtual environment following the manual procedure described here:

Python Virtual Environment | Virtual Environment Creation

Update pip and, if requirements.txt has declared dependencies, install them:

Python Virtual Environment | Virtual Environments and pip

Add .gitignore

venv/
__pycache__/
idea/
.idea/

Setup your Package and Modules

Python code is organized in modules, which are grouped together in packages. A package may contain multiple modules.

To start with, you can pick the name of the package, and create a directory with that name under src. The package and module names have restrictions, documented here:

Python Module Names
Python Package Names

.
├─ src
│   └─ somepkg
│       ├─ some_module_1.py 
│       ├─ some_module_2.py 
│      ...
...

The modules from under src/somepkg will be available to the interpreter as long as the .../src/somepkg directory is listed in the PYTHONPATH environment variable. For more details see:

Locating Module Files

To import features of those modules in a source file, within the project or outside, make sure the import path specifies the package name and the module name. The import path matches the file system path where the slashes are replaced by dots:

import somepkg.some_module_1 as some_module_1
...

More details in:

Importing

Add __main__.py

Add the initial __main__.py:

.
├─ src
│   └─ somepkg
│       ├─ __main__.py 
│       ...
...

For an example of idiomatic __main__.py and more details on __main__.py see:

Modularization | __main__.py

Add the run script

#!/usr/bin/env bash

PYTHONPATH="$(dirname "$0")/src"
export PYTHONPATH
"$(dirname "$0")/.venv/bin/python" -m somepkg "$@"
chmod a+x ./run

Expand requirements.txt

Add your dependency to requirements.txt:

# ...
PyGithub == 1.58.2

and run:

.venv/bin/pip install -r requirements.txt

Also see:

pip | requirements.txt

initialize: Initialization and Dependency Maintenance Script

Do I really need this?

#!/usr/bin/env bash
# shellcheck disable=SC2086

python -m venv "$(dirname $0)/venv"
"$(dirname $0)/venv/bin/python" -m pip install --upgrade pip
"$(dirname $0)/venv/bin/pip" install -r requirements.txt

if ! trust_root.sh --help >/dev/null 2>&1; then
  cat 1>&2 <<EOF

trust_root.sh not found on your system.

Install it with:

 [...]
EOF
else
  trust_root.sh "$(dirname $0)/venv"
fi

Alternative (needs refactoring):

# If the virtual environment does not exist, create it based on requirements.txt. If it does exist, recreate if '--force-init' is present among options.
function manage_venv() {
  force_init=false
  while [[ -n $1 ]]; do
    if [[ $1 == '--force-init' ]]; then
      force_init=true
    fi
    shift
  done
  if [[ -d $(dirname $0)/venv ]]; then
    if ${force_init}; then
      rm -rf "$(dirname $0)/venv"
    else
      return 0
    fi
  fi
  echo "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || { echo "python3 not in PATH" 1>&2; exit 1; }
  python3 -m venv "$(dirname $0)/venv"
  "$(dirname $0)/venv/bin/pip" install -r "$(dirname $0)/requirements.txt"
}

Set the PyCharm Project

A project set up this way will be compatible with PyCharm. To complete the PyCharm setup:

  • Set the Python interpreter. Use the interpreter from .venv/bin/python. To verify that it was imported correctly go to Settings → Project: ... → Python Interpreter → Python Interpreter.
  • Designate the src as the source root directory: Right Click → Mark Directory as → Sources Root
  • Designate the tests as the test sources root directory: Right Click → Mark Directory as → Test Sources Root

Setup PyCharm Debugging

To setup main script debugging:

Edit Configurations → The + sign → Python

Name: "__main__.py arg1 arg2"

Script path: Click on the folder icon and navigate. The final result is similar to: /Users/ovidiu/projects/pygithub/src/pygithub_experiment/__main__.py.

Parameters: ...

Environment variables:

PYTHONUNBUFFERED=1;GITHUB_PAT=...;MY_ENV_VAR_1=val1

A Build-System Independent Format for Source Trees

Process this: PEP 517 – A build-system independent format for source trees https://peps.python.org/pep-0517/