Calling Python from bash: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(34 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Internal=
=Internal=
* [[Bash#Subjects|bash]]
* [[Python_Language_Modularization#Python_Script|Python Modularization | Python Script]]
* [[Python_Language_Modularization#Python_Script|Python Modularization | Python Script]]
* [[Bash#Subjects|bash]]
* [[Python_Virtual_Environment#Overview|Python Virtual Environment]]


=Overview=
=Overview=
=Inline Python Code=
This article summarizes a series of aspects related to invoking Python program from bash.
Use bash  [[Bash_Input/Output#Here-Docs|here-doc]]:
 
<syntaxhighlight lang='bash'>
=Running a Python Program with a Bash Wrapper=
  python3 <<EOF
 
print('blah')
Check out these links for the meaning of [[Python_Language_Modularization#Standalone_Program|standalone Python program]] and [[Python_Language_Modularization#Python_Script|Python script]].
EOF
 
</syntaxhighlight>
An example of a <code>run</code> script for a simple Python project is available here:
 
{{Internal|Python_Project_Layout#Add_the_run_script|Python Project Layout &#124; The <tt>run</tt> script}}
 
<font color=darkkhaki>
 
 
Refactor all this and reconcile with: {{Internal|Python_Project_Layout#Overview|Python Project Layout}}
 
 
Problems to be solved:


Also see: {{Internal|Python_Language_Modularization#Python_Script|Python Script}}
1. The problem of Python binaries. We either rely on the correct version of the Python to be present on the executing image, or we encase it it in a virtual environment. There we need to be aware of the problem of the symbolic links and binary formats.


=Code in External Script=
2. The problem of package dependencies and requirements.txt
==External Script==


==External Module==
3. The layout of the source code: sources and tests, or package dependencies in a Python package repository.
Given an external module <code>my_module.py</code> with the following content:
<syntaxhighlight lang='python'>
def my_function(arg1, arg2, arg3):
    print('this is my_function(' + arg1 + ", " + arg2 + ", " + arg3 + ")")
</syntaxhighlight>
the module can be called generically from a bash script as follows:
<syntaxhighlight lang='bash'>
#!/usr/bin/env bash


function call-python() {
</font>
  local python_interpreter=$1
  local module_path=$2
  local function_name=$3
  [[ -z ${python_interpreter} ]] && fail "'python_interpreter' not provided"
  [[ -z ${module_path} ]] && fail "'module_path' not provided"
  [[ -z ${function_name} ]] && fail "'function_name' not provided"
  shift 3
  local args
  for i in "${@}"; do
    [[ -z ${args} ]] && args="'${i}'" || args="${args}, '${i}'"
  done
  local module_name
  module_name=$(basename "${module_path}" .py)
  # instead of PATH python3 you may want to use the interpreter from a specific virtual environment
  (PYTHONPATH="$(dirname ${module_path})"; export PYTHONPATH; ${python_interpreter} <<EOF
import ${module_name}


${module_name}.${function_name}(${args})
==Python Source Code and Tests Maintained within the Project Structure==
EOF
<font size=-1>
  )
.
}
├─ .gitignore <font color=darkgray><i># Optional</I></font>
├─ src
├─ tests
├─ venv
├─ requirements.txt
└─ <wrapper-bash-script>
</font>


call-python ./my_module.py my_function blue red green
To maintain this subtree as part of a larger Python project whose <code>.gitignore</code> ignores virtual environment directories, add this to the subtree's <code>.gitignore</code>:
</syntaxhighlight>


The output will be:
<syntaxhighlight lang='text'>
<syntaxhighlight lang='text'>
this is my_function(blue, red, green)
!venv
</syntaxhighlight>
</syntaxhighlight>


⚠️ <Font color=darkkhaki>Numbers will be converted to strings before being passed to the function, the implementation is incomplete.</font>
<font color=darkkhaki>TODO: reconcile with [[Python Project Layout]]</font>.


=Using the Interpreter from the a Virtual Environment=
==Only Externally Published Packages==
<font color=darkkhaki>TODO</font>
<font size=-1>
.
├─ venv
├─ requirements.txt
└─ <wrapper-bash-script>
</font>
 
==Using the Interpreter from the a Virtual Environment==
<font color=darkkhaki>
<font color=darkkhaki>
TODO
If you want to use the interpreter from a specific virtual environment instead of the interpreter found in PATH, explicitly use the path to the binary from the virtual environment directory:
If you want to use the interpreter from a specific virtual environment instead of the interpreter found in PATH, explicitly use the path to the binary from the virtual environment directory:
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
Line 67: Line 68:
</syntaxhighlight>
</syntaxhighlight>
</font>
</font>
===Bash Wrapper that Bootstraps a Virtual Environment===
<syntaxhighlight lang='bash'>
function init_venv() {
  [[ -d $(dirname $)/venv ]] && return 0
  info "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || fail "python3 not in PATH"
  python3 -m venv "$(dirname $)/venv"
}
</syntaxhighlight>
Also see: {{Internal|Python_Virtual_Environment#Overview|Python Virtual Environment}}


=Execute a Complex Python Program as a Command Line Command=
=Inline Python Code=
Use bash  [[Bash_Input/Output#Here-Docs|here-doc]]:
<syntaxhighlight lang='bash'>
  python3 <<EOF
print('blah')
EOF
</syntaxhighlight>


Embed logic into a Python program and invoke the program as a regular command, from a pipeline for example:
Also see: {{Internal|Python_Language_Modularization#Python_Script|Python Script}}


<syntaxhighlight lang='bash'>
=Bash to Python Function Mapping=
./cicd/build-and-publish
 
</syntaxhighlight>
{{Internal|Bash to Python Function Mapping#Overview|Bash to Python Function Mapping}}

Latest revision as of 21:10, 18 May 2023

Internal

Overview

This article summarizes a series of aspects related to invoking Python program from bash.

Running a Python Program with a Bash Wrapper

Check out these links for the meaning of standalone Python program and Python script.

An example of a run script for a simple Python project is available here:

Python Project Layout | The run script


Refactor all this and reconcile with:

Python Project Layout


Problems to be solved:

1. The problem of Python binaries. We either rely on the correct version of the Python to be present on the executing image, or we encase it it in a virtual environment. There we need to be aware of the problem of the symbolic links and binary formats.

2. The problem of package dependencies and requirements.txt

3. The layout of the source code: sources and tests, or package dependencies in a Python package repository.

Python Source Code and Tests Maintained within the Project Structure

.
├─ .gitignore # Optional
├─ src
├─ tests
├─ venv
├─ requirements.txt
└─ <wrapper-bash-script>

To maintain this subtree as part of a larger Python project whose .gitignore ignores virtual environment directories, add this to the subtree's .gitignore:

!venv

TODO: reconcile with Python Project Layout.

Only Externally Published Packages

TODO

.
├─ venv
├─ requirements.txt
└─ <wrapper-bash-script>

Using the Interpreter from the a Virtual Environment

TODO

If you want to use the interpreter from a specific virtual environment instead of the interpreter found in PATH, explicitly use the path to the binary from the virtual environment directory:

$(dirname $0)/venv/bin/python ...

Bash Wrapper that Bootstraps a Virtual Environment

function init_venv() {
  [[ -d $(dirname $)/venv ]] && return 0
  info "initializing venv ..."
  python3 --version 1>/dev/null 2>&1 || fail "python3 not in PATH"
  python3 -m venv "$(dirname $)/venv"
}

Also see:

Python Virtual Environment

Inline Python Code

Use bash here-doc:

  python3 <<EOF
print('blah')
EOF

Also see:

Python Script

Bash to Python Function Mapping

Bash to Python Function Mapping