How can I start a python project the proper way? Extensible, with tests, as library and cli application, in a virtual environment and with an older python version (raspberry pi).

I’ve spent days conquering the same problems over and over again. Notorious import problems. Imports from parent modules not working. Tests work but the cli doesn’t. Pipenv not installing the right python version. You name it.

Each time I think I’m all set, I’m encountering another missed step in my endeavour for clean code.😶

This ’essay’ shall be the base for my future python projects.

A minimal python project

Our starting point is: https://packaging.python.org/en/latest/tutorials/packaging-projects/ I’ll use the absolute minimum. E.g. pyproject.toml contains no description nor licence etc. Furthermore I’ve deleted files e.g. readme.md and directories e.g. tests.

/home/ra/tmp/$ tree packaging_tutorial

packaging_tutorial
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        └── __init__.py

3 directories, 3 files

Let’s have a look at the content of pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "example_package"
version = "0.0.1"

What’s most counterintuitive is that the name key under the [project] section in pyproject.toml must match the name of the package (example_package) and not the parent’s directory name (packaging_tutorial). In my mind the parent directory is the project’s name. In fact, the parent directory name seems completely negligible in perspective of development.

Installing the package in a virtual environment using pipenv with python 3.8

We replicate the steps from¹ not from pypi but locally from within the project’s folder (thus the dot . after -e).

pipenv install -e . --python 3.8

The -e flag (editable) allows further development (hence editing) alongside the installation.

But am I really installing if using a virtual environment? At this moment I’ll leave the flag since I haven’t detected any downsides to it yet.

Ensure the virtual environment has been created

pipenv --venv

/home/ra/.local/share/virtualenvs/packaging_tutorial-okktpVF3

Activate this project’s virtualenv

pipenv shell

Ensure version 3.8 is used as virtual python environment

python -V

Check the new directory structure

tree packaging_tutorial

packaging_tutorial
├── Pipfile
├── Pipfile.lock
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        └── __init__.py

3 directories, 5 files

Let’s have a look at Pipfile’s content:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
example-package = {file = ".", editable = true}

[dev-packages]

[requires]
python_version = "3.8"
python_full_version = "3.8.12"

Activate the virtual environment

pipenv shell

python
# Python 3.8.12 (default, Dec  4 2021, 10:54:00)
# [GCC 11.1.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Mind that I changed the name from example_package_YOUR_USERNAME_HERE to example_package

>>> from example_package import example
>>> example.add_one(2)

Review the installation in perspective of pipenv and python 3.8

At this stage we’ve successfully replicated the package installation locally with pipenv and another python version (3.8).

Run something from the cli

Great. We’re able to import the add_one function from the example.py module in our virtual environment.

For better depiction I’ll add a print statement to our example.py module:

def add_one(number):
    return number + 1


print(add_one(1))

Now let’s run it from the command line interface (cli)

python -m example

home/ra.local/share/virtualenvs/packaging_tutorial-okktpVF3/bin/python: No module named example

Let’s check what’s actually installed in our virtual environment

pip list

Package         Version Editable project location
--------------- ------- -------------------------------
example_package 0.0.1   /home/ra/tmp/packaging_tutorial
pip             24.0
setuptools      69.2.0
wheel           0.43.0

Apparently we’re all set. Let’s try calling the module differently:

python -m src.example_package.example

This works but somehow feels wrong. Why do I need to be that explicit? What about the claimed automatic discovery² for our src-layout³?

After further reading⁴, let’s try only the package name:

python -m example_package

No module named example_package.__main__; 'example_package' is a package and cannot be directly executed

At least, we got a new hint from the error, which moreover matches the setuptools instructions⁴. Thus, let’s make it so and add __main__.py

1
2
3
4
5

# __main__.py
from . import add_one

if __name__ == '__main__':
    add_one(1)

ImportError: cannot import name 'add_one' from 'example_package' (/home/ra/tmp/packaging_tutorial/src/example_package/__init__.py)

At this point it looks like __init__.py is consulted due to the highlighted line 2 in __main__.py.

Since we installed the package, we will import this package itself in our modules’s cli entrypoint __main__.py.

Let’s change __main__.py

from example_package import example
# Works as well:
# from . import example

if __name__ == '__main__':
    example.add_one(1)

Now running python -m example_package in terminal being in /packaging_tutorial works.

Review our current (cli) project’s state

At this point we’re able to

1. pipenv install a package using an older python version
2. import our package as library
3. run the packages’s module from the terminal/cli

tree packaging_tutorial

packaging_tutorial
├── Pipfile
├── Pipfile.lock
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        ├── __init__.py
        └── __main__.py

3 directories, 6 files