title: Packaging with Python

Packaging with Python

Python pip

The Python Package Index (PyPI) is a repository of software for the Python programming language.

You can install the python pip manager with the following command

Some outdated distributions still have python2 as default python, use python3 command

On Ubuntu

apt install python3-pip

On OSX

brew install python3

With docker:

docker run -it python:3.8-alpine sh

installing packages with pip

And then you can install any packages you want on https://pypi.org/

sudo pip3 install setuptools==50.0.2 # system wide install
sudo pip3 install setuptools==50.0.2 --user # user install
pip3 install twine wheel --user

pip is just another python module

python3 -m pip install --user --upgrade pip

Packaging python projects

You can follow the official guide (https://packaging.python.org/tutorials/packaging-projects/)

Creating the package files

This is the recommended basic structure your project should have to easily build a pip packages:

your_project/
├── LICENSE
├── README.md
├── example_pkg/
│   └── __init__.py
├── setup.py
└── tests/

But we can adapt it to follow the LBMC guide of good practices

your_project/
├── LICENSE
├── README.md
├── src/
│   └── example_pkg/
│      └── __init__.py
│   └── setup.py
│   └── tests/

All you python code goes in the example_pkg/ folder. The most important file for the packaging is the setup.py file.

Here is a basic setup.py file:

import setuptools

with open("../README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-YOUR-USERNAME-HERE", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
········"Programming·Language·::·Python·::·3",
········"License·::·OSI·Approved·::·CEA·CNRS·Inria·Logiciel·Libre·License,·\
 version·2.1·(CeCILL-2.1)",
········"Operating·System·::·OS·Independent"
····],
    python_requires='>=3.6',
)

• name is the distribution name of your package. This can be any name as long as only contains letters, numbers, _ , and -. It also must not already be taken on pypi.org. Be sure to update this with your username, as this ensures you won’t try to upload a package with the same name as one which already exists when you upload the package.
• version is the package version see PEP 440 for more details on versions.
• author and author_email are used to identify the author of the package.
• description is a short, one-sentence summary of the package.
• long_description is a detailed description of the package. This is shown on the package detail package on the Python Package Index. In this case, the long description is loaded from README.md which is a common pattern.
• long_description_content_type tells the index what type of markup is used for the long description. In this case, it’s Markdown.
• url is the URL for the homepage of the project. For many projects, this will just be a link to GitHub, GitLab, Bitbucket, or similar code hosting service.
• packages is a list of all Python import packages that should be included in the Distribution Package. Instead of listing each package manually, we can use find_packages() to automatically discover all packages and subpackages. In this case, the list of packages will be example_pkg as that’s the only package present.
• classifiers gives the index and pip some additional metadata about your package. In this case, the package is only compatible with Python 3, is licensed under the MIT license, and is OS-independent. You should always include at least which version(s) of Python your package works on, which license your package is available under, and which operating systems your package will work on. For a complete list of classifiers, see https://pypi.org/classifiers/.

Creating distribution archives

Now you just have to run the following command in the same directory as the setup.py file:

python3 setup.py sdist bdist_wheel

It will create files in the dist/ directory

dist/
  example_pkg_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
  example_pkg_YOUR_USERNAME_HERE-0.0.1.tar.gz

The tar.gz file is a Source Archive whereas the .whl file is a Built Distribution. Newer pip versions preferentially install built distributions, but will fall back to source archives if needed. You should always upload a source archive and provide built archives for the platforms your project is compatible with.

What can you do with those two file ?

Install them:

You can use the .whl or the .tar.gz file to install your package

pip3 install dist/example_pkg_YOUR_USERNAME_HERE-0.0.1.tar.gz --user

Upload them

You can upload your package to pypi, but first you can run test on https://test.pypi.org/. As https://pypi.org is an archive, if you upload broken packages, they will stay there.

You first need to create an account https://test.pypi.org/account/register/

Then we use the twine tools that we installed before

twine upload --skip-existing --repository testpypi dist/*

The output should look like that:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading example_pkg_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
100%|█████████████████████| 4.65k/4.65k [00:01<00:00, 2.88kB/s]
Uploading example_pkg_YOUR_USERNAME_HERE-0.0.1.tar.gz
100%|█████████████████████| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

To install your package from https://test.pypi.org you can use the following pip options:

pip install --index-url https://test.pypi.org/simple/ --no-deps example-pkg-YOUR-USERNAME-HERE --user

You should be able to open a python console anywhere and run:

>>> import example_pkg

When everything is OK, you can create an account on https://pypi.org and use the twinecommand without the --repository testpypi option.

Creating executable software

You can also use pip to distribute executable software. To do that, you have to specify the __main__ function to execute when calling your software in the setup.py file.

setuptools.setup(
  	...
		entry_points={
        'console_scripts': ['example_pkg=example_pkg.__main__:main'],
    },
  	...
)

You can have different executable in this list with the format EXECUTABLE_NAME=LIBRARY.FILE:FUNCTION

After the installation, calling example_pkgwill run your software if your $PATH is correctly configured.

Adding dependencies to your package

As your project will grow more complex, you will split it into different file for code clarity.

Your __init__.py file will need to contain a list of all the .py files in the example_pkg repository:

#!/usr/bin/env python3
# -*-coding:Utf-8 -*

"""
idr library
"""

name = "midr"
__all__ = ["__main__",
    "idr", "samic", "archimedean", "archimedean_plots",
    "log", "narrowpeak", "raw_matrix", "auxiliary"]

As you don't want to reinvente the wheel, you may also import other python library (which could be installed with pip). You can specify a list of these library in the setup.pyfile:

setuptools.setup(
  	...
		install_requires=[
        'cmake>=3.18'
        'scipy>=1.3',
        'numpy>=1.16',
        'pynverse>=0.1',
        'pandas>=0.25.0',
        'mpmath>=1.1.0',
        'matplotlib>=3.0.0'
    ],
  	...
)

Don't forget to specify the version of each dependency to ensure that the function you use are present in the installed library.

If, some package are required for the installation of your package (for example here cmake), you should also add them to the install_requireslist.