publishing on pypi

some notes on packaging & releasing python modules on pypi

update 2022-08-28:

• “releasing on testpypi”: incluce the extra-index-url
• “building your distribution”: use build instead of setup.py sdist bdist_wheel

example project structure

mypackage
    mypackage
        module1
            __init__.py
        module2
            __init__.py
        app.py
    tests
    .bumpversion.cfg
    .travis.yml
    setup.py
    MANIFEST.in
    LICENSE
    README.md

needed files

setup.py

(based on https://packaging.python.org/tutorials/packaging-projects/)

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-your-username",
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
    
    # enables the use of MANIFEST.in
    include_package_data=True,
    
    # if you are writing a cli tool, add your command here!
    # more infos here: 
    # https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation
    entry_points={
        'console_scripts': [
            'my_command = mypackage.app:myfunction'
        ]
    }
)

adding requirements to setup.py

there are a lot of discussions on what you should use: a requirements.txt (and then read it programmatically in your setup.py) or just a list in the setup.py.

the packaging user guide explains, that install_requires should specify the minimal, abstract requirements (without any sub-requirements) for your project to run, whereas your requirements.txt contains a long list of pinned versions of all required packages and their requirements.

also, a requirements.txt is usually used when you write a service. so, its used to describe a whole “environment” for your service to run in.

using extras_require, you can add different ’levels’ of requirements.

requirements = ['some requirements']
dev_requirements = requirements + ['additional development requirements']
test_requirements = dev_requirements + ['test requirements']

setuptools.setup(
    ...,
    install_requires=requirements,
    extras_require={
            'dev': dev_requirements,
            'test': test_requirements
    },
    ...
)

to install the default requirements, run pip install --editable . or, for the dev environment, pip install -e .[dev]

MANIFEST.in

if you set include_package_data to True in setup.py, you enabled MANIFEST.in. all files specified here are added to your distribution.

its probably a good idea to use whatever is in your file system: (source)

graft mypackage docs
global-exclude *.py[cod] __pycache__ *.so

tests

if you are writing a library its probably a good idea to test your code on different python versions. i found travis really good for that.

language: python

matrix:
  include:
    - name: "xenial 3.5"
      python: "3.5"
      dist: xenial

    - name: "xenial 3.6"
      python: "3.6"
      dist: xenial

    - name: "xenial 3.7"
      python: "3.7"
      dist: xenial

    - name: "osx 3.7"
      os: osx
      osx_image: xcode10.2  # Python 3.7.2 running on macOS 10.14.3
      language: shell       # 'language: python' is an error on Travis CI macOS
      python: "3.7"

    - name: "osx 3.6"
      os: osx
      osx_image: xcode10.2  # Python 3.7.2 running on macOS 10.14.3
      language: shell       # 'language: python' is an error on Travis CI macOS
      python: "3.6"

install: pip3 install -e .[test] || pip install -e .[test]

# https://docs.travis-ci.com/user/languages/python/#running-python-tests-on-multiple-operating-systems
script: python3 -m pytest || python -m pytest

# Push the results back to codecov
after_success:
  - codecov

(another option would be tox, but i never actually tried that)

versioning

you can use bump2version to update the version where needed. just put a .bumpversion.cfg in the root of your project, and bump your version with bump2version [patch|minor|major] (use --dry-run to see the changes before running the actual bump command!)

[bumpversion]
current_version = 0.4.1
commit = True
tag = True

[bumpversion:file:setup.py]
search = version='{current_version}'
replace = version='{new_version}'

actual releasing on pypi

building your distribution

update 2022-08-28:
setup.py bdist_wheel does not work anymore, and the official docs are updated to use build, so i updated this as well. (-> docs)

building the dist packages needs the build package, so if you havent done that already, install it with

pip install build --user

then, create a sdist distribution (basically just the sources) and a wheel (a “built” package):

python -m build

releasing on pypi

you can use twine to upload your created package to pypi.

to test, you can release it on testpypi first

twine upload --verbose --repository testpypi dist/*

then the install from test pypi.

the --extra-index-url is needed for packages that dont exist on pypi, as a fallback. otherwise you will likely see something like Could not find a version that satisfies the requirement [...]

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple mypackage

to release on pypi, run

twine upload --verbose --repository pypi dist/*

stuff to read

https://blog.ionelmc.ro/presentations/packaging/

“Less known packaging features and tricks” - a nice presentation about packaging

https://packaging.python.org/overview/

the python packaging guide. a lot to read.