/
Publishing HySDS core packages to PyPI

Publishing HySDS core packages to PyPI

Owned by Dustin Lo
Last updated: Dec 29, 2022
7 min read

Relevant Guide(s)

HySDS libraries

HySDS core libraries are consisted of 4 different python packages:

with the hysds python library having dependencies of prov_es, osaka and hysds_commons

from setuptools import setup, find_packages setup( ... include_package_data=True, zip_safe=False, install_requires=[ ... "osaka>=0.0.1", "prov_es>=0.2.0", "hysds_commons>=0.1", ... ], setup_requires=["pytest-runner"], tests_require=["pytest"], )

The order in which the packages are built will be:

  1. prov_es

  2. osaka

  3. hysds_commons

  4. hysds

 

A major next step in deployments of HySDS is to publish the HySDS core python libraries to PyPI (and potentially conda). That way we can simply pip install the package straight away instead of having to download the package from Artifactory and/or GitHub

Setting up the package(s)

Tools used:

If you do not have setuptools instead, you can run pip install setuptools on your machine (and virtualenv)

We will be using the osaka library as an example, this is the (simplified) project structure, with the core code in the osaka directory, which will also be the package name; ie: import osaka

. ├── COPYING ├── LICENSE ├── README.md ├── archived_history.txt ├── osaka │   ├── __init__.py │   ├── __main__.py │   ├── base.py │   ├── cooperator.py │   ├── lock.py │   ├── main.py │   ├── storage │   │   ├── __init__.py │   │   ├── az.py │   │   ├── example.py │   │   ├── file.py │   │   ├── ftp.py │   │   ├── gs.py │   │   ├── http.py │   │   ├── s3.py │   │   ├── sftp.py │   │   └── webdav.py │   ├── transfer.py │   └── utils.py ├── requirements.txt └── setup.py

the packages and subdirectories will each have an __init__.py file, which will allow the package to be imported

setup.py

the next step is to look at setup.py

newer versions of setuptools will print this warning message:

“SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools."

we should probably move to a more modern & supported tool such as project.toml:

“Modern Python packages can contain a pyproject.toml file, first introduced in PEP 518 and later expanded in PEP 517, PEP 621 and PEP 660. This file contains build system requirements and information, which are used by pip to build the package.”

osaka's setup.py file is as followed with the keywords:

  • name - the package name (name used for the import statement)

  • version - package version (can only be used once each time you publish the package to PyPI)

  • description - short description (at most 200 characters)

  • long_description - we will point this to the README, will display in the package PyPI page

  • install_requires - list of package dependencies

  • entrypoint - used if your python package has a CLI (command-line) interface

  • classifiers - “tags” that will make your project more searchable (list of classifiers here)

from __future__ import unicode_literals from __future__ import print_function from __future__ import division from setuptools import setup, find_packages import osaka def readme(): with open("README.md") as f: return f.read() setup( name="osaka", # TODO: if the name already exists in pypi, will opt for hysds-osaka version=osaka.__version__, description=osaka.__description__, long_description_content_type="text/markdown", long_description=readme(), url=osaka.__url__, packages=find_packages(exclude=["tests.*", "tests"]), include_package_data=True, zip_safe=False, install_requires=[ "requests>=2.7.0", "easywebdav==1.2.0", "filechunkio==1.6.0", "azure-storage-blob==1.4.0", "awscli>=1.17.1", "boto3>=1.11.1", "google-cloud-storage>=0.22.0", "six>=1.10.0", "configparser>=3.5.0", "future>=0.17.1", "backoff>=1.3.1", "mock>=4.0.3", "moto>=2.0.6", ], entry_points={ "console_scripts": ["osaka = osaka.__main__:main"] }, classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Operating System :: POSIX :: Linux", "Operating System :: Unix", "Operating System :: MacOS :: MacOS X", ], )

Creating the package

To create the python package we will have to run the following commands:

  • pip install -e .

    • installs the package locally

    • this will allow us to test the import of the package

  • python setup.py sdist bdist_wheel

    • sdist - the source distribution of the package, consisting of the .tar.gz and the .whl file

    • bdist_wheel - copy of the entire project, along with other files

after building the source distribution your project structure should have a dist/ folder:

├── dist │   ├── osaka-1.2.0-py3-none-any.whl │   └── osaka-1.2.0.tar.gz

Publishing the packages

Tools used:

Testing locally:

We will use the pypi-server to run a local PyPI server to test out publishing:

$ pip install pypiserver # Or: pypiserver[passlib,cache] $ mkdir ~/packages # Copy packages into this directory.

Running the server:

pypi-server -p 8080 -v -o -P . -a . ~/packages

  • -p - server port (ex. 8080)

  • -v - verbose

  • -o - allow for overwriting package + version (this will not be allowed when publishing to PyPI’s production and test servers)

  • -P . - disable required password for publishing

  • -a . - disable required username for publishing

  • ~/packages - package location in server

accessed at http://localhost:8080

in the root directory of the package we’ll publish to the server (leave username + password blank)

$ twine upload --repository-url http://localhost:8080 dist/* --verbose

output:

Uploading distributions to http://localhost:8080 INFO dist/osaka-1.2.0-py3-none-any.whl (52.9 KB) INFO dist/osaka-1.2.0.tar.gz (38.3 KB) INFO Querying keyring for username Enter your username: WARNING Your username is empty. Did you enter it correctly? WARNING See https://twine.readthedocs.io/#entering-credentials for more information. INFO Querying keyring for password Enter your password: WARNING Your password is empty. Did you enter it correctly? WARNING See https://twine.readthedocs.io/#entering-credentials for more information. INFO username: <empty> INFO password: <empty> Uploading osaka-1.2.0-py3-none-any.whl 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.8/64.8 kB • 00:00 • 68.3 MB/s INFO Response from http://localhost:8080/: 200 OK Uploading osaka-1.2.0.tar.gz 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB • 00:00 • 96.2 MB/s INFO Response from http://localhost:8080/: 200 OK

now the package location will have the following files

$ ls -l ~/packages total 192 -rw-r--r-- 1 ******** ****** 54163 Dec 14 16:43 osaka-1.2.0-py3-none-any.whl -rw-r--r-- 1 ******** ****** 39215 Dec 14 16:43 osaka-1.2.0.tar.gz

next we will pip install the the osaka library from our local PyPI server

$ pip install -i http://localhost:8080 osaka

package officially installed

$ pip list ... osaka 1.2.0 ...

the package location is now in <env>/lib/python3.9/site-packages

Publishing to the PyPI test server

$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*

package + version can only be published once

ex. cannot re-publish osaka v1.2.0 , otherwise it will return a 400

the package is published here Client Challenge

 

 

Continuous integration (CI)

When we create a new release of a HySDS package (osaka, prov_es, hysds_commons or hysds) we can trigger a GitHub webhook and trigger a job (ie. CircleCI) to publish the package to PyPI

GitHub WebHook trigger event

Will need to iron out the details:

  • CircleCI vs Jenkins

  • will it be a separate CircleCI job or part of an existing workflow

    • publish to a test server first (ie. local PyPI server) before publishing to the production PyPI server

 

Auto-releasing packages with CircleCI

the following workflow, build-and-deploy, will be triggered when a new tag/release is created on GitHub

build-and-deploy workflow is composed of 2 jobs:

  • build

    • installs pytest and installs the python package locally

    • runs pytest

  • publish-pypi (only if the previous build job completes)

    • initializes the .pypirc file with the URL, credentials, etc. for publishing

    • pip install twine

    • builds the python package distribution

    • publishes to PyPI with twine

CircleCI’s config.yml:

version: 2.1 jobs: build: docker: - image: circleci/python:3.9 auth: username: $DOCKER_USER password: $DOCKER_PASS steps: - checkout - run: name: Install dependencies command: | pip install pytest==7.2.0 pip install . - run: name: pytest command: | pytest . publish-pypi: docker: - image: circleci/python:3.9 auth: username: $DOCKER_USER password: $DOCKER_PASS steps: - checkout - run: name: Init .pypirc command: | echo -e "[pypi]" >> ~/.pypirc echo -e "repository: https://upload.pypi.org/legacy/" >> ~/.pypirc echo -e "username: $PYPI_USER" >> ~/.pypirc echo -e "password: $PYPI_PASSWORD" >> ~/.pypirc - run: name: Install tools command: | pip install twine==4.0.2 - run: name: Install and publish to PyPI command: | pip install . python setup.py sdist bdist_wheel twine upload dist/* --verbose --config-file ~/.pypirc workflows: version: 2 build-and-deploy: jobs: - build: context: - docker-hub-creds - git-oauth-token - pypi-creds - publish-pypi: context: - docker-hub-creds - git-oauth-token requires: - build filters: tags: only: /[0-9]+(\.[0-9]+)*/ branches: ignore: /.*/

 

To-do list:

Publish HySDS packages to PyPI test server (prov_es, osaka, hysds_commons, hysds)
Publish HySDS packages to PyPI’s official server
Integrate CircleCI to build and publish to PyPI whenever we make a new release
Publish HySDS to conda (optional)

Related content

Note: JPL employees can also get answers to HySDS questions at Stack Overflow Enterprise:

https://stackoverflow.jpl.nasa.gov/