Relevant Guide(s)

HySDS libraries

HySDS core libraries are consisted of 4 different python packages:

prov_es - https://github.com/hysds/prov_es
osaka - https://github.com/hysds/osaka
hysds_commons - https://github.com/hysds/hysds_commons
hysds - https://github.com/hysds/hysds

with the hysds python library having dependencies of prov_es, osaka and hysds_commons

from setuptools import setup, find_packages

setup(
    ...
    include_package_data=True,
    zip_safe=False,
    install_requires=[
        ...
        "osaka>=0.0.1",
        "prov_es>=0.2.0",
        "hysds_commons>=0.1",
       ...
    ],
    setup_requires=["pytest-runner"],
    tests_require=["pytest"],
)

The order in which the packages are built will be:

prov_es
osaka
hysds_commons
hysds

A major next step in deployments of HySDS is to publish the HySDS core python libraries to PyPI (and potentially conda). That way we can simply pip install the package straight away instead of having to download the package from Artifactory and/or GitHub

Setting up the package(s)

Tools used:

If you do not have setuptools instead, you can run pip install setuptools on your machine (and virtualenv)

We will be using the osaka library as an example, this is the (simplified) project structure, with the core code in the osaka directory, which will also be the package name; ie: import osaka

.
├── COPYING
├── LICENSE
├── README.md
├── archived_history.txt
├── osaka
│   ├── __init__.py
│   ├── __main__.py
│   ├── base.py
│   ├── cooperator.py
│   ├── lock.py
│   ├── main.py
│   ├── storage
│   │   ├── __init__.py
│   │   ├── az.py
│   │   ├── example.py
│   │   ├── file.py
│   │   ├── ftp.py
│   │   ├── gs.py
│   │   ├── http.py
│   │   ├── s3.py
│   │   ├── sftp.py
│   │   └── webdav.py
│   ├── transfer.py
│   └── utils.py
├── requirements.txt
└── setup.py

the packages and subdirectories will each have an __init__.py file, which will allow the package to be imported

setup.py

the next step is to look at setup.py

newer versions of setuptools will print this warning message:

“SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools."

we should probably move to a more modern & supported tool such as project.toml:

“Modern Python packages can contain a pyproject.toml file, first introduced in PEP 518 and later expanded in PEP 517, PEP 621 and PEP 660. This file contains build system requirements and information, which are used by pip to build the package.”

osaka's setup.py file is as followed with the keywords:

name - the package name (name used for the import statement)
version - package version (can only be used once each time you publish the package to PyPI)
description - short description (at most 200 characters)
long_description - we will point this to the README, will display in the package PyPI page
install_requires - list of package dependencies
entrypoint - used if your python package has a CLI (command-line) interface
classifiers - “tags” that will make your project more searchable (list of classifiers here)

from __future__ import unicode_literals
from __future__ import print_function
from __future__ import division
from setuptools import setup, find_packages
import osaka


def readme():
    with open("README.md") as f:
        return f.read()


setup(
    name="osaka",  # TODO: if the name already exists in pypi, will opt for hysds-osaka
    version=osaka.__version__,
    description=osaka.__description__,
    long_description_content_type="text/markdown",
    long_description=readme(),
    url=osaka.__url__,
    packages=find_packages(exclude=["tests.*", "tests"]),
    include_package_data=True,
    zip_safe=False,
    install_requires=[
        "requests>=2.7.0",
        "easywebdav==1.2.0",
        "filechunkio==1.6.0",
        "azure-storage-blob==1.4.0",
        "awscli>=1.17.1",
        "boto3>=1.11.1",
        "google-cloud-storage>=0.22.0",
        "six>=1.10.0",
        "configparser>=3.5.0",
        "future>=0.17.1",
        "backoff>=1.3.1",
        "mock>=4.0.3",
        "moto>=2.0.6",
    ],
    entry_points={
        "console_scripts": ["osaka = osaka.__main__:main"]
    },
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Intended Audience :: Science/Research",
        "License :: OSI Approved :: Apache Software License",
        "Programming Language :: Python :: 3 :: Only",
        "Programming Language :: Python :: 3.5",
        "Programming Language :: Python :: 3.5",
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
        "Programming Language :: Python :: 3.8",
        "Programming Language :: Python :: 3.9",
        "Operating System :: POSIX :: Linux",
        "Operating System :: Unix",
        "Operating System :: MacOS :: MacOS X",
    ],
)

Creating the package

To create the python package we will have to run the following commands:

pip install -e .
- installs the package locally
- this will allow us to test the import of the package
python setup.py sdist bdist_wheel
- sdist - the source distribution of the package, consisting of the .tar.gz and the .whl file
- bdist_wheel - copy of the entire project, along with other files

after building the source distribution your project structure should have a dist/ folder:

Publishing the packages

Tools used:

https://github.com/pypiserver/pypiserver - local pypi server for testing
https://hub.docker.com/r/pypiserver/pypiserver - local PyPI server in docker
https://twine.readthedocs.io/en/stable/ - CLI tool for publishing python packages

Testing locally:

We will use the pypi-server to run a local PyPI server to test out publishing:

Running the server:

-p - server port (ex. 8080)
-v - verbose
-o - allow for overwriting package + version (this will not be allowed when publishing to PyPI’s production and test servers)
-P . - disable required password for publishing
-a . - disable required username for publishing
~/packages - package location in server

accessed at http://localhost:8080

in the root directory of the package we’ll publish to the server (leave username + password blank)

output:

now the package location will have the following files

next we will pip install the the osaka library from our local PyPI server

package officially installed

the package location is now in <env>/lib/python3.9/site-packages

Publishing to the PyPI test server

package + version can only be published once

ex. cannot re-publish osaka v1.2.0 , otherwise it will return a 400

the package is published here https://test.pypi.org/project/osaka/

Continuous integration (CI)

When we create a new release of a HySDS package (osaka, prov_es, hysds_commons or hysds) we can trigger a GitHub webhook and trigger a job (ie. CircleCI) to publish the package to PyPI

Will need to iron out the details:

CircleCI vs Jenkins
will it be a separate CircleCI job or part of an existing workflow
- publish to a test server first (ie. local PyPI server) before publishing to the production PyPI server

Auto-releasing packages with CircleCI

the following workflow, build-and-deploy, will be triggered when a new tag/release is created on GitHub

build-and-deploy workflow is composed of 2 jobs:

build
- installs pytest and installs the python package locally
- runs pytest
publish-pypi (only if the previous build job completes)
- initializes the .pypirc file with the URL, credentials, etc. for publishing
- pip install twine
- builds the python package distribution
- publishes to PyPI with twine

CircleCI’s config.yml:

To-do list:

Publish HySDS packages to PyPI test server (prov_es, osaka, hysds_commons, hysds)

Publish HySDS packages to PyPI’s official server

Integrate CircleCI to build and publish to PyPI whenever we make a new release

Publish HySDS to conda (optional)

HySDS-Core

Publishing HySDS core packages to PyPI