Publishing HySDS core packages to PyPI
Relevant Guide(s)
HySDS libraries
HySDS core libraries are consisted of 4 different python packages:
prov_es
- https://github.com/hysds/prov_esosaka
- https://github.com/hysds/osakahysds_commons
- https://github.com/hysds/hysds_commonshysds
- GitHub - hysds/hysds: Hybrid Cloud Science Data System Framework
with the hysds
python library having dependencies of prov_es
, osaka
and hysds_commons
from setuptools import setup, find_packages
setup(
...
include_package_data=True,
zip_safe=False,
install_requires=[
...
"osaka>=0.0.1",
"prov_es>=0.2.0",
"hysds_commons>=0.1",
...
],
setup_requires=["pytest-runner"],
tests_require=["pytest"],
)
The order in which the packages are built will be:
prov_es
osaka
hysds_commons
hysds
A major next step in deployments of HySDS is to publish the HySDS core python libraries to PyPI (and potentially conda). That way we can simply pip install
the package straight away instead of having to download the package from Artifactory and/or GitHub
Setting up the package(s)
Tools used:
If you do not have setuptools
instead, you can run pip install setuptools
on your machine (and virtualenv
)
We will be using the osaka
library as an example, this is the (simplified) project structure, with the core code in the osaka
directory, which will also be the package name; ie: import osaka
.
├── COPYING
├── LICENSE
├── README.md
├── archived_history.txt
├── osaka
│ ├── __init__.py
│ ├── __main__.py
│ ├── base.py
│ ├── cooperator.py
│ ├── lock.py
│ ├── main.py
│ ├── storage
│ │ ├── __init__.py
│ │ ├── az.py
│ │ ├── example.py
│ │ ├── file.py
│ │ ├── ftp.py
│ │ ├── gs.py
│ │ ├── http.py
│ │ ├── s3.py
│ │ ├── sftp.py
│ │ └── webdav.py
│ ├── transfer.py
│ └── utils.py
├── requirements.txt
└── setup.py
the packages and subdirectories will each have an __init__.py
file, which will allow the package to be imported
setup.py
the next step is to look at setup.py
newer versions of setuptools
will print this warning message:
“SetuptoolsDeprecationWarning: setup.py
install is deprecated. Use build and pip and other standards-based tools."
we should probably move to a more modern & supported tool such as project.toml:
“Modern Python packages can contain a pyproject.toml
file, first introduced in PEP 518 and later expanded in PEP 517, PEP 621 and PEP 660. This file contains build system requirements and information, which are used by pip to build the package.”
osaka
's setup.py
file is as followed with the keywords:
name - the package name (name used for the
import
statement)version - package version (can only be used once each time you publish the package to PyPI)
description - short description (at most 200 characters)
long_description - we will point this to the README, will display in the package PyPI page
install_requires - list of package dependencies
entrypoint - used if your python package has a CLI (command-line) interface
classifiers - “tags” that will make your project more searchable (list of classifiers here)
from __future__ import unicode_literals
from __future__ import print_function
from __future__ import division
from setuptools import setup, find_packages
import osaka
def readme():
with open("README.md") as f:
return f.read()
setup(
name="osaka", # TODO: if the name already exists in pypi, will opt for hysds-osaka
version=osaka.__version__,
description=osaka.__description__,
long_description_content_type="text/markdown",
long_description=readme(),
url=osaka.__url__,
packages=find_packages(exclude=["tests.*", "tests"]),
include_package_data=True,
zip_safe=False,
install_requires=[
"requests>=2.7.0",
"easywebdav==1.2.0",
"filechunkio==1.6.0",
"azure-storage-blob==1.4.0",
"awscli>=1.17.1",
"boto3>=1.11.1",
"google-cloud-storage>=0.22.0",
"six>=1.10.0",
"configparser>=3.5.0",
"future>=0.17.1",
"backoff>=1.3.1",
"mock>=4.0.3",
"moto>=2.0.6",
],
entry_points={
"console_scripts": ["osaka = osaka.__main__:main"]
},
classifiers=[
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Operating System :: POSIX :: Linux",
"Operating System :: Unix",
"Operating System :: MacOS :: MacOS X",
],
)
Creating the package
To create the python package we will have to run the following commands:
pip install -e .
installs the package locally
this will allow us to test the
import
of the package
python setup.py sdist bdist_wheel
sdist
- the source distribution of the package, consisting of the.tar.gz
and the.whl
filebdist_wheel
- copy of the entire project, along with other files
after building the source distribution your project structure should have a dist/
folder:
Publishing the packages
Tools used:
GitHub - pypiserver/pypiserver: Minimal PyPI server for uploading & downloading packages with pip/easy_install - local
pypi
server for testinghttps://hub.docker.com/r/pypiserver/pypiserver - local PyPI server in docker
twine 5.1.1 documentation - CLI tool for publishing python packages
Testing locally:
We will use the pypi-server
to run a local PyPI server to test out publishing:
Running the server:
-p - server port (ex. 8080)
-v - verbose
-o - allow for overwriting package + version (this will not be allowed when publishing to PyPI’s production and test servers)
-P . - disable required password for publishing
-a . - disable required username for publishing
~/packages - package location in server
http://localhost:8080
in the root directory of the package we’ll publish to the server (leave username + password blank)
output:
now the package location will have the following files
next we will pip
install the the osaka
library from our local PyPI server
package officially installed
the package location is now in <env>/lib/python3.9/site-packages
Publishing to the PyPI test server
package + version can only be published once
ex. cannot re-publish osaka v1.2.0
, otherwise it will return a 400
the package is published here osaka
Continuous integration (CI)
When we create a new release of a HySDS package (osaka
, prov_es
, hysds_commons
or hysds
) we can trigger a GitHub webhook and trigger a job (ie. CircleCI) to publish the package to PyPI
Will need to iron out the details:
CircleCI vs Jenkins
will it be a separate CircleCI job or part of an existing workflow
publish to a test server first (ie. local PyPI server) before publishing to the production PyPI server
Auto-releasing packages with CircleCI
the following workflow, build-and-deploy
, will be triggered when a new tag/release is created on GitHub
build-and-deploy
workflow is composed of 2 jobs:
build
installs
pytest
and installs the python package locallyruns
pytest
publish-pypi
(only if the previousbuild
job completes)initializes the
.pypirc
file with the URL, credentials, etc. for publishingpip install twine
builds the python package distribution
publishes to PyPI with
twine
CircleCI’s config.yml
:
To-do list:
prov_es
, osaka
, hysds_commons
, hysds
)