Step1: Pypi and TestPypi account

Click link to register Pypi and TestPypi account.

These two accounts need to register seperately. The Pypi account if for official library upload and the test one is for library testing.

I strongly recommend to upload your library to TestPypi and use it first before you upload it to the official Pypi.

Step2: Setup environment

Need to install setuptools , wheel , twine using the following commands:

pip install setuptools wheel twine

To save your time, you can just use anaconda to install the following environment.yml :

name: pipenv
channels:
  - defaults
dependencies:
  - pip==21.1.2
  - python==3.7.11
  - pip:
    - setuptools
    - wheel
    - twine

Using the following command in your terminal to create a new pipenv virtual environment:

conda env create -f environment.yml

Step 3: Construct files

3.1 Overall structure

project_name
.
├── LICENSE
├── README.md
├── setup.py
├── package_name
│	├── __init__.py
│	├── main.py
│   ├── utils
│   │   ├── __init__.py
│   │   ├── utils.py
│   │   └── example
│			├── __init__.py
├── tests
│   ├── __init__.py
│   └── example
│       ├── __init__.py

You can make your structure like above but you need to have some necessary files, which I will introduce one by one.

Here is my example that you can refer to: MiSTGen

3.2 License

Each package you uploaded to PyPi requirs a license.

To choose your license:

  • Find it in this website: https://choosealicense.com/
  • Or pick one when you create your Github repo

Your license should be included in a LICENSE file in the root directory.

3.3 Setup.py

setup.py is the build script for setuptools. It tells setuptools your package (such as name and version) and the code files to include.

#!/usr/bin/env python
from setuptools import setup, find_packages
import package_name

with open("README.md", "r", encoding='utf-8') as fh:
    long_description = fh.read()
    
setup(
    name="lib_name",
    version=package_name1.__version__,
    author="Your name",
    author_email="Your email",
    description="A short description",
    long_description=long_description,
    long_description_content_type="text/markdown",
    license="Your selected license",
    url="Your github url",
    packages=find_packages(),
    install_requires=[
        "matplotlib==3.4.2",
        "numpy",
        "cvxopt"
        ],
    classifiers=[
        "Topic :: Software Development :: Libraries :: Python Modules",
        "Programming Language :: Python",
        "Programming Language :: Python :: 3.7",
    ],
)

  • name is your library name, search on PyPi to avoid any “same name” conflict.

  • version is your version number, when you upload your package, make sure the lastest version is higher than previous version. Here I put it in the package_name/__init__.py file. You can also put some necessary info in this file, just like me:

    __title__ = 'package_name'
__version__ = '0.1.0'
__author__ = 'Author_names'
__copyright__ = 'Copyright 2022 author_names'
  • description is a short description.
  • long_description is a long description, you can use what every you put into your README.md file.
  • url is your project url address, you can put your github_url here.
  • packages is your package list. setuptools.find_packages() will automatically final all your packages under root directory. Here you have only one package package_name.
  • install_requires contains all the dependencies that this library is needed to run.
  • classifiers is written based on PyPi Classifiers, it contains at least your using python version.
  • For more details, please check Packaging Python Projects

Step 4: Build program

Run following command in root directory which is where setup.py located.

  • To check and see if your setup.py file is correct:
python setup.py check

you should be able to see: running check if there is no errors.

  • Build program:
python setup.py sdist bdist_wheel

After you build your program(step 4), you will see two files in the build directory of your root directory: one is xxx.tar.gz file, which is your source file, and the other one is your distribution xxx.hl file.

Step 5: Upload

To upload your package to the PyPi and TestPyPi, we need to setup our configuration file first to prevent constantly input our username and password everytime. (I’m a lazy person so I’ll only introduce the lazy way, lol)

5.1 Set up your local pypirc file

Inorder to make your upload easier and save the time of inputing user name and password, create a .pypirc in your home directory:

  • Windows: that would normally be C:\\User\Username\
  • Linx: put .pypirc under ~/directory.

What you need to modify in your .pypirc would be your user name and password.

[pypi]
repository = https://upload.pypi.org/legacy/
username = (your user name)
password = (your password)

[testpypi]
username = (your user name)
password = (your password)

5.2 Upload your package

  • To upload to the PyPi:
twine upload dist/*
  • Before you do the above step, I strongly recommend you to upload your package to TestPyPi first:
twine upload --repository testpypi dist/*

Caveat:

  • TestPyPi is just for testing if you can upload your project correctly, but it cannot be used as the really pip install packages for you to test since if you have multiple packages install require in your setup.py file, like matlibplot, TestPyPi website may not have this package and correct version for your test package to install and you will see something like:
  ERROR: Could not find a version that satisfies the requirement matplotlib (from opencoodx) (from versions: none)
  ERROR: No matching distribution found for matplotlib
  • To test the function of your package, you still need to upload your package to PyPi

Step6: How to use

After you publish toPyPi or TestPyPi , you should be able to install it useing the command:

  • For testing
pip install -i https://test.pypi.org/simple/ your_lib_name
  • For official 
pip install your_lib_name

Here is my example: mistgen

Some other problems

Q1: How to include non-python files in your tar.gz file?

When you open your xxx.tar.gz file, you will find other are some data files (.txt, .yml,.yaml) or other program files like (.pyx , .pyd) are not in your source file. You can create a MANIFEST.in or setup.cfg your root directory:

project_name
.
├── LICENSE
├── README.md
├── setup.py
├── setup.cfg
├── MANIFEST.in
├── package_name

In the MANIFEST.in, you can write the following:

global-include *.pyx

Then you will include all the .pyxin your package, but this will be very slow since it will traverse all the package files.

Or you could just include some specific directory or file, like this:

include package_name/package_data/*
include package_name/utils/abc.pyx

Then you will include everything inside package_data directory and abc.pyx file in the utils directory.

In this way, you can include any non-python(.py) files into your source file.

Q2: How to include non-python files in your whl file?

When people use pip install your_package_name to install your python package, this command will download everything in your .whl file.

Even after you specify everything you want to include in the MANIFEST.in, your .whl file still doesn’t have those non-python files that you need, like data files (.txt, .yml,.yaml) or other program files like (.pyx , .pyd).

In this case, you need to modify your setup.py like following:

setup(
    ...
    package_data={"":["*.yaml","*.pyx"]},
)

In this way, you can then include all the .yaml and .pyx files into your distribution whl file.

for more details, you can check here

Reference

  • https://packaging.python.org/en/latest/guides/using-testpypi/
  • https://packaging.python.org/en/latest/tutorials/packaging-projects/
  • https://blog.csdn.net/MemoryD/article/details/84295855
  • https://towardsdatascience.com/how-to-upload-your-python-package-to-pypi-de1b363a1b3
  • https://packaging.python.org/en/latest/guides/using-manifest-in/
  • https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  • https://setuptools.pypa.io/en/latest/userguide/datafiles.html