Creating a Package

Basics: Creating and Distributing Distributions

If you have some useful Python modules that you think others might benefit from, but aren’t sure how to go about packaging them up and distributing them, then this short document is for you. By the end of it, you’ll be a contributor to the The Python Package Index (PyPI).

For a more detailed look at packaging a larger project, see this example.

Let’s begin.

Background

Suppose you’ve written a couple modules to help you keep track of your towel (location.py and utils.py), and you’d like to share them. First thing to do is come up with a CamelCase project name for them. Let’s go with “TowelStuff” since it seems appropriate and also it has not yet been used on the The Python Package Index (PyPI).

Arranging your file and directory structure

“TowelStuff” will be the name of our project as well as the name of our distribution. We should also come up with a package name within which our modules will reside (to avoid naming conflicts with other modules). For this example, there’s only one package, so let’s reuse the project name and go with “towelstuff”. Make the layout of your project directory (described below) look like this:

TowelStuff/
    bin/
    CHANGES.txt
    docs/
    LICENSE.txt
    MANIFEST.in
    README.txt
    setup.py
    towelstuff/
        __init__.py
        location.py
        utils.py
        test/
            __init__.py
            test_location.py
            test_utils.py

Here’s what you should do for each of those listed above:

  • Put into bin any scripts you’ve written that use your towelstuff package and which you think would be useful for your users. If you don’t have any, then remove the bin directory.
  • For now, the CHANGES.txt file should only contain:

    v<version>, <date> -- Initial release.
    

    since this is your very first version (version number will be described below) and there are no changes to report.

  • The docs dir should contain any design docs, implementation notes, a FAQ, or any other docs you’ve written. For now, stick to plain text files ending in ”.txt”. This author (JohnMG) likes to use Pandoc’s Markdown, but many Pythoneers use reStructuredText.
  • The LICENSE.txt file is often just a copy/paste of your license of choice. We recommend going with a commonly-used license, such as the GPL, BSD, or MIT.
  • The MANIFEST.in file should contain this:

    include *.txt
    recursive-include docs *.txt
    
  • The README.txt file should be written in reST so that the PyPI can use it to generate your project’s PyPI page. Here’s a 10-second intro to reST that you might use to start with:

    ===========
    Towel Stuff
    ===========
    
    Towel Stuff provides such and such and so and so. You might find
    it most useful for tasks involving <x> and also <y>. Typical usage
    often looks like this::
    
        #!/usr/bin/env python
    
        from towelstuff import location
        from towelstuff import utils
    
        if utils.has_towel():
            print "Your towel is located:", location.where_is_my_towel()
    
    (Note the double-colon and 4-space indent formatting above.)
    
    Paragraphs are separated by blank lines. *Italics*, **bold**,
    and ``monospace`` look like this.
    
    
    A Section
    =========
    
    Lists look like this:
    
    * First
    
    * Second. Can be multiple lines
      but must be indented properly.
    
    A Sub-Section
    -------------
    
    Numbered lists look like you'd expect:
    
    1. hi there
    
    2. must be going
    
    Urls are http://like.this and links can be
    written `like this <http://www.example.com/foo/bar>`_.
    

    You might also consider adding a “Contributors” section and/or a “Thanks also to” section to list the names of people who’ve helped.

    By the way, to see how the above README.txt looks rendered in html, see the TowelStuff project at the PyPI.

  • setup.py – Create this file and make it look like this:

    from distutils.core import setup
    
    setup(
        name='TowelStuff',
        version='0.1.0',
        author='J. Random Hacker',
        author_email='jrh@example.com',
        packages=['towelstuff', 'towelstuff.test'],
        scripts=['bin/stowe-towels.py','bin/wash-towels.py'],
        url='http://pypi.python.org/pypi/TowelStuff/',
        license='LICENSE.txt',
        description='Useful towel-related stuff.',
        long_description=open('README.txt').read(),
        install_requires=[
            "Django >= 1.1.1",
            "caldav == 0.1.4",
        ],
    )
    

    but, of course, replace the towel stuff with your own project and package names. For more details about picking version numbers, see versioning, but ‘0.1.0’ will work just fine for a first release (this is using the common “major.minor.micro” numbering convention).

    Use the install_requires argument to automatically install dependencies when your package will be installed and include information about dependencies (so that package management tools like Pip can use the information). It takes a string or list of strings containing requirement specifiers.

    The syntax consists of a project’s PyPI name, optionally followed by a comma-separated list of version specifiers. Modern packaging tools implement version specifiers syntax described in PEP 345 and resolve version comparison in compliance with PEP 386.

If you have no scripts to distribute (and thus no bin dir), you can remove the above line which begins with “scripts”.
  • Inside the towelstuff directory, __init__.py can be empty. Likewise, inside towelstuff/test, that __init__.py can be empty as well. If you have no tests written yet, you can leave the two other module files in towelstuff/test empty for now too. When writing your tests, use the standard unittest module.

For our example, TowelStuff does not depend upon any other distributions (it only depends upon what’s already in the Python standard library). To specify dependencies upon other distributions, see the more detailed Project example.

Creating your distribution file

Create your distribution file like so:

$ cd path/to/TowelStuff
$ python setup.py sdist

Running that last command will create a MANIFEST file in your project directory, and also a dist and build directory. Inside that dist directory is the distribution that you’ll be uploading to the PyPI. In our case, the distribution file will be named TowelStuff-0.1.0.tgz. Feel free to poke around in the dist directory to look at your distribution.

Uploading your distribution file

Before uploading you first need to create an account at http://pypi.python.org/pypi . Once that’s complete, register your distribution at the PyPI like so:

$ cd path/to/TowelStuff
$ python setup.py register

Use your existing login (choice #1). It will prompt you to save the login info for future use (to which I agree). Then upload:

$ python setup.py sdist upload

This builds the distribution one last time and then uploads it.

Thanks for your contribution!

Updating your distribution

Down the road, after you’ve made updates to your distribution and wish to make a new release:

  1. increment the version number in your setup.py file,
  2. update your CHANGES.txt file,
  3. if necessary, update the “Contributors” and “Thanks also to” sections of your README.txt file.
  4. run python setup.py sdist upload again.

Entry points

Entry points are a Setuptools/Distribute feature that’s really handy in one specific case: register something under a specific key in package A that package B can query for.

Distribute itself uses it. If you’re packaging your project up properly, you’ve probably used the console_scripts entry point:

setup(name='zest.releaser',
      ...
      entry_points={
          'console_scripts':
              ['release = zest.releaser.release:main',
               'prerelease = zest.releaser.prerelease:main',
               ]}
      )

console_scripts is an entry point that Setuptools looks up. It looks up all entry points registered under the name console_scripts and uses that information to generate scripts. In the above example that’d be a bin/release script that runs the main() method in zest/releaser/release.py.

You can use that for your own extension mechanism. For zest.releaser I needed some extension mechanism. I wanted to be able to do extra things on prerelease/release/postrelease time.

  • Downloading an external javascript library into a package that cannot be stored in (zope’s) svn repository directly due to licensing issues. Before packaging and releasing it, that is. Automatically so you don’t forget it.
  • Uploading a version.cfg to scp://somewhere/kgs/ourmainproduct-version.cfg after making a release to use it as a so-called “known good set” (KGS).
  • Possibly modifying values (like a commit message) inside zest.releaser itself while doing a release. (I do get modification requests from time to time “hey, can you make x and y configurable”). So now every zest.releaser step (prerelease, release, postrelease) is splitted in two: a calculation phase and a “doing” phase. The results of the first phase are stored in a dict that gets used in the second phase. And you can register an entry point that gets passed that dict so you can modify it. See the entry point documentation of zest.releaser for details.

An entry point for zest.releaser is configured like this in your setup.py:

entry_points={
    'console_scripts':
        ['myscript = my.package.scripts:main'],
    'zest.releaser.prereleaser.middle':
        ['dosomething = my.package.some:some_entrypoint, ]
}

Replace prereleaser and middle in zest.releaser.prereleaser.middle with prerelease/release/postrelease and before/middle/after where needed. (For this specific zest.releaser example).

Now, how to use this in your program? The best way is to show a quick example from zest.releaser where we query and use one of our entry points:

import pkg_resources

...
def run_entry_point(data):
    # Note: data is zest.releaser specific: we want to pass
    # something to the plugin group = 'zest.releaser.prerelease.middle'

    for entrypoint in pkg_resources.iter_entry_points(group=group):
        # Grab the function that is the actual plugin.
        plugin = entrypoint.load() # Call the plugin
        plugin(data)

So: pretty easy and simple way to allow other packages to register something that you want to know. Extra plugins, extra render methods, extra functionality you want to register in your web application, etcetera.

Packaging for a Particular Operating System (OS)

General Packaging Guidelines for Unix

General Packaging Guidelines for Windows