Contribute Your Package to the World¶
The Python Package Index (PyPI)¶
XXX put here the Monty Python Cheeseshop extract
Tools like Pip or zc.buildout are using PyPI as the default
location to find distributions to install. When
install Foo is called, it will browse PyPI to find the latest available
version of the
Foo project using The Simple Index Protocol. If it finds
it, it will download it and install it automatically.
This automatic installation ala Pip Installs Python (Pip) will work only if the distribution is using the Distutils-based structure and contains a
This means that any serious Python project should use Distutils (see the basics for doing this) and should at the minimum be registered at PyPI. Uploading releases there is also a good practice.
??? Why should a project at a minimum register at PyPI? good practice, yes; at a minimum, no
Registering a project at PyPI is done by using Distutils‘
command. This command performs a simple HTTP post using basic authentication
with the login name and password stored in the
located in your home directory. This login has to be registered at PyPI, so
you should go there and create an account before running
register for the
Another option is to call
register once on any Distutils-based project.
It will register the new account for you and all you’ll have to do is to reply
to the confirmation e-mail PyPI will send you:
$ python setup.py register running register warning: register: missing required meta-data: version We need to know who you are, so please choose either: 1. use your existing login, 2. register as a new user, 3. have the server generate a new password for you (and email it to you), or 4. quit Your selection [default 1]: ...
Once this is done,
register will ask you if you want to save your login
information in the
.pypirc file. By default, this will store the login name
and the password:
[distutils] index-servers = pypi [pypi] username:tarek password:sigourney_as_an_avatar_is_scary
For security reasons, starting at Python 2.6, you can remove the password from
the file if you want
register to prompt you to type it everytime.
A recent GSoC project called Keyring was created in 2009 in order
to use any available system keyring like KWallet or KeyChain to
store the PyPI password. The project exists and may be used in Distutils
Once your account is ready, registering it at PyPI will create a new web page there, using the metadata fields of your project.
- name will be used as the unique identifier in PyPI. For example, if
your project’s name is
Foo, its page will be located at http://pypi.python.org/pypi/foo. The rule at PyPI is first in, first served, meaning that once you have registered the Foo project, your login is the owner of the PyPI foo identifier and no one else (unless you authorize them) will be able to register a project under that name.
- long_description will be used to fill that page. An Html to reStructuredText rendered will be used on the field before it is displayed.
A good practice is to use reST, and make sure your
field doesn’t contain any reStructuredText syntax error. See
rest_example for a quick introduction on how to write a reST compatible
To perform a check, install
docutils by using Pip
pip install docutils) and run:
$ python setup.py --long-description | rst2html.py > /dev/null
Under Windows, make sure the sys.prefix + ‘Scripts/’ path is in the
environment variable and run:
$ python.exe setup.py --long-description | rst2html.py > dummy.html
If your text contains any reST error or warning, they will be displayed.
Starting at Python 2.7, you can use the
check command instead of calling
rst2html.py script, as long as
docutils is installed:
$ python setup.py check
The check command will check that all fields are compliant before you register the project at PyPI.
PyPI also allows developers to upload their project’s distributions. This can
be done manually via a web form, but also through Distutils by using the
This command will upload freshly created archives via HTTP to PyPI. The
usual way to perform this is to call
upload right after
and the commands used to create archives. For instance, to upload a source
distribution and update the project’s page, one may do:
$ python setup.py register sdist upload
Note that you can upload several archives in one step if wanted:
$ python setup.py register sdist bdist upload
A good practice when uploading distributions at PyPI is to always upload the source distribution, unless your project is not open source of course. Binary distribution are optional especially if your project doesn’t contain any extension to be compiled. This will help automatic installers like Pip to get and install your project on any platform. Uploading only a binary distribution will restrict automatic installation to the platform and Python version it was compiled with.
The Simple Index protocol¶
For example, calling:
$ pip install distribute
will look at the
http://pypi.python.org/simple/distribute page, which
is a list of URLs for the Distribute project. These URLs include:
- the Distribute archives uploaded at PyPI
- a link to the project’s home page
- extra links contained in the project’s description fields
The Package Index follows these rules (taken from Setuptools’ documentation):
Its pages are in HTML
Individual project version pages’ URLs must be of the form
baseis the package index’s base URL. The base URL for PyPI is :
/versionpart of a project page’s URL (but keeping the trailing
/) should result in a page that is either:
- The single active version of that project, as though the version had been explicitly included, OR
- A page with links to all of the active version pages for that project.
Depending on the project’s configuration, PyPI will display a) or b).
Individual project version pages should contain direct links to downloadable distributions where possible. The project’s “long_description” field may contain URLs that will be displayed.
Where available, MD5 information should be added to download URLs by appending a fragment identifier of the form
...is the 32-character hex MD5 digest.
Individual project version pages should identify any “homepage” or “download” URLs using
rel="download"attributes on the HTML elements linking to those URLs. Use of these attributes will cause EasyInstall to always follow the provided links, unless it can be determined by inspection that they are downloadable distributions. If the links are not to downloadable distributions, they are retrieved, and if they are HTML, they are scanned for download links. They are not scanned for additional “homepage” or “download” links, as these are only processed for pages that are part of a package index site.
The root URL of the index, if retrieved with a trailing
/, must result in a page containing links to all projects’ active version pages.
(Note: This requirement is a workaround for the absence of case-insensitive
safe_name()matching of project names in URL paths. If project names are matched in this fashion (e.g. via the PyPI server, mod_rewrite, or a similar mechanism), then it is not necessary to include this all-packages listing page.)
If a package index is accessed via a
file://URL, then EasyInstall will automatically use
index.htmlfiles, if present, when trying to read a directory with a trailing
/on the URL.
The XML-RPC interface¶
XXX pointer to PyPI developer doc + demo with yolk
Limitations of the system¶
Within the last few years, PyPI has become a first class citizen in the Python
development ecosystem. For instance Plone developers that are using
zc.buildout to install Plone-based applications are making hundreds
of calls to PyPI each time they build their projects. That’s because many web
applications are using hundreds of small Python distributions nowadays.
This has a bad side effect: if PyPI is down, it becomes impossible to build
some applications. PyPI acts as a
Single Point of Failure (SPOF).
The PyPI server is quite robust and its uptime is probably around 99.99%, although people hit the SPOF problem once or twice per year. The community has started to develop strategies to avoid it:
zc.buildoutcomes with a cache that will keep a copy of downloaded archives on the local system. This avoids getting packages at PyPI when they were already downloaded once.
- the collective.eggproxy
project provides a cache server that acts like
a proxy between developers and PyPI, and caches PyPI files like
- the z3c.pypimirror project is a mirroring script that syncs a local copy of the simple index so tools can use it instead of calling PyPI.
In the last year, PEP 381 was added to describe a mirroring protocol and
set up an official “ring” of PyPI mirrors. This system should be ready
by the end of the year 2010, and will provide a DNS entry under
mirrors.pypi.org listing all mirrors IPs. This will allow tools like
Pip Installs Python (Pip) to list all mirrors and pick the closest one, or fall back in
case a server is down.
Private and secondary PyPI servers¶
XXX explain here that other servers can have a pypi feature (like plone.org, or private)
XXX explain that pip can point to any pypi-like server
XXX conclude on the multiple-index merge idea