o

W Jannis Leidel

h jezdez@enn.io

d @enn_io
 6
How I learned to stop
worrying & love
Python Packaging
 6
1) The past of Python packaging
2) Common pitfalls and gotchas
3) Everyday software development
4) The future of Python packaging
Ambiguous terms

- Python package
A directory with Python files (module)
and a __init__.py file
- Release
A version of a specific software, result
of a development cycle, e.g. “Django 1.3”
- Distribution
Source or binary form of a release,
e.g. zipped tarball or Windows installer
 1
The past of Python
packaging, a brief
historical overview
Distutils history

“[..] a standard mechanism for building,

distributing, and installing Python
modules – or, more realistically, multi-
module distributions.”
Greg Ward

- creates distributions
- installs to library directory
- builds C extensions
- processes documentation
Distutils enhancements

- PEP 241 (2001)

Metadata in PKG-INFO files added to
with package distributions
- PEP 301 (2002)
Package Index and Trove classifiers
added and register command
- PEP 314 (2003)
Metadata 1.1 with license, platform,
download URL, dependencies fields
Distutils enhancements

- PEP 345 (2005-today)

Metadata 1.2, better dependencies
- PEP 376 (2009-today)
Database of Installed Python packages
- PEP 386 (2009-today)
Sane version comparison module and
better version handling on PyPI
 2
Common pitfalls and
gotchas of packaging
Python software
The standard setup.py
from  distutils.core  import  setup

setup(
       name='MyApp',
       version='0.1.0',
       description='My  app  that  does  things',
       author='John  Doe',
       author_email='johndoe@acme.com',
       license='BSD',
       packages=['myapp'],
       package_data  =  {
               'myapp':  [
                       'templates/*.html',
               ],
       },
)
Long description
Rendered on PyPI with docutils
reStructuredText renderer

import  codecs
def  readme(filename):
       file  =  codecs.open(filename,  'utf-­‐8')        
       return  unicode(file.read())

setup(
       #  ...
       long_description  =  readme('README.rst'),
)
package_data
Lists all additional, non-Python files of a
release to be installed from a distribution
setup(
       #  ...
       package_data  =  {
               'myapp':  [
                       'static/myapp/*/*',
                       'templates/myapp/*',
                       'locale/*/LC_MESSAGES/*'
               ],
   },
)
MANIFEST.in
Template for the “MANIFEST” file that
lists all files to be put in a distribution

include  README.rst
recursive-­‐include  docs  *.txt
recursive-­‐include  myapp/locale  *.mo  *.po
recursive-­‐include  myapp/static  *.css  *.png
recursive-­‐include  myapp/templates  *.html
Use sane versions, dammit!

- 0.18 “Catty And The Major”

- .000001
- “unreleased.unofficialdev”
Don’t do that.
- Use PEP 386 for formal versions, e.g.
“1.0.1” or “0.5a4” etc
- Use release names in any prose
documentation and changelogs,
e.g. super hero names
 3
Using packaging in
everyday’s software
development
System package management or not?

- Install binary packages globally using

your operatings system’s package
management, e.g. database adapters,
PIL, lxml
- Vendorize packages that require
modifications or short term patches
- Install often updated and your project
specific packages locally, e.g. Django,
pytz, your own app
virtualenv

- Isolated Python environments

- Create by topic, e.g. by “branch”, “client”,
“milestone”, “staging/prod” etc.
- Use virtualenv-wrapper for even more
shell helpers
- pip included in every virtual environment
pip

- Sane installation and uninstallation

- freezing and unfreezing of requirements
$  server1:~  pip  freeze  >  dev-­‐deps.txt

$  server2:~  pip  install  -­‐r  dev-­‐deps.txt

- VCS integration for quick and dirty use

- PyPI mirror support (PEP 381), e.g.
d.pypi.python.org
Simple own package index

- HTTP server with directory index feature,

e.g. Apache’s DirectoryIndex
- PyPI clone Chishop
pip  install  -­‐f  http://localhost/dists/

pip  install  -­‐i  http://pypi.corp.local/

- Use ~/.pip/pip.conf
[install]

index-­‐url  =  http://pypi.corp.local/
Local index

- Use ad-hoc HTTP server

$  python  -­‐m  SimpleHTTPServer  8000

- .pydistutils.cfg sdist dir

[sdist]
dist-­‐dir  =  /var/www/dists

- Enables pip download-cache

[global]
download-­‐cache  =  ~/.pip-­‐downloads
 4
The future
of packaging
with Distutils
What is distutils2?

- toolbox with reference implementation

for PEP 345, PEP 376, PEP 386
- the standalone pysetup tool that
installs and uninstalls distributions
(e.g. “pysetup install Django”)
- metadata available outside of Python file
in setup.cfg
- already in Python 3 trunk as “packaging”
Other distutils2 features

- defined requirements depending on

environment
Requires-­‐Dist:  bar;  python_version  ==  
'2.4'  or  python_version  ==  '2.5'
Requires-­‐Python:  >=2.5,<3

- Obsoleting other releases

Obsoletes-­‐Dist:  OtherProject  (<3.0)
Moving from distutils to distutils2

- “pysetup create” for conversion setup.py

to setup.cfg for forward compatibility
- “pysetup generate-setup” for conversion
setup.cfg to setup.py for backwards
compatibility
Example distutils2 setup.cfg
[metadata]
name  =  myapp
version  =  0.1
author  =  John  Doe
author-­‐email  =  john@doe.com
summary  =  My  awesome  app
description-­‐file  =  README.rst
home-­‐page  =  http://johndoe.com/p/myapp/
project-­‐url:  Repository,  http://code.johndoe.com/
myapp/
classifier  =  Development  Status  ::  3  -­‐  Alpha
License  ::  OSI  Approved  ::  MIT  License

[files]
packages  =  myapp
extra_files  =  setup.py  README
resources  =
     etc/settings.py.dist  {confdir}/myapp
Other files that are shipped with distutils2

- METADATA
- RECORD
a list of installed files (CSV formatting)
- RESOURCES
list of non-python files
- INSTALLER
name of the installer
- REQUESTED
exists if distribution wasn’t installed as
dependency
 6
1) Keep your setup.py files simple
2) Follow the standards
3) Use virtualenv/pip
4) Look out for distutils2/packaging
5) In doubt, read the
“Hitchhiker’s Guide to Packaging”
Thanks! o
Questions? W Jannis Leidel

h jezdez@enn.io
Don‘t forget to
d @enn_io

while we’re
in Amsterdam!