A buildout to tame the snake pit

After reading philiKON’s post about building Python 2.5 with readline on Mac OS X Leopard and getting my new MacBook Pro, I decided to create a buildout which does all the hard work and is reproducible when I managed to mess up the Python installation.

If you know how to run a buildout and don’t want to know any of the details, then just fetch it from http://svn.plone.org/svn/collective/buildout/python-macosx/. It will build Python 2.4, 2.5, 2.6 and 3.0. For the 2.x Python’s it will also create a virtualenv and install zc.buildout and PIL into it. A script called 'install-links' will be created, which will link the main scripts and binaries into /opt/local/bin when run.

The biggest issue I ran into while creating the buildout was getting the readline library properly picked up by the configuration part of the Python build. The only way I got it working, was to use a common installation prefix for readline and Python.

The buildout does several things:

  • It builds Python 2.4-2.6 and Python 3.0 with readline support
  • It creates virtualenv’s for the 2.x Python’s
  • It installs zc.buildout and PIL into those virtualenv’s (You should make sure that any dependencies for PIL are already installed, on Leopard with just XCode, you need to install the jpeg library. I use MacPorts for that, with the following command: sudo port install jpeg.)
  • It creates a script which links the main scripts and binaries into /opt/local/bin

You can easily customize the destination of the links by using a custom cfg file which extends the main buildout.cfg one like this:

[buildout]
extends = buildout.cfg
[install-links]
prefix = /opt/local
The /bin will be appended by default.
Advertisements
Published in: on December 14, 2008 at 4:16 pm  Comments (11)  
Tags: , ,

The URI to TrackBack this entry is: https://fschulze.wordpress.com/2008/12/14/a-buildout-to-tame-the-snake-pit/trackback/

RSS feed for comments on this post.

11 CommentsLeave a comment

  1. rocks! 🙂

  2. Thanks! Working like a charm.

  3. Note, that PIL will build without JPEG support unless you have the proper libs installed first.

    For Macport users e.g. this means you first need to execute the following port command:

    sudo port install jpeg

  4. You also need to install wget on macos in order for the buildout to run. Maybe you could detect OS and if so use ‘ftp’ command to fetch the urls instead?

    -Matt

  5. Great buildout!

    If you dont have macports you can extends the buildout.cfg and give PIL support for JPEG
    this is part of my custom.cfg

    [buildout]
    extends = buildout.cfg
    parts =
    opt
    readline-patch
    readline
    python-2.4-build
    python-2.4-virtualenv
    python-2.4
    python-2.4-jpeg
    python-2.4-pil

    [python-2.4]
    command =
    ${buildout:directory}/bin/virtualenv-2.4 ${buildout:directory}/python-2.4
    ${buildout:directory}/python-2.4/bin/easy_install-2.4 -U zc.buildout

    [python-2.4-jpeg]
    recipe = hexagonit.recipe.cmmi
    url = http://www.ijg.org/files/jpegsrc.v6b.tar.gz
    configure-options =
    –prefix=${buildout:directory}/python-2.4
    make-targets =
    install-lib

    [python-2.4-pil]
    recipe = plone.recipe.command
    command =
    ${buildout:directory}/python-2.4/bin/easy_install-2.4 -f http://dist.repoze.org/plone/latest -U PIL
    update-command = ${python-2.4:command}

  6. I finally caved in and used the buildout a little while back. Works like a charm!

    The only problem that I’ve encountered is dependencies in MacPorts. For instance, I’m trying to install xorg-xcb-util which depends on the python26 port, I get this error:

    Error: Target org.macports.activate returned: Image error: /opt/local/bin/python2.6 already exists and does not belong to a registered port. Unable to activate port python26.

    Do you know if it is possible to tell MacPorts to use an alternate install for Python?

    • Florian and I discussed this on irc. It doesn’t look like MacPorts supports what I want here. It **has** to use the python it compiles.

      We decided that we’d let MacPorts build it’s own versions of Python as well. Then change the buildout to create a directory with all the executables we will need so that we can put that in our $PATH.

      If I get a chance I’ll modify the buildout to do just that. But if someone gets to it before me, I won’t shed a tear 😉

  7. You might want to increase stack size on generated python2.4 binary, because OSX default is too small for normal Zope usage: binary stack will run out before Python stack, causing silent death.

    http://homepage.mac.com/eric.c/hpc/contents/documentation/How%20to%20increase%20the%20stack%20size%20on%20Mac%20OS%20X.pdf

    After many tries yesterday I couldn’t fix it – Python still dies silently on stack overflow.

    The problem is that setting LDFLAGS don’t seem to have effect since Python makefile uses special C++ command to build the binary (name was python.exe for me).

    But even after I manually modified Makefile (in /var/), rebuild python.exe and verified new stack size, I couldn’t get this working.

    In Makefile:

    LINKFORSHARED= -u _PyMac_Error -Wl,-stack_size,0x10000000

    cs78177012:Python-2.4.6 moo$ size -m python.exe
    Segment __PAGEZERO: 4096
    Segment __TEXT: 909312
    Section __text: 832155
    Section __cstring: 69938
    Section __literal8: 552
    Section __const: 2560
    total 905205
    Segment __DATA: 159744
    Section __data: 147344
    Section __dyld: 28
    Section __const: 100
    Section __bss: 10032
    Section __common: 500
    total 158004
    Segment __UNIXSTACK: 268435456
    Segment __IMPORT: 4096
    Section __pointers: 924
    Section __jump_table: 1020
    total 1944
    Segment __LINKEDIT: 327680

  8. Edit python-2.4/lib/python2.4/site.py

    Put in sys.setrecursionlimit(300).
    This will force smaller Python stack not exceeding native OSX C stack.

  9. […] on a cat! Since I started to tame the snakepit, I made many improvements to make it even less […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: