Proper exit code for buildout generated scripts

I recently ran into the issue that our Hudson installation didn’t report test failures. After digging for a bit, I discovered, that scripts installed by zc.recipe.egg don’t return the exit code for some cases. There is a bugreport at https://bugs.launchpad.net/zc.buildout/+bug/164629. In our case the generated coverage script depended on this and because the sys.exit was missing, the test failures were masked.

The following is a way to fix this in your buildouts now until it’s fixed in zc.buildout:

[buildout]
extensions = buildout.extensionscripts
extension-scripts =
    ${buildout:directory}/src/buildout-utils.py:patchScriptGeneration

The patchScriptGeneration function in src/buildout-utils.py looks like this:

def patchScriptGeneration(buildout):
    from zc.buildout import easy_install
    if not 'sys.exit(' in easy_install.script_template:
        easy_install.script_template = easy_install.script_template.replace(
            "%(module_name)s.%(attrs)s(%(arguments)s)",
            "sys.exit(%(module_name)s.%(attrs)s(%(arguments)s))")
Published in: on September 24, 2010 at 4:30 pm  Comments (1)  

mr.developer 1.3 – Fewer surprises

The last releases of mr.developer (1.2 and 1.3) reduce the amount of surprises.

The packages from auto-checkout are now automatically added and removed from the list of development packages when you switch buildout configurations. Before you had to run develop reset and rerun buildout for that to work. If you have an existing checkout, you may want to reset it, so that this change is picked up.

The last used buildout configuration is now read directly. That means if you change source declarations in your buildout configuration, then you don’t have to rerun buildout anymore for mr.developer to pick up those changes. You still have to run buildout after changing the develop status of a package, but that’s the same as with a plain buildout without mr.developer.

Published in: on November 15, 2009 at 4:20 pm  Comments (5)  

Snakes on a cat!

Since I started to tame the snakepit, I made many improvements to make it even less scary.

The python buildout now auto detects the platform you are on, so it works for many more systems then just Mac OS X without a custom configuration. On Mac OS X it automatically detects whether you are on Leopard or Snow Leopard. Since today it also works on Snow Leopard with 32bit CPUs.

Published in: on November 5, 2009 at 10:31 am  Comments (3)  

Controlling Solr with supervisord

A while ago I found a solution to run Solr with supervisord, without leaking java processes, which you get when you use the generated solr-instance script with supervisord.

You need something like this in your supervisord.conf:

[program:solr]
directory = ${solr-instance:jetty-destination}/..
command = java -jar start.jar
autostart = true
autorestart = true

The key is the directory option. With that the command is executed in that directory, which starts Solr correctly and doesn’t leave java processes when stopping it with supervisord.

I discovered today that Solr, when used with collective.recipe.solrinstance, by default logs every query to stdout.

The solution is to change the command like the following:

command = java -Djava.util.logging.config.file=${buildout:directory}/etc/solr-logging.properties -jar start.jar

Now you just need to add the solr-logging.properties file with something like this as content:

# Default global logging level:
.level = WARNING
Published in: on September 11, 2009 at 8:10 pm  Comments (2)  

Keep trac of your subversion location

Many people use trac and Subversion together. Inside trac you can browse the contents of the Subversion repository. There is a plugin to link from that browser to your actual Subversion repository called SubversionLocationPlugin.

For the other way around you can use the following XSLT. Most of it just recreates the default Subversion look. The interesting part is the <xsl:element name="a"> in <xsl:template match="index">. There the link is created and it’s the place where you can set your own URL of your trac instance (the trac_prefix variable). To use this file, you add the SVNIndexXSLT option (see “Customizing the look” in the Subversion book) into you Apache configuration and set it to the url under which the file is accessible.

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="html"/>
  <xsl:template match="*"/>
  <xsl:template match="svn">
    <html>
      <head>
        <title>
          Revision <xsl:value-of select="index/@rev"/>:
          <xsl:value-of select="index/@path"/>
        </title>
      </head>
      <body>
        <xsl:apply-templates/>
        <hr noshade="noshade"/>
        <em>Powered by </em>
        <xsl:element name="a">
          <xsl:attribute name="href">
            <xsl:value-of select="@href"/>
          </xsl:attribute>
          <xsl:text>Subversion</xsl:text>
        </xsl:element>
        <xsl:text> </xsl:text>
        <xsl:value-of select="@version"/>.
      </body>
    </html>
  </xsl:template>
  <xsl:template match="index">
    <xsl:element name="a">
      <xsl:variable name="trac_prefix" select="'https://dev.example.com/browser'"/>
      <xsl:attribute name="href">
        <xsl:value-of select="concat($trac_prefix, @path)"/>
      </xsl:attribute>
      Jarn-Trac Location
    </xsl:element>
    <h2>
      Revision <xsl:value-of select="@rev"/>:
      <xsl:value-of select="@path"/>
    </h2>
    <ul>
      <xsl:apply-templates select="updir"/>
      <xsl:apply-templates select="dir"/>
      <xsl:apply-templates select="file"/>
    </ul>
  </xsl:template>
  <xsl:template match="updir">
    <li><a href="..">..</a></li>
  </xsl:template>
  <xsl:template match="dir">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
        <xsl:text>/</xsl:text>
      </xsl:element>
    </li>
  </xsl:template>
  <xsl:template match="file">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
      </xsl:element>
    </li>
  </xsl:template>
</xsl:stylesheet>

The following XSLT is slightly more complex. It has an added <xsl:choose> part which adds links to customer specific trac instances.

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="html"/>
  <xsl:template match="*"/>
  <xsl:template match="svn">
    <html>
      <head>
        <title>
          Revision <xsl:value-of select="index/@rev"/>:
          <xsl:value-of select="index/@path"/>
        </title>
      </head>
      <body>
        <xsl:apply-templates/>
        <hr noshade="noshade"/>
        <em>Powered by </em>
        <xsl:element name="a">
          <xsl:attribute name="href">
            <xsl:value-of select="@href"/>
          </xsl:attribute>
          <xsl:text>Subversion</xsl:text>
        </xsl:element>
        <xsl:text> </xsl:text>
        <xsl:value-of select="@version"/>.
      </body>
    </html>
  </xsl:template>
  <xsl:template match="index">
    <xsl:element name="a">
      <xsl:variable name="trac_prefix" select="'https://dev.example.com/browser'"/>
      <xsl:attribute name="href">
        <xsl:value-of select="concat($trac_prefix, @path)"/>
      </xsl:attribute>
      Jarn-Trac Location
    </xsl:element>
    <xsl:choose>
      <xsl:when test="starts-with(@path, '/customers/foo')">
        -
        <xsl:element name="a">
          <xsl:variable name="trac_prefix" select="'https://trac.example.com/foo/browser'"/>
          <xsl:attribute name="href">
            <xsl:value-of select="concat($trac_prefix, substring-after(@path, '/customers/foo'))"/>
          </xsl:attribute>
          Foo-Trac Location
        </xsl:element>
      </xsl:when>
      <xsl:when test="starts-with(@path, '/customers/bar')">
        -
        <xsl:element name="a">
          <xsl:variable name="trac_prefix" select="'https://trac.example.com/bar/browser'"/>
          <xsl:attribute name="href">
            <xsl:value-of select="concat($trac_prefix, substring-after(@path, '/customers/bar'))"/>
          </xsl:attribute>
          Bar-Trac Location
        </xsl:element>
      </xsl:when>
    </xsl:choose>
    <h2>
      Revision <xsl:value-of select="@rev"/>:
      <xsl:value-of select="@path"/>
    </h2>
    <ul>
      <xsl:apply-templates select="updir"/>
      <xsl:apply-templates select="dir"/>
      <xsl:apply-templates select="file"/>
    </ul>
  </xsl:template>
  <xsl:template match="updir">
    <li><a href="..">..</a></li>
  </xsl:template>
  <xsl:template match="dir">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
        <xsl:text>/</xsl:text>
      </xsl:element>
    </li>
  </xsl:template>
  <xsl:template match="file">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
      </xsl:element>
    </li>
  </xsl:template>
</xsl:stylesheet>

I hope this is useful for some people. Since I’ve never used XSLT before and I struggled quite a bit with building these, I would love some feedback on my solution and possible improvements or simplifications.

Published in: on August 9, 2009 at 11:09 am  Leave a Comment  

A different fridge to store your eggs

Today I wanted to release a new version of ImageRepository. For quite some time now, I didn’t update the product pages at plone.org, because it was very time consuming and cumbersome. But recently plone.org was upgraded to a newer Plone version and the PloneSoftwareCenter was updated to a version which supports the PyPI API for egg uploads.

Python 2.6 contains updates which add the mupload and mregister commands to distutils. For older versions, which are still in common use in the Zope/Plone world, there is collective.dist which adds the same commands. After I followed the instructions to update my ~/.pypirc file, I tried to register. Unfortunately by default the egg name is normalized to “products-imagerepository”, but the original location is named just “imagerepository”. I was told by David Glick that there is a tab for distutils settings in the project edit page. After I set the primary distutils id to “Products.ImageRepository”, the mregister command worked fine! Now I just had to remove the download_url setting from setup.py, so there would be no external link created on each registration.

So, after all these steps I finally made a 0.9 release of ImageRepository and it was easily uploaded to both plone.org and PyPI.

Published in: on February 12, 2009 at 4:32 pm  Comments (3)  

Where is that egg?

In the Zope/Plone world working with zc.buildout is the norm nowadays. Many people use an egg cache for speed and disk space preservation. That way eggs aren’t stored directly in your buildout anymore and it can by cumbersome to get there location from the commandline. One way to solve this is collective.recipe.omelette but that only exposes the paths for all eggs at once. I wrote a small shell function to solve that issue.

function eggpath() {
        local name=$1;
        shift;
        local files=$*;
        if [ -z $files ]; then
                if [ -e ./bin/instance* ]; then
                        local files="$files ./bin/instance*";
                fi
                if [ -e ./bin/client* ]; then
                        local files="$files ./bin/client*";
                fi
        fi
        grep -o -iE "/.*$name.*.egg" $files | sort -u;
}

With this you can get the path of an egg from any script. First argument is a regexp which is used for matching the name, most of the time this is a simple string with a part of the egg name. Optionally you can provide the name of the script(s) which should be checked, if it’s not given the function will look for bin/instance* or bin/client* scripts by default, which are the most common names for the Zope scripts in Plone buildouts.

My most common use case for this is for opening the egg with my editor like this: mate `eggpath some.egg`.

I hope this is useful for other people as well.

Published in: on February 4, 2009 at 3:56 pm  Comments (1)  

A buildout to tame the snake pit

After reading philiKON’s post about building Python 2.5 with readline on Mac OS X Leopard and getting my new MacBook Pro, I decided to create a buildout which does all the hard work and is reproducible when I managed to mess up the Python installation.

If you know how to run a buildout and don’t want to know any of the details, then just fetch it from http://svn.plone.org/svn/collective/buildout/python-macosx/. It will build Python 2.4, 2.5, 2.6 and 3.0. For the 2.x Python’s it will also create a virtualenv and install zc.buildout and PIL into it. A script called 'install-links' will be created, which will link the main scripts and binaries into /opt/local/bin when run.

The biggest issue I ran into while creating the buildout was getting the readline library properly picked up by the configuration part of the Python build. The only way I got it working, was to use a common installation prefix for readline and Python.

The buildout does several things:

  • It builds Python 2.4-2.6 and Python 3.0 with readline support
  • It creates virtualenv’s for the 2.x Python’s
  • It installs zc.buildout and PIL into those virtualenv’s (You should make sure that any dependencies for PIL are already installed, on Leopard with just XCode, you need to install the jpeg library. I use MacPorts for that, with the following command: sudo port install jpeg.)
  • It creates a script which links the main scripts and binaries into /opt/local/bin

You can easily customize the destination of the links by using a custom cfg file which extends the main buildout.cfg one like this:

[buildout]
extends = buildout.cfg
[install-links]
prefix = /opt/local
The /bin will be appended by default.
Published in: on December 14, 2008 at 4:16 pm  Comments (11)  
Tags: , ,