Proper exit code for buildout generated scripts

I recently ran into the issue that our Hudson installation didn’t report test failures. After digging for a bit, I discovered, that scripts installed by zc.recipe.egg don’t return the exit code for some cases. There is a bugreport at https://bugs.launchpad.net/zc.buildout/+bug/164629. In our case the generated coverage script depended on this and because the sys.exit was missing, the test failures were masked.

The following is a way to fix this in your buildouts now until it’s fixed in zc.buildout:

[buildout]
extensions = buildout.extensionscripts
extension-scripts =
    ${buildout:directory}/src/buildout-utils.py:patchScriptGeneration

The patchScriptGeneration function in src/buildout-utils.py looks like this:

def patchScriptGeneration(buildout):
    from zc.buildout import easy_install
    if not 'sys.exit(' in easy_install.script_template:
        easy_install.script_template = easy_install.script_template.replace(
            "%(module_name)s.%(attrs)s(%(arguments)s)",
            "sys.exit(%(module_name)s.%(attrs)s(%(arguments)s))")
Published in: on September 24, 2010 at 4:30 pm  Comments (1)  

mr.developer 1.3 – Fewer surprises

The last releases of mr.developer (1.2 and 1.3) reduce the amount of surprises.

The packages from auto-checkout are now automatically added and removed from the list of development packages when you switch buildout configurations. Before you had to run develop reset and rerun buildout for that to work. If you have an existing checkout, you may want to reset it, so that this change is picked up.

The last used buildout configuration is now read directly. That means if you change source declarations in your buildout configuration, then you don’t have to rerun buildout anymore for mr.developer to pick up those changes. You still have to run buildout after changing the develop status of a package, but that’s the same as with a plain buildout without mr.developer.

Published in: on November 15, 2009 at 4:20 pm  Comments (5)  

Snakes on a cat!

Since I started to tame the snakepit, I made many improvements to make it even less scary.

The python buildout now auto detects the platform you are on, so it works for many more systems then just Mac OS X without a custom configuration. On Mac OS X it automatically detects whether you are on Leopard or Snow Leopard. Since today it also works on Snow Leopard with 32bit CPUs.

Published in: on November 5, 2009 at 10:31 am  Comments (3)  

Controlling Solr with supervisord

A while ago I found a solution to run Solr with supervisord, without leaking java processes, which you get when you use the generated solr-instance script with supervisord.

You need something like this in your supervisord.conf:

[program:solr]
directory = ${solr-instance:jetty-destination}/..
command = java -jar start.jar
autostart = true
autorestart = true

The key is the directory option. With that the command is executed in that directory, which starts Solr correctly and doesn’t leave java processes when stopping it with supervisord.

I discovered today that Solr, when used with collective.recipe.solrinstance, by default logs every query to stdout.

The solution is to change the command like the following:

command = java -Djava.util.logging.config.file=${buildout:directory}/etc/solr-logging.properties -jar start.jar

Now you just need to add the solr-logging.properties file with something like this as content:

# Default global logging level:
.level = WARNING
Published in: on September 11, 2009 at 8:10 pm  Leave a Comment  

Speedboost with a SSD for my MacBook Pro

After reading a lot about SSDs and the speedup they can give to a computer, I decided to get one myself. A fellow developer added a SSD to his late 2006 MacBook Pro, wrote about it and gave lots of tips. Since I have a late 2008 Unibody MacBook Pro, I had to get a slightly different adapter to replace the SuperDrive with a HDD in the optical drive bay from NewmodeUS.

At first I wanted to order the Intel X-25M 80GB in June for ~330 Euro. Shortly after ordering it, the supplier told me it would take 2 weeks to deliver. Due to an unexpected invoice I decided to cancel the order. I’m glad I did. I now got the next generation Intel X-25M 160GB Postville for ~400 Euro.

Thanks to the guides from iFixIt and the new enclosures of the Unibody Macs, the replacement was quite easy.

Some tips for the replacement:

After removing the 8 screws of the bottom plate, lift it straight up and don’t tilt it too much. There are latches on the left and right which open with slight force. If you tilt the bottom plate, then you may bend the ventilation grille at the back, or damage something with it. I don’t think it’s a real danger, just be careful.

The black cable going above the SuperDrive to the back uses some adhesive, lift it off carefully.

The plug of the black cable needs to be pulled out sideways, I used a knife, but better is to use something made of plastic. I pulled it off by pushing it sideways alternating on both sides.

I had to leave off the upper half of the optical bay adapter or it wouldn’t have fit.

I only used one screw near the logic board to secure the adapter, since the SuperDrive has additional screw holes which aren’t on the adapter. To compensate, I fitted something between subwoofer (the black plastic thing below the SuperDrive) and the adapter to keep it in place.

At first I had replaced the HDD with the SSD and put the HDD in the optical bay adapter. That was a very tight fit and the HDD was much louder than before, because the noise goes through the DVD slot and the ventilation grille. I did this because I had read that the deep sleep mode doesn’t work when using the drive in the optical bay as the boot drive. After a while I got really annoyed by the noise and I switched the HDD and SSD. Unfortunately I can confirm that deep sleep doesn’t work if the boot drive is in the optical bay! For me it’s the lesser evil though. Maybe Snow Leopard will fix this, I hope to know soon🙂

Once I had everything in place, I booted up the computer and held “Alt” after the startup sound to select the other disk as the boot device. I then used SuperDuper! to copy the OS from the HDD to the SSD (except for some large folders like “Music” and “Movies”. After selecting the new drive as the startup drive in the system preferences, I rebooted …

I watched in awe while the system booted up in mere seconds, I logged in and all the icons in the menu bar warped in, the QuickSilver logo came and went in a second and the system was ready after about 30 seconds total since the restart!

Other places I really notice the SSD speedup are the following:

  • Mail.app starts up almost instantly and opening a folder with thousands of mails is a breeze now.
  • Opera (which I use for reading newsgroups) starts in just a few seconds instead of churning along. Updating the newsgroups also takes a fraction of the time before and is mostly limited by the internet connection now.
  • Searching with Spotlight is much faster, especially in Mail.app, where I sometimes don’t notice that all mails have been searched instead of only the current folder.
  • QuickSilver is much more responsive.
  • Search in Project with TextMate is much better (but still not as fast as ack on the HDD).
  • Opening lots of images in iPhoto or Aperture is much faster
  • In general I almost never experience short lags or delays anymore as I did before.

Oh and if the HDD goes to sleep the MBP is now really quiet, unless you run something CPU intensive and the fans go on🙂

Published in: on August 28, 2009 at 10:49 am  Comments (6)  

Keep trac of your subversion location

Many people use trac and Subversion together. Inside trac you can browse the contents of the Subversion repository. There is a plugin to link from that browser to your actual Subversion repository called SubversionLocationPlugin.

For the other way around you can use the following XSLT. Most of it just recreates the default Subversion look. The interesting part is the <xsl:element name="a"> in <xsl:template match="index">. There the link is created and it’s the place where you can set your own URL of your trac instance (the trac_prefix variable). To use this file, you add the SVNIndexXSLT option (see “Customizing the look” in the Subversion book) into you Apache configuration and set it to the url under which the file is accessible.

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="html"/>
  <xsl:template match="*"/>
  <xsl:template match="svn">
    <html>
      <head>
        <title>
          Revision <xsl:value-of select="index/@rev"/>:
          <xsl:value-of select="index/@path"/>
        </title>
      </head>
      <body>
        <xsl:apply-templates/>
        <hr noshade="noshade"/>
        <em>Powered by </em>
        <xsl:element name="a">
          <xsl:attribute name="href">
            <xsl:value-of select="@href"/>
          </xsl:attribute>
          <xsl:text>Subversion</xsl:text>
        </xsl:element>
        <xsl:text> </xsl:text>
        <xsl:value-of select="@version"/>.
      </body>
    </html>
  </xsl:template>
  <xsl:template match="index">
    <xsl:element name="a">
      <xsl:variable name="trac_prefix" select="'https://dev.example.com/browser'"/>
      <xsl:attribute name="href">
        <xsl:value-of select="concat($trac_prefix, @path)"/>
      </xsl:attribute>
      Jarn-Trac Location
    </xsl:element>
    <h2>
      Revision <xsl:value-of select="@rev"/>:
      <xsl:value-of select="@path"/>
    </h2>
    <ul>
      <xsl:apply-templates select="updir"/>
      <xsl:apply-templates select="dir"/>
      <xsl:apply-templates select="file"/>
    </ul>
  </xsl:template>
  <xsl:template match="updir">
    <li><a href="..">..</a></li>
  </xsl:template>
  <xsl:template match="dir">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
        <xsl:text>/</xsl:text>
      </xsl:element>
    </li>
  </xsl:template>
  <xsl:template match="file">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
      </xsl:element>
    </li>
  </xsl:template>
</xsl:stylesheet>

The following XSLT is slightly more complex. It has an added <xsl:choose> part which adds links to customer specific trac instances.

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="html"/>
  <xsl:template match="*"/>
  <xsl:template match="svn">
    <html>
      <head>
        <title>
          Revision <xsl:value-of select="index/@rev"/>:
          <xsl:value-of select="index/@path"/>
        </title>
      </head>
      <body>
        <xsl:apply-templates/>
        <hr noshade="noshade"/>
        <em>Powered by </em>
        <xsl:element name="a">
          <xsl:attribute name="href">
            <xsl:value-of select="@href"/>
          </xsl:attribute>
          <xsl:text>Subversion</xsl:text>
        </xsl:element>
        <xsl:text> </xsl:text>
        <xsl:value-of select="@version"/>.
      </body>
    </html>
  </xsl:template>
  <xsl:template match="index">
    <xsl:element name="a">
      <xsl:variable name="trac_prefix" select="'https://dev.example.com/browser'"/>
      <xsl:attribute name="href">
        <xsl:value-of select="concat($trac_prefix, @path)"/>
      </xsl:attribute>
      Jarn-Trac Location
    </xsl:element>
    <xsl:choose>
      <xsl:when test="starts-with(@path, '/customers/foo')">
        -
        <xsl:element name="a">
          <xsl:variable name="trac_prefix" select="'https://trac.example.com/foo/browser'"/>
          <xsl:attribute name="href">
            <xsl:value-of select="concat($trac_prefix, substring-after(@path, '/customers/foo'))"/>
          </xsl:attribute>
          Foo-Trac Location
        </xsl:element>
      </xsl:when>
      <xsl:when test="starts-with(@path, '/customers/bar')">
        -
        <xsl:element name="a">
          <xsl:variable name="trac_prefix" select="'https://trac.example.com/bar/browser'"/>
          <xsl:attribute name="href">
            <xsl:value-of select="concat($trac_prefix, substring-after(@path, '/customers/bar'))"/>
          </xsl:attribute>
          Bar-Trac Location
        </xsl:element>
      </xsl:when>
    </xsl:choose>
    <h2>
      Revision <xsl:value-of select="@rev"/>:
      <xsl:value-of select="@path"/>
    </h2>
    <ul>
      <xsl:apply-templates select="updir"/>
      <xsl:apply-templates select="dir"/>
      <xsl:apply-templates select="file"/>
    </ul>
  </xsl:template>
  <xsl:template match="updir">
    <li><a href="..">..</a></li>
  </xsl:template>
  <xsl:template match="dir">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
        <xsl:text>/</xsl:text>
      </xsl:element>
    </li>
  </xsl:template>
  <xsl:template match="file">
    <li>
      <xsl:element name="a">
        <xsl:attribute name="href">
          <xsl:value-of select="@href"/>
        </xsl:attribute>
        <xsl:value-of select="@name"/>
      </xsl:element>
    </li>
  </xsl:template>
</xsl:stylesheet>

I hope this is useful for some people. Since I’ve never used XSLT before and I struggled quite a bit with building these, I would love some feedback on my solution and possible improvements or simplifications.

Published in: on August 9, 2009 at 11:09 am  Leave a Comment  

A different fridge to store your eggs

Today I wanted to release a new version of ImageRepository. For quite some time now, I didn’t update the product pages at plone.org, because it was very time consuming and cumbersome. But recently plone.org was upgraded to a newer Plone version and the PloneSoftwareCenter was updated to a version which supports the PyPI API for egg uploads.

Python 2.6 contains updates which add the mupload and mregister commands to distutils. For older versions, which are still in common use in the Zope/Plone world, there is collective.dist which adds the same commands. After I followed the instructions to update my ~/.pypirc file, I tried to register. Unfortunately by default the egg name is normalized to “products-imagerepository”, but the original location is named just “imagerepository”. I was told by David Glick that there is a tab for distutils settings in the project edit page. After I set the primary distutils id to “Products.ImageRepository”, the mregister command worked fine! Now I just had to remove the download_url setting from setup.py, so there would be no external link created on each registration.

So, after all these steps I finally made a 0.9 release of ImageRepository and it was easily uploaded to both plone.org and PyPI.

Published in: on February 12, 2009 at 4:32 pm  Comments (3)  

Where is that egg?

In the Zope/Plone world working with zc.buildout is the norm nowadays. Many people use an egg cache for speed and disk space preservation. That way eggs aren’t stored directly in your buildout anymore and it can by cumbersome to get there location from the commandline. One way to solve this is collective.recipe.omelette but that only exposes the paths for all eggs at once. I wrote a small shell function to solve that issue.

function eggpath() {
        local name=$1;
        shift;
        local files=$*;
        if [ -z $files ]; then
                if [ -e ./bin/instance* ]; then
                        local files="$files ./bin/instance*";
                fi
                if [ -e ./bin/client* ]; then
                        local files="$files ./bin/client*";
                fi
        fi
        grep -o -iE "/.*$name.*.egg" $files | sort -u;
}

With this you can get the path of an egg from any script. First argument is a regexp which is used for matching the name, most of the time this is a simple string with a part of the egg name. Optionally you can provide the name of the script(s) which should be checked, if it’s not given the function will look for bin/instance* or bin/client* scripts by default, which are the most common names for the Zope scripts in Plone buildouts.

My most common use case for this is for opening the egg with my editor like this: mate `eggpath some.egg`.

I hope this is useful for other people as well.

Published in: on February 4, 2009 at 3:56 pm  Comments (1)  

A buildout to tame the snake pit

After reading philiKON’s post about building Python 2.5 with readline on Mac OS X Leopard and getting my new MacBook Pro, I decided to create a buildout which does all the hard work and is reproducible when I managed to mess up the Python installation.

If you know how to run a buildout and don’t want to know any of the details, then just fetch it from http://svn.plone.org/svn/collective/buildout/python-macosx/. It will build Python 2.4, 2.5, 2.6 and 3.0. For the 2.x Python’s it will also create a virtualenv and install zc.buildout and PIL into it. A script called 'install-links' will be created, which will link the main scripts and binaries into /opt/local/bin when run.

The biggest issue I ran into while creating the buildout was getting the readline library properly picked up by the configuration part of the Python build. The only way I got it working, was to use a common installation prefix for readline and Python.

The buildout does several things:

  • It builds Python 2.4-2.6 and Python 3.0 with readline support
  • It creates virtualenv’s for the 2.x Python’s
  • It installs zc.buildout and PIL into those virtualenv’s (You should make sure that any dependencies for PIL are already installed, on Leopard with just XCode, you need to install the jpeg library. I use MacPorts for that, with the following command: sudo port install jpeg.)
  • It creates a script which links the main scripts and binaries into /opt/local/bin

You can easily customize the destination of the links by using a custom cfg file which extends the main buildout.cfg one like this:

[buildout]
extends = buildout.cfg
[install-links]
prefix = /opt/local
The /bin will be appended by default.
Published in: on December 14, 2008 at 4:16 pm  Comments (11)  
Tags: , ,
Follow

Get every new post delivered to your Inbox.