version qualifiers with CBI and git

When building an RCP app or feature, you don't want bundle qualifiers that change with every build, that removes the benefit of updating only what has changed. But you do want to be sure that a given version.qualifier is repeatable (built from the same source). One cool little feature that came out of the CBI work was a way to derive bundle qualifiers from git.

The Eclipse Project has been built with PDE build and map files for a long time now. The map file entry told PDE build what version and what location to check out during a build. This checks out org.eclipse.jface from CVS in one of our older builds, and will produce the plugin org.eclipse.jface_3.6.2.M20110210-1200.jar:

plugin@org.eclipse.jface=M20110210-1200,:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse,

With our move to git, we've updated them to GIT entries so they can be checked out:

plugin@org.eclipse.jface=GIT,tag=v20120817-083647,repo=git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git,path=bundles/org.eclipse.jface

This will produce org.eclipse.jface_3.8.101.v20120817-083647.jar. But the map based git fetch factory only works because it copies the plugins from the git repo that contains them to the buildDirectory/plugins directory. The tag used both for the qualifier and to check out a version of the bundle for the build.  But when there are multiple bundles are in a git repo and you can only checkout one tag at a time, the CVS pattern doesn't hold up for git.

When moving to the CBI based build, we'd be building all of the bundles in a git repo at the same time. In this scenario, it doesn't make sense to tie the bundle qualifier to a repository tag of some kind. The important functionality is 1) the qualifier doesn't change if the source code doesn't change and 2) the qualifier is repeatable when derived from the same build input.

That's where the notion of using the UTC commit timestamp of the last commit to touch a project came from. Tycho provides tycho-buildtimestamp-jgit, a timestamp provider that can be configured to work with tycho-packaging-plugin (in the 0.16.0 SNAPSHOT, I believe).

Example:

        <plugin>
          <groupId>org.eclipse.tycho</groupId>
          <artifactId>tycho-packaging-plugin</artifactId>
          <version>${tycho.version}</version>
          <dependencies>
            <dependency>
              <groupId>org.eclipse.tycho.extras</groupId>
              <artifactId>tycho-buildtimestamp-jgit</artifactId>
              <version>${tycho-extras.version}</version>
            </dependency>
          </dependencies>
          <configuration>
            <strictBinIncludes>false</strictBinIncludes>
            <format>'v'yyyyMMdd-HHmm</format>
            <timestampProvider>jgit</timestampProvider>
            <jgit.ignore>
            </jgit.ignore>
          </configuration>
        </plugin>

Now building our repo will generate predictable and reproducible qualifiers.  This will also help reduce the number of vDATE-TIME tags we push to our git repo, as only the build input commit needs to be tagged.  In 4.2.2 we get org.eclipse.jface_3.8.101.v20120914-1338.jar.

There are some drawback to this approach, but all in all it provides a qualifier convention that doesn't require a lot of configuration.

CBI and the eclipse project build

The CBI initiative is working at providing some infrastructure and templates to make it easy to get a new eclipse project up and running. So with a little setup, a new project can take advantage of some of the technologies (maven/tycho) and services (hudson.eclipse.org, eclipse jar signing) that the Eclipse Foundation offers.

One of CBI's trial projects is the Eclipse Project build (now called Eclipse Classic SDK :-) It's like the poster child for build edge conditions.

Right now the Eclipse project is built with a rather large set of PDE build scripts. But as we recently went through a lot of work trying to get them to build on eclipse.org, I can attest to the fact that they're not terribly portable.

With the need for various organizations to be able to build eclipse in their own environments, the CBI initiative is taking advantage of the fact that the second person to build something with maven gets a (mostly :-) free ride.

After cloning a large git repo made up mostly of submodules, all you have to do is install 2 poms and you're off to the races:

bash$ git clone -b Juno_RC4_R4 --recursive  \
  ssh://git.eclipse.org/gitroot/cbi/platform-aggregator.git \
  R4_platform-aggregator

bash$ cd R4_platform-aggregator

bash$ mvn -f eclipse-parent/pom.xml \
  clean install \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

bash$ mvn -f maven-cbi-plugin/pom.xml \
  clean install \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

bash$ mvn \
  clean install \
  -Dmaven.test.skip=true \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

Once finished, your build repo is in R4_platform-aggregator/TMP/org.eclipse.sdk.epp/target/repository and some Eclipse SDK zips are in R4_platform-aggregator/TMP/org.eclipse.sdk.epp/target/products. These locations might move when we migrate some releng information over to eclipse.platform.releng.eclipsebuilder.

This is still a work in progress, so the build doesn't produce all of the build output of our tradition PDE-driven build. Work is still being done to get the compile levels correct, and figure out a replacement or callout to the customCallBacks.xml.

There are also some other hiccups, like having to run the build twice as the first time it fails with a compile error in org.eclipse.jface. But in short order you have yourself a product build.

For more information see the CBI wiki.

p2 can update your eclipse.ini

I regularly switch my base SDK install to try out new developer integration builds, and it's not to much trouble to get it set up correctly thanks to p2 and the director.

eclipse/eclipse \
-application org.eclipse.equinox.p2.director \
-noSplash \
-repository \
http://download.eclipse.org/eclipse/updates/4.2-I-builds,\
http://download.eclipse.org/e4/updates/0.12-I-builds,\
http://download.eclipse.org/releases/juno,\
http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/repository \
-installIUs \
org.apache.commons.jxpath/1.3.0.v200911051830,\
org.apache.batik.xml/1.6.0.v201011041432,\
org.dojotoolkit/1.6.1.v201105210659,\
org.eclipse.nebula.widgets.gallery/1.0.0.R_0_20090212,\
org.eclipse.egit.feature.group,\
org.eclipse.jgit.feature.group,\
org.eclipse.emf.sdk.feature.group,\
org.eclipse.xtext.sdk.feature.group,\
org.eclipse.wst.xml_ui.feature.feature.group

As you can see I use the director to set up my eclipse/running target with orbit bundles I need, with EGit, and with SDKs that I need like EMF, XText, XML Editors and Tools. I love a good automated process.

With all functionality in use and with 23 platform and e4 git repositories for EGit/JGit to work through, it's easy to get an OOME with eclipse default settings. Because I'm testing downloads instead of upgrades, that means I have to update my eclipse.ini every time. It's minor, but it breaks the automated process.

Since p2 provides the touchpoints that can modify the eclipse.ini already, it should be simple to hand-craft a local repo with an IU that can update the eclipse.ini as part of my director call.

   <unit id='toolingorg.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200' singleton='false'>
      <provides size='2'>
        <provided namespace='org.eclipse.equinox.p2.iu' name='toolingorg.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200'/>
        <provided namespace='toolingorg.eclipse.sdk.ide' name='org.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200'/>
      </provides>
      <requires>
        <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.sdk.ide' range='[0.0.0,5.0.0)'/>
      </requires>
      <touchpoint id='org.eclipse.equinox.p2.osgi' version='1.0.0'/>
      <touchpointData size='1'>
        <instructions size='2'>
          <instruction key='unconfigure'>
            addJvmArg(jvmArg:-Xms40m);addJvmArg(jvmArg:-Xmx384m);removeJvmArg(jvmArg:-Xms64m);removeJvmArg(jvmArg:-Xmx1024m);
          </instruction>
          <instruction key='configure'>
            removeJvmArg(jvmArg:-Xms40m);removeJvmArg(jvmArg:-Xmx384m);addJvmArg(jvmArg:-Xms64m);addJvmArg(jvmArg:-Xmx1024m);
          </instruction>
        </instructions>
      </touchpointData>
    </unit>

I added a .feature.group IU to make it stand out that the fix has been installed into this version of eclipse, and voila.