Looking at High Availablility with Eclipse Orion

I've had the opportunity to do some work on Eclipse Orion lately, and I'm really enjoying it.  There are opportunities to learn about technology on both the server (Java and/or Nodejs) and the client side (HTML, CSS, and JS).  It's a lot to learn :-)

One of the things I've been looking at is High Availability in Orion 7.0.  If we use horizontal scaling, we can upgrade one server while leaving another server to continue to provide service (we're working with some kind of proxy in front of the servers, like NGINX, to direct traffic).

Anthony has been working on consolidating Orion metastore/metadata, on distributing upgrades on user access or background processes, and on a policy to cover how we add/upgrade new metadata properties so that older servers could still function against the new metadata.

I've been focused on allowing multiple servers to access the same user content and search indices at the same time.

By default, there's only one small change in Orion 7.0.  The search indices which used to reside in osgi.instance.area/.plugins/org.eclipse.orion.server.core.search will now reside in osgi.instance.area/.plugins/org.eclipse.orion.server.core.search/<version> (where the current version is v16).

You can use properties (See web-ide.conf) to activate the new configuration, where the user content and the search indices are in a common location but each server still has its own local osgi.instance.area.

For example, if I had 2 servers that had their local OSGi instance areas at /opt/orion/workspace (not shared between servers) and a common NFS mount at /home/data, I could set up the orion configuration on each server to serve up the same content using:

• orion.file.content.location=/home/data/orion/content
• orion.search.index.location=/home/data/orion/search
• orion.file.content.locking=true

The search indices and user content should not point to the same directory.  The file content locking uses file system (per process) locks to protect read-writes of user metadata and instructs only one process to run the search indexing jobs at a time.

We've done some preliminary testing, but we always welcome more testing and feedback.  If you get a chance to use this, please let me know :-)

version qualifiers with CBI and git

When building an RCP app or feature, you don't want bundle qualifiers that change with every build, that removes the benefit of updating only what has changed. But you do want to be sure that a given version.qualifier is repeatable (built from the same source). One cool little feature that came out of the CBI work was a way to derive bundle qualifiers from git.

The Eclipse Project has been built with PDE build and map files for a long time now. The map file entry told PDE build what version and what location to check out during a build. This checks out org.eclipse.jface from CVS in one of our older builds, and will produce the plugin org.eclipse.jface_3.6.2.M20110210-1200.jar:

plugin@org.eclipse.jface=M20110210-1200,:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse,

With our move to git, we've updated them to GIT entries so they can be checked out:

plugin@org.eclipse.jface=GIT,tag=v20120817-083647,repo=git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git,path=bundles/org.eclipse.jface

This will produce org.eclipse.jface_3.8.101.v20120817-083647.jar. But the map based git fetch factory only works because it copies the plugins from the git repo that contains them to the buildDirectory/plugins directory. The tag used both for the qualifier and to check out a version of the bundle for the build.  But when there are multiple bundles are in a git repo and you can only checkout one tag at a time, the CVS pattern doesn't hold up for git.

When moving to the CBI based build, we'd be building all of the bundles in a git repo at the same time. In this scenario, it doesn't make sense to tie the bundle qualifier to a repository tag of some kind. The important functionality is 1) the qualifier doesn't change if the source code doesn't change and 2) the qualifier is repeatable when derived from the same build input.

That's where the notion of using the UTC commit timestamp of the last commit to touch a project came from. Tycho provides tycho-buildtimestamp-jgit, a timestamp provider that can be configured to work with tycho-packaging-plugin (in the 0.16.0 SNAPSHOT, I believe).

Example:

        <plugin>
          <groupId>org.eclipse.tycho</groupId>
          <artifactId>tycho-packaging-plugin</artifactId>
          <version>${tycho.version}</version>
          <dependencies>
            <dependency>
              <groupId>org.eclipse.tycho.extras</groupId>
              <artifactId>tycho-buildtimestamp-jgit</artifactId>
              <version>${tycho-extras.version}</version>
            </dependency>
          </dependencies>
          <configuration>
            <strictBinIncludes>false</strictBinIncludes>
            <format>'v'yyyyMMdd-HHmm</format>
            <timestampProvider>jgit</timestampProvider>
            <jgit.ignore>
            </jgit.ignore>
          </configuration>
        </plugin>

Now building our repo will generate predictable and reproducible qualifiers.  This will also help reduce the number of vDATE-TIME tags we push to our git repo, as only the build input commit needs to be tagged.  In 4.2.2 we get org.eclipse.jface_3.8.101.v20120914-1338.jar.

There are some drawback to this approach, but all in all it provides a qualifier convention that doesn't require a lot of configuration.

CBI and the eclipse project build

The CBI initiative is working at providing some infrastructure and templates to make it easy to get a new eclipse project up and running. So with a little setup, a new project can take advantage of some of the technologies (maven/tycho) and services (hudson.eclipse.org, eclipse jar signing) that the Eclipse Foundation offers.

One of CBI's trial projects is the Eclipse Project build (now called Eclipse Classic SDK :-) It's like the poster child for build edge conditions.

Right now the Eclipse project is built with a rather large set of PDE build scripts. But as we recently went through a lot of work trying to get them to build on eclipse.org, I can attest to the fact that they're not terribly portable.

With the need for various organizations to be able to build eclipse in their own environments, the CBI initiative is taking advantage of the fact that the second person to build something with maven gets a (mostly :-) free ride.

After cloning a large git repo made up mostly of submodules, all you have to do is install 2 poms and you're off to the races:

bash$ git clone -b Juno_RC4_R4 --recursive  \
  ssh://git.eclipse.org/gitroot/cbi/platform-aggregator.git \
  R4_platform-aggregator

bash$ cd R4_platform-aggregator

bash$ mvn -f eclipse-parent/pom.xml \
  clean install \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

bash$ mvn -f maven-cbi-plugin/pom.xml \
  clean install \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

bash$ mvn \
  clean install \
  -Dmaven.test.skip=true \
  -Dmaven.repo.local=/opt/pwebster/lts/R4_localrepo

Once finished, your build repo is in R4_platform-aggregator/TMP/org.eclipse.sdk.epp/target/repository and some Eclipse SDK zips are in R4_platform-aggregator/TMP/org.eclipse.sdk.epp/target/products. These locations might move when we migrate some releng information over to eclipse.platform.releng.eclipsebuilder.

This is still a work in progress, so the build doesn't produce all of the build output of our tradition PDE-driven build. Work is still being done to get the compile levels correct, and figure out a replacement or callout to the customCallBacks.xml.

There are also some other hiccups, like having to run the build twice as the first time it fails with a compile error in org.eclipse.jface. But in short order you have yourself a product build.

For more information see the CBI wiki.

p2 can update your eclipse.ini

I regularly switch my base SDK install to try out new developer integration builds, and it's not to much trouble to get it set up correctly thanks to p2 and the director.

eclipse/eclipse \
-application org.eclipse.equinox.p2.director \
-noSplash \
-repository \
http://download.eclipse.org/eclipse/updates/4.2-I-builds,\
http://download.eclipse.org/e4/updates/0.12-I-builds,\
http://download.eclipse.org/releases/juno,\
http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/repository \
-installIUs \
org.apache.commons.jxpath/1.3.0.v200911051830,\
org.apache.batik.xml/1.6.0.v201011041432,\
org.dojotoolkit/1.6.1.v201105210659,\
org.eclipse.nebula.widgets.gallery/1.0.0.R_0_20090212,\
org.eclipse.egit.feature.group,\
org.eclipse.jgit.feature.group,\
org.eclipse.emf.sdk.feature.group,\
org.eclipse.xtext.sdk.feature.group,\
org.eclipse.wst.xml_ui.feature.feature.group

As you can see I use the director to set up my eclipse/running target with orbit bundles I need, with EGit, and with SDKs that I need like EMF, XText, XML Editors and Tools. I love a good automated process.

With all functionality in use and with 23 platform and e4 git repositories for EGit/JGit to work through, it's easy to get an OOME with eclipse default settings. Because I'm testing downloads instead of upgrades, that means I have to update my eclipse.ini every time. It's minor, but it breaks the automated process.

Since p2 provides the touchpoints that can modify the eclipse.ini already, it should be simple to hand-craft a local repo with an IU that can update the eclipse.ini as part of my director call.

   <unit id='toolingorg.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200' singleton='false'>
      <provides size='2'>
        <provided namespace='org.eclipse.equinox.p2.iu' name='toolingorg.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200'/>
        <provided namespace='toolingorg.eclipse.sdk.ide' name='org.eclipse.e4.fix.jvmArg' version='4.2.0.I20120131-2200'/>
      </provides>
      <requires>
        <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.sdk.ide' range='[0.0.0,5.0.0)'/>
      </requires>
      <touchpoint id='org.eclipse.equinox.p2.osgi' version='1.0.0'/>
      <touchpointData size='1'>
        <instructions size='2'>
          <instruction key='unconfigure'>
            addJvmArg(jvmArg:-Xms40m);addJvmArg(jvmArg:-Xmx384m);removeJvmArg(jvmArg:-Xms64m);removeJvmArg(jvmArg:-Xmx1024m);
          </instruction>
          <instruction key='configure'>
            removeJvmArg(jvmArg:-Xms40m);removeJvmArg(jvmArg:-Xmx384m);addJvmArg(jvmArg:-Xms64m);addJvmArg(jvmArg:-Xmx1024m);
          </instruction>
        </instructions>
      </touchpointData>
    </unit>

I added a .feature.group IU to make it stand out that the fix has been installed into this version of eclipse, and voila.

EGit and a shortlived topic branch

I've been experimenting with git workflows in Eclipse/EGit, trying to leverage some of the (supposed) power or functionality we get from using git for our SCM. Fortunately, the EGit folks were able to offer some hints as to how to make that happen.

The easiest way to get started is to create the topic branch off of your main remote branch, using Team>Switch To>New Branch. Because we want to avoid small merges, I already have branch.autosetuprebase=always set, and that's reflected in the defaults in the EGit Create Branch dialog. In my case I created pwebster/bug357532 off of origin/master. That way changes to master will be picked up and my local changes rebased automatically.

I also used the suggested pattern of <committerId>/branchName. We hope to have a receive hook in place soon that allows committers to delete their own topic branches if pushed to the public repo.

But if you aren't working on a feature branch, you probably want to push your topic branch commits back to your remote branch, in my case origin/master. You can do this by setting a push refspec on that remote. I went to my repository configuration in the Preferences>Team>Git>Configuration dialog and adding the following property: remote.origin.push=pwebster/bug358020:refs/heads/master.

Now for a short lived topic branch, you're set. Pulling will keep you up to date with origin/master, and you can push your fix to origin/master without having to do a lot of switching/fast-forward merging/pushing/deleting.

Where the git did that go?

I just came back from Paternity leave in the Eclipse SDK milestone (M3) week. The leave was great. Coming back was, as always, exciting. A slightly misconfigured tool had removed branches and tags (and an aggressive prune/gc had removed commits) during a push to our eclipse.platform.ui repo, leading to Bug 361707 - I have broken the platform-ui git repository. Because tags and commits were gone, we also broke all of the warmup I builds for our M3 week. My apologies to all that suffered.

These are all done by legitimate actions supported by git ... it just turns out that if you have a process backed by a central canonical repository, you really don't want your source disappearing out from under you. "I might have pushed back most of our repo" is not an acceptable level of support :-)

We were able to find a mostly complete repo, disable the Eclipse Foundation commit hook and push back the repo. We are discussing ways to limit this kind of damage in the future. See

• Bug 362076 - Better policy to guard against deleting all branches and tags from our public repos
• Bug 362361 - Better policy ... enforce receive.denyDeletes = true
• Bug 362363 - Better policy ... provide hooks to allow a committer to delete /branchname branches

Git Migration and tagging

The Eclipse SDK, or Eclipse Classic as it now tends to be called, has been building and releasing for just about 10 years, using the PDE Build system. Our releases are based on map files, which contain the IDs, SCM, and co-ordinates of the plugins we need to build.

When moving from CVS, where you can tag individual files, to git, where you tag a commit (the repo), this can cause headaches.

The common practice of tagging the projects that change for each week's I-builds means that cvs2git generates a master branch that looks like a fir tree :-) The delete commits remove all of the projects that weren't tagged. That makes the commit accurate, but that's not the shape of the git repo we want. A little scripting and we moved the tags from the delete commits back onto the main branch.

The other problem comes from the practice of tagging the release (R3_6) but only branching projects that change (R3_6_maintenance). In order for cvs2git to make the branch correct, the first commit on the R3_6_maintenance branch is delete commit. To make the important maintenance branches look correct, you can fix this one of 2 ways. Either rebase the branch around the delete commit (before you publish it to the public), or pre-condition the CVS repo so that all projects that are tagged R3_6 are branched R3_6_maintenance as well.

The scripts I used to fix Platform UI are in the org.eclipse.migration/scripts directory.