I've been experimenting with git workflows in Eclipse/EGit, trying to leverage some of the (supposed) power or functionality we get from using git for our SCM. Fortunately, the EGit folks were able to offer some hints as to how to make that happen.
The easiest way to get started is to create the topic branch off of your main remote branch, using Team>Switch To>New Branch. Because we want to avoid small merges, I already have branch.autosetuprebase=always set, and that's reflected in the defaults in the EGit Create Branch dialog. In my case I created pwebster/bug357532 off of origin/master. That way changes to master will be picked up and my local changes rebased automatically.
I also used the suggested pattern of <committerId>/branchName. We hope to have a receive hook in place soon that allows committers to delete their own topic branches if pushed to the public repo.
But if you aren't working on a feature branch, you probably want to push your topic branch commits back to your remote branch, in my case origin/master. You can do this by setting a push refspec on that remote. I went to my repository configuration in the Preferences>Team>Git>Configuration dialog and adding the following property: remote.origin.push=pwebster/bug358020:refs/heads/master.
Now for a short lived topic branch, you're set. Pulling will keep you up to date with origin/master, and you can push your fix to origin/master without having to do a lot of switching/fast-forward merging/pushing/deleting.
Tuesday, November 15, 2011
Wednesday, November 09, 2011
Where the git did that go?
I just came back from Paternity leave in the Eclipse SDK milestone (M3) week. The leave was great. Coming back was, as always, exciting. A slightly misconfigured tool had removed branches and tags (and an aggressive prune/gc had removed commits) during a push to our eclipse.platform.ui repo, leading to Bug 361707 - I have broken the platform-ui git repository. Because tags and commits were gone, we also broke all of the warmup I builds for our M3 week. My apologies to all that suffered.
These are all done by legitimate actions supported by git ... it just turns out that if you have a process backed by a central canonical repository, you really don't want your source disappearing out from under you. "I might have pushed back most of our repo" is not an acceptable level of support :-)
We were able to find a mostly complete repo, disable the Eclipse Foundation commit hook and push back the repo. We are discussing ways to limit this kind of damage in the future. See
These are all done by legitimate actions supported by git ... it just turns out that if you have a process backed by a central canonical repository, you really don't want your source disappearing out from under you. "I might have pushed back most of our repo" is not an acceptable level of support :-)
We were able to find a mostly complete repo, disable the Eclipse Foundation commit hook and push back the repo. We are discussing ways to limit this kind of damage in the future. See
- Bug 362076 - Better policy to guard against deleting all branches and tags from our public repos
- Bug 362361 - Better policy ... enforce receive.denyDeletes = true
- Bug 362363 - Better policy ... provide hooks to allow a committer to delete
/branchname branches
Tuesday, August 09, 2011
Git Migration and tagging
The Eclipse SDK, or Eclipse Classic as it now tends to be called, has been building and releasing for just about 10 years, using the PDE Build system. Our releases are based on map files, which contain the IDs, SCM, and co-ordinates of the plugins we need to build.
When moving from CVS, where you can tag individual files, to git, where you tag a commit (the repo), this can cause headaches.
The common practice of tagging the projects that change for each week's I-builds means that cvs2git generates a master branch that looks like a fir tree :-) The delete commits remove all of the projects that weren't tagged. That makes the commit accurate, but that's not the shape of the git repo we want. A little scripting and we moved the tags from the delete commits back onto the main branch.
The other problem comes from the practice of tagging the release (R3_6) but only branching projects that change (R3_6_maintenance). In order for cvs2git to make the branch correct, the first commit on the R3_6_maintenance branch is delete commit. To make the important maintenance branches look correct, you can fix this one of 2 ways. Either rebase the branch around the delete commit (before you publish it to the public), or pre-condition the CVS repo so that all projects that are tagged R3_6 are branched R3_6_maintenance as well.
The scripts I used to fix Platform UI are in the org.eclipse.migration/scripts directory.
When moving from CVS, where you can tag individual files, to git, where you tag a commit (the repo), this can cause headaches.
The common practice of tagging the projects that change for each week's I-builds means that cvs2git generates a master branch that looks like a fir tree :-) The delete commits remove all of the projects that weren't tagged. That makes the commit accurate, but that's not the shape of the git repo we want. A little scripting and we moved the tags from the delete commits back onto the main branch.
The other problem comes from the practice of tagging the release (R3_6) but only branching projects that change (R3_6_maintenance). In order for cvs2git to make the branch correct, the first commit on the R3_6_maintenance branch is delete commit. To make the important maintenance branches look correct, you can fix this one of 2 ways. Either rebase the branch around the delete commit (before you publish it to the public), or pre-condition the CVS repo so that all projects that are tagged R3_6 are branched R3_6_maintenance as well.
The scripts I used to fix Platform UI are in the org.eclipse.migration/scripts directory.
Friday, July 29, 2011
Git Migration and ignore files
As I mentioned in Git Migration I've been involved in the Platform UI CVS to Git migration using the cvs2git tool.
After your first successful conversion of a test repo, you fire it up in eclipse and check it out. That's when you see ... a lot of untracked changes! Oh yeah, all our .cvsignore files are useless :-)
Obviously we wanted a .gitignore at the root of our repo, from the beginning of time ... just like the catchphrase "there's an app for that", git will allow you to re-write your history so that it looks like the .gitignore was in every branch and tag since day one (How do I make a git commit in the past?). Obviously since this is a re-write, you need to do this as the last step before you publish your repo.
Here's the example as fed into bash:
Now I have a .gitignore at the root of my repo, and my status and EGit views are not cluttered with spurious untracked files. Rinse and repeat for more specialized .gitignores in various project directories that need them.
After your first successful conversion of a test repo, you fire it up in eclipse and check it out. That's when you see ... a lot of untracked changes! Oh yeah, all our .cvsignore files are useless :-)
Obviously we wanted a .gitignore at the root of our repo, from the beginning of time ... just like the catchphrase "there's an app for that", git will allow you to re-write your history so that it looks like the .gitignore was in every branch and tag since day one (How do I make a git commit in the past?). Obviously since this is a re-write, you need to do this as the last step before you publish your repo.
- create suitable file contents
- add it to your repo
- re-write all of the index trees in each commit to add the file
- reset your repo
Here's the example as fed into bash:
bash$ cat - >../gitignore <<EOF bin/ *~ *.rej *.bak *.patch javacore.* heapdump.* core.* Snap.* target/ EOF bash$ new_file=$(git hash-object -w ../gitignore) bash$ git filter-branch \ --index-filter \ 'git update-index --add --cacheinfo 100644 '"$new_file"' .gitignore' \ --tag-name-filter cat \ -- --all bash$ git reset --hard
Now I have a .gitignore at the root of my repo, and my status and EGit views are not cluttered with spurious untracked files. Rinse and repeat for more specialized .gitignores in various project directories that need them.
Wednesday, July 20, 2011
Git Migration
A large part of the Open Source world seems hot on git these days (and DVCSs in general). I can kinda see why. For tracking changes, the notion of a commit definitely makes more sense than updating the versions on files that CVS has to offer. The ease of creating git repos plays well with one of my own beliefs, which is "Even for one's own little projects, if it's not in a VCS it may as well be written on the back of a napkin". Social coding sites like GitHub offer an ease of contributing to projects that contributors love, and git makes it easier to keep contributions (commits) from going stale than when dealing with patches in email or bug tacking systems. Git itself has useful tools for visualizing your repo (like gitk), tracking down when a bad behaviour was introduced (git blame), and a dozen others besides.
And for those that miss their C roots, you can pretend your git hashes are pointers: 5eec2b04a70f704a7cdc1b77db59bb90ec03dd68 :-)
With the Eclipse Foundation also interested in supporting git as its main VCS, the Eclipse Project decided to move to git at the beginning of the Juno Release cycle. I've been involved with the Platform UI component migration.
I'll write a second post soon with more details about the problems we encountered and some of the steps we took, but after a grueling 2 weeks the conversion of this component is done. Woo Woo!
And for those that miss their C roots, you can pretend your git hashes are pointers: 5eec2b04a70f704a7cdc1b77db59bb90ec03dd68 :-)
With the Eclipse Foundation also interested in supporting git as its main VCS, the Eclipse Project decided to move to git at the beginning of the Juno Release cycle. I've been involved with the Platform UI component migration.
I'll write a second post soon with more details about the problems we encountered and some of the steps we took, but after a grueling 2 weeks the conversion of this component is done. Woo Woo!
- git://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git
- git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git
- git://git.eclipse.org/gitroot/e4/org.eclipse.e4.tools.git
Friday, June 17, 2011
p2 cheatsheet
I was writing a p2 mirror script suggestion for a colleague. When it morphed into a sorta p2 cheatsheet, I thought I'd blog about it (so I can find it again when I need it :-).
repo syntax
Most repos are specified via URLs like http:, but they can be files on the disk or even zipped repos:
Just a note: destinations are (almost) always normal directory paths, not URLs.
Mirroring repos
You can quickly (well, I/O bound :-) mirror 2 builds repos so you can compare them on your disk. Here's an ant script for current, important, build repos.
You can run that from an eclipse instance, either in eclipse using an Eclipse Application, or from the command line (my preference):
It'll spit out some messages about unsatisfied IUs, but that's OK because you're using the slicer, not the provisioner.
IUs and version numbers
If you're looking at plugin or IU versions, you can quickly get a list in the "IU=version" format:
If you pick a composite repo like http://download.eclipse.org/eclipse/updates/3.6 you can list all of the IUs, and then install a specific version of org.eclipse.sdk.ide, for example.
Creating a product for your platform
You can use the same scripts that the PDE build uses to generate a complete install for a platform:
The exact same thing can be accomplished using the command-line director (just with a lot of arguments :-)
Installing into your own eclipse
This is the shortest command line of all, and the easiest way to take a new eclipse and install features from scripts.
More info
More about the director command line: http://help.eclipse.org/
More about the p2 tasks and all the options they can take: http://help.eclipse.org/
repo syntax
Most repos are specified via URLs like http:, but they can be files on the disk or even zipped repos:
- http://download.eclipse.org/
eclipse/updates/3.7-I-builds - file:///home/data/httpd/
download.eclipse.org/eclipse/ updates/3.7-I-builds - jar:file:///opt/pwebster/emf-
xsd-Update-2.7.0RC3.zip!/
Just a note: destinations are (almost) always normal directory paths, not URLs.
Mirroring repos
You can quickly (well, I/O bound :-) mirror 2 builds repos so you can compare them on your disk. Here's an ant script for current, important, build repos.
<project name="Mirror" default="mirrorRepos">
<property name="eclipse36Mirror" value="${baseDir}/eclipse36" />
<property name="eclipse36Repo" value="http://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200" />
<property name="eclipse37Mirror" value="${baseDir}/eclipse37" />
<property name="eclipse37Repo" value="http://download.eclipse.org/eclipse/updates/3.7-I-builds/I20110613-1736" />
<target name="mirrorRepos">
<echo message="Mirror from ${eclipse36Repo} to ${eclipse36Mirror}" />
<p2.mirror destination="file:${eclipse36Mirror}" ignoreerrors="true">
<source location="${eclipse36Repo}" />
<!--slicingOptions includeOptional="false" includeNonGreedy="false" latestVersionOnly="true"/-->
<!--iu id="org.eclipse.sdk.ide"/-->
</p2.mirror>
<echo message="Mirror from ${eclipse37Repo} to ${eclipse37Mirror}" />
<p2.mirror destination="file:${eclipse37Mirror}" ignoreerrors="true">
<source location="${eclipse37Repo}" />
<!--slicingOptions includeOptional="false" includeNonGreedy="false" latestVersionOnly="true"/-->
<!--iu id="org.eclipse.sdk.ide"/-->
</p2.mirror>
</target>
</project>
You can run that from an eclipse instance, either in eclipse using an Eclipse Application, or from the command line (my preference):
bash$ eclipse/eclipse -noSplash \ -application org.eclipse.ant.core.antRunner \ -DbaseDir=/some/useful/basedir -buildfile mirror.xml
It'll spit out some messages about unsatisfied IUs, but that's OK because you're using the slicer, not the provisioner.
IUs and version numbers
If you're looking at plugin or IU versions, you can quickly get a list in the "IU=version" format:
bash$ eclipse/eclipse -noSplash \ -application org.eclipse.equinox.p2.director \ -repository http://download.eclipse.org/eclipse/updates/3.7-I-builds/I20110613-1736 \ -list
If you pick a composite repo like http://download.eclipse.org/eclipse/updates/3.6 you can list all of the IUs, and then install a specific version of org.eclipse.sdk.ide, for example.
Creating a product for your platform
You can use the same scripts that the PDE build uses to generate a complete install for a platform:
<project name="Build Product" default="buildProduct">
<property name="p2.repo.URL" value="http://download.eclipse.org/eclipse/updates/3.7-I-builds" />
<property name="iuId" value="org.eclipse.sdk.ide" />
<!--property name="iuId" value="org.eclipse.platform.ide"/-->
<!--property name="iuId" value="org.eclipse.rcp.id"/-->
<property name="profile" value="SDKProfile" />
<property name="os" value="linux" />
<property name="ws" value="gtk" />
<property name="arch" value="x86_64" />
<target name="buildProduct">
<ant antfile="${eclipse.pdebuild.scripts}/genericTargets.xml" target="runDirector">
<property name="p2.director.installPath" value="${baseDir}/p2_${os}_${ws}_${arch}/eclipse" />
<property name="p2.director.profile" value="${profile}" />
<property name="p2.director.iu" value="${iuId} " />
<property name="os" value="${os}" />
<property name="ws" value="${ws}" />
<property name="arch" value="${arch}" />
<property name="p2.repo" value="${p2.repo.URL}" />
<!--property name="p2.director.version" value="an.IU.version"/-->
</ant>
</target>
</project>
bash$ eclipse/eclipse -noSplash \ -application org.eclipse.ant.core.antRunner \ -DbaseDir=/some/useful/basedir -buildfile productBuild.xml
The exact same thing can be accomplished using the command-line director (just with a lot of arguments :-)
bash$ eclipse/eclipse -noSplash \ -application org.eclipse.equinox.p2.director \ -repository http://download.eclipse.org/eclipse/updates/3.7-I-builds \ -installIU org.eclipse.sdk.ide \ -destination /some/useful/basedir/linux_gtk_x86_64 \ -bundlepool /some/useful/basedir/linux_gtk_x86_64 \ -profile SDKProfile \ -profileProperties org.eclipse.update.install.features=true \ -p2.os linux -p2.ws gtk -p2.arch x86_64 -roaming
Installing into your own eclipse
This is the shortest command line of all, and the easiest way to take a new eclipse and install features from scripts.
bash$ eclipse/eclipse -noSplash \ -application org.eclipse.equinox.p2.director \ -repository http://download.eclipse.org/eclipse/updates/3.7-I-builds \ -i org.eclipse.releng.tools.feature.group
More info
More about the director command line: http://help.eclipse.org/
More about the p2 tasks and all the options they can take: http://help.eclipse.org/
Monday, March 28, 2011
maven and tycho vs an Orion feature
EclipseCon 2011 has come and gone, and I was lucky enough to be there. One of the nice things about going to speak at EclipseCon is the opportunity to see the other projects and technologies congregating around the Eclipse Foundation. I'll blog later about the talks I gave, but one of the things I tried while down there was using Maven and Tycho to build a p2 repo.
I picked one of the Orion features. My efforts can be seen in git://git.eclipse.org/gitroot/e4/org.eclipse.orion.server.git in the mavenExperiment branch.
There are some things maven needs to get started, which I put in a parent pom. A tycho-version property (I needed the 0.11.0-SNAPSHOT so a javax.servlet import package could resolve) and a pluginRepository so I could load that version of tycho. The repositories to build from: 3 p2 repos in my case. Some build plugin configuration I copied from the tycho RCP example, including configuring the qualifier with <format>'v'yyyyMMdd-HHmm</format>
The plugin poms are straight forward, as I want manifest first builds. You just need to make sure the versions and IDs match. Same with feature poms.
To generate the p2 repo you need to specify 2 things. An eclipse-repository pom (relatively straight forward, although I've copied in extra maven plugin statements). And a category.xml that lists the features you want in the repo.
I list all of my modules in my aggregator pom, and then run "mvn clean install".
Ta da: a p2 repo with my 2 features in it.
I'm still encountering some problem either with my configuration or with tycho itself. I open bugs for the $HOME/.m2/repository interfering with a second build: https://issues.sonatype.org/browse/TYCHO-606 and trying to add extra categories to the p2 repo using category.xml results in the generated repo missing things: https://issues.sonatype.org/browse/TYCHO-605
I think there's some work to be done on the Execution Environment side as well, although JIRA appears to be down at the moment.
Once the build has been configured to run with maven, the next person just has to check out the code and "mvn clean install". That looks promising. I'm glad I had the opportunity to try out maven and tycho.
I picked one of the Orion features. My efforts can be seen in git://git.eclipse.org/gitroot/e4/org.eclipse.orion.server.git in the mavenExperiment branch.
There are some things maven needs to get started, which I put in a parent pom. A tycho-version property (I needed the 0.11.0-SNAPSHOT so a javax.servlet import package could resolve) and a pluginRepository so I could load that version of tycho. The repositories to build from: 3 p2 repos in my case. Some build plugin configuration I copied from the tycho RCP example, including configuring the qualifier with <format>'v'yyyyMMdd-HHmm</format>
The plugin poms are straight forward, as I want manifest first builds. You just need to make sure the versions and IDs match. Same with feature poms.
To generate the p2 repo you need to specify 2 things. An eclipse-repository pom (relatively straight forward, although I've copied in extra maven plugin statements). And a category.xml that lists the features you want in the repo.
I list all of my modules in my aggregator pom, and then run "mvn clean install".
Ta da: a p2 repo with my 2 features in it.
I'm still encountering some problem either with my configuration or with tycho itself. I open bugs for the $HOME/.m2/repository interfering with a second build: https://issues.sonatype.org/browse/TYCHO-606 and trying to add extra categories to the p2 repo using category.xml results in the generated repo missing things: https://issues.sonatype.org/browse/TYCHO-605
I think there's some work to be done on the Execution Environment side as well, although JIRA appears to be down at the moment.
Once the build has been configured to run with maven, the next person just has to check out the code and "mvn clean install". That looks promising. I'm glad I had the opportunity to try out maven and tycho.
Thursday, March 10, 2011
A working state from a p2 update
In one of our recent build/update cycles we made a change that was source compatible but not binary compatible. In this case (there are no errors anywhere) it's easy to miss that you need to tag the consuming bundle as well.
The build ran fine, but the previous build's consuming bundle was used when creating the update site (since we didn't tag that bundle, the qualifier did not change).
When an unsuspecting co-worker updated to the latest build, his system would no longer start at all with a number of java.lang.NoSuchMethodErrors.
So how do you fix your install, given that you can't even bring up the workbench? As it turns out, p2 can help you. The SDK profile saves its state on each installation change. You can usually find this information in:
Then you just need to pick the last stable version of your installation, and you can revert your install and restart your system.
Obviously you'll need to use a different eclipse install and the necessary -destination arguments if the corrupt bundle is *in* the director part of your install's p2 framework :-)
The director won't run this command without specifying a -repository, but it's not important which one you use so I used the installations own SDKProfile repository.
For more director information see p2 director.
The build ran fine, but the previous build's consuming bundle was used when creating the update site (since we didn't tag that bundle, the qualifier did not change).
When an unsuspecting co-worker updated to the latest build, his system would no longer start at all with a number of java.lang.NoSuchMethodErrors.
So how do you fix your install, given that you can't even bring up the workbench? As it turns out, p2 can help you. The SDK profile saves its state on each installation change. You can usually find this information in:
bash-4.1$ ls eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile 1299467192263.profile.gz 1299499722731.profile.gz 1299761082054.profile.gz 1299467192395.profile.gz 1299499723473.profile.gz 1299761705917.profile.gz 1299467285942.profile.gz 1299499764937.profile.gz state.properties 1299467287251.profile.gz 1299500329056.profile.gz
Then you just need to pick the last stable version of your installation, and you can revert your install and restart your system.
eclipse/eclipse -noSplash \ -application org.eclipse.equinox.p2.director \ -repository file:$(pwd)/eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile \ -profile SDKProfile \ -revert 1299499722731
Obviously you'll need to use a different eclipse install and the necessary -destination arguments if the corrupt bundle is *in* the director part of your install's p2 framework :-)
The director won't run this command without specifying a -repository, but it's not important which one you use so I used the installations own SDKProfile repository.
For more director information see p2 director.
Wednesday, February 23, 2011
PDE build as workspace export
A question came up on the newgroups^H^H^H^Hforums recently about a headless way to build/export features and plugins from a workspace. It turns out it is possible, using the pluginPath property.
When you configure the PDE build build.properties:
But I wouldn't recommend it. Without digging into some arcane PDE build properties, it pollutes your workspace with build files, @dot directories, etc.
When you configure the PDE build build.properties:
- Your provided topLevelElementId has to be your feature, or a "master" feature.
- You must provide an elementPath property that's the absolute path to your master feature.
- Your pluginPath can then point to your workspace.
- You probably want to skipBase, skipMaps, and skipFetch
- p2.gathering = true will generate the p2 metadata
- p2.category.site=file:/
/site.project/site.xml will include the site categories in the generated build repo
But I wouldn't recommend it. Without digging into some arcane PDE build properties, it pollutes your workspace with build files, @dot directories, etc.
Subscribe to:
Posts (Atom)
