Data in Motion

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Combining Eclipse Way with XP and Scrum -- a Case Study ======

//[Updated with light edits; February 1, 2018]//

In our previous installment, I gave Eclipse another shout out for their amazing perfect track record of shipping on time for 13 years.  I then described how I had the privilege of working with a team that, by intentionally adopting practices from Eclipse Way into our XP/Scrum process, achieved a similar perfect track record over a period of four years, shipping every six weeks, with only a single emergency bug fix release.

Here I'd like to describe the main practices from Eclipse Way that we incorporated into our XP and Scrum process that I believed played a crucial role in our success.  These practices are, in the order I will discuss them:

  - The Eclipse Modularity Story
  - 6-week Release Cadence
  - Ship Each Milestone
  - Strut your Successes

Let's look at each in turn.

===== The Eclipse Modularity Story =====

Eclipse is built by multiple distributed teams numbering nearly 700 concurrent engineers (as-of the 2014 release).  In order to enable this many engineers to work together without constantly breaking each others' code, Eclipse designed a plug-in system into the architecture from the beginning.

In my experience, there are two key aspects to this plug-in system that help it to succeed as well as it has both technically and from a product-management perspective.  These are:

  - **Internal vs. External API:**  Eclipse separates code that is intended to be consumed by other projects from internal implementation detail.  Further, developers provide strong guarantees to customers that External API may be evolved, but not in a way that breaks their code.  Internal APIs, on the other hand, may break, change, or be completely rewritten as needed.
  - **Isolate plug-ins from changes in dependencies of dependencies:**  Eclipse's plug-in engine sandboxes and isolates plug-ins from being able to view or change transitive dependencies of dependencies.  This minimizes "version hell" caused by these transitive dependencies utilizing conflicting versions of the same library.  The purpose is again to enable interdependent projects to evolve independently.

Here are a few more thoughts on each of these topics.

==== Separate internal and external API ====

A plug-in is a Java Jar that is divided into two kinds of APIs:
  * Internal APIs that are not for external usage
  * External APIs that can be used by anybody and will be supported as a "contract" forever.

Eclipse Platform commits to binary class level compatibility for all APIs that are marked External. This provides a strong contract that customers can rely on to ensure that code they write today will work tomorrow.

Steve Northover, the "father of SWT", (the Eclipse GUI library) likes to say, "API is forever and @deprecated is a lie; somebody will always depend on your code".

The article [[http://www.eclipse.org/articles/article.php?file=Article-API-Use/index.html|How to use the Eclipse API]], while deprecated as-of this writing in my opinion still provides valuable perspective on API best practices.  In addition, Eclipse's wiki contains several articles on [[https://wiki.eclipse.org/Evolving_Java-based_APIs|Evolving Java-based APIs]] with detailed information for practitioners interested in building strong API guarantees into their products, even if they are not building on top of Eclipse or OSGi infrastructure.

API is a deep topic and worth much more than this brief introduction.  If there is demand, I can write more about that.  

==== Isolate plug-ins from changes in dependencies of dependencies ====

In Eclipse, plug-ins run inside Eclipse's OSGi container.  OSGi adds an additional benefit to Eclipse's plug-in and API story:

In addition to allowing plug-in jars to declare what packages are APIs, the Eclipse container allows plug-ins to declare their dependencies similar to the way Maven projects declare dependencies.  

And the container enforces that a given plug-in jar only has access to the exact libraries/versions that it declares as dependencies, and **not** any transitive dependencies of dependencies.

Since plug-ins do not see transitive dependencies among each other, this minimizes conflicts caused by dependencies of dependencies depending on differing versions of the same library and enables plug-ins to evolve independently of each other by multiple teams around the world.

While this approach doesn't solve every problem related to "jar version hell", it goes a long way toward keeping differing teams from breaking each others' code when they change their dependencies or dependencies' versions.

We obtained substantial benefit by using Eclipse's OSGi container.  Non-Eclipse projects may wish to consider [[http://karaf.apache.org/|Apache Karaf]] as an alternative.

That said, while we realized substantial gains by using an OSGi container, I believe that disciplined adoption of Eclipse's API conventions can realize significant modularity gains, even when implemented outside of an OSGi container.

Lastly, this blogger would encourage the Java community to evolve a solution that unifies the current dichotomy between Maven's dependency architecture and OSGi/P2's.

To summarize, one of our critical success factors was our adoption of Eclipse's philosophy of strong APIs with strong API guarantees.  This enabled us to build our own product as a set of small modules that could be evolved independently.  And it enabled us to evolve independently to our customers without fear of breaking them.

===== 6-week Release Cadence =====

Scrum or XP projects typically utilize a two-week sprint / iteration as their basic unit of product delivery.  Stories are planned at the beginning of the sprint and are demoed/delivered at the end.  The team's velocity is measured over time and this measurement is used to determine how much work can be committed in subsequent sprints.

Eclipse Way is different here, but not in an incompatible manner.  Eclipse uses 6 week iterations ended by a formal milestone release.

There are multiple ways one can combine Eclipse Way with Scrum/Agile.  In our project, we simply overlaid the Eclipse Way 6-week release cadence over the top of our two-week sprints.  This had several interesting effects.

  * Scrum stories could be defined at developer granularity, rather than as strict user stories.  Here is the distinction:
    * User stories are a discrete bit of user-visible functionality
    * Developer granularity is a task that is a part of a larger "theme" or group of user-visible functions

In Eclipse Way, user stories only need to be delivered every six weeks--on the release schedule--as opposed to at the end of each sprint.

This encouraged us to group related user stories together as "themes" to be delivered in a single milestone.  This improved our estimations by enabling us to subdivide user stories to a small enough granularity that they could easily be estimated.  And this reduced end-of-sprint stress by eliminating the need to get the burn-down chart to absolute zero at the end of the sprint.

In addition, since the commitment we made to the customer was at the level of themes, (which are imprecise groupings of stories that deliver substantially on a particular set of features) we had flexibility about the exact amount of functionality that was delivered as a part of that theme in each release.  Consequently, we found that delivering on themes rather than specific user stories tended to be less stressful and easier to negotiate with the customer.  

In practice, it is often unnecessary to pin down the exact two-week period during which a specific user story will be delivered.  A six-week window is usually more than enough precision for most customers, and many only require once-per-quarter precision.  

Lastly, adopting a six-week release schedule does not preclude introducing traceability into one's process, where customer requirements can be traced directly through themes, stories, and tasks to lines of code.

===== Ship Each Milestone =====

Shipping software should be like a good airplane trip--boring and uneventful.  However, all too often, shipping software is stressful and painful.

At Eclipse, shipping is a core value.  Software has no value unless it is delivered to customers.  So, in order to reduce and/or eliminate the pain of shipping, Eclipse practices shipping often.

At Eclipse, each six-week milestone build is verified and delivered as a full shipped product.

For the development team, this requires turning the "ship product crank" every six weeks.  This has several benefits:

  * If a team ships often, they have a strong incentive to automate as much as possible about the entire process.  This goes beyond automated unit and integration test suites to automating the release process itself to the greatest extent possible.
  * Successful shipping teams create release checklists and execute the checklist like a seasoned airline pilot readying an airplane for takeoff.
  * If a team ships often long enough, they will create and document contingency plans such that even catastrophic events like failure of crucial build servers can be taken in stride without jeopardizing the ship date.

As a team matures in its shipment process, shipping software can become truly the most dull part of the sprint.  And to this engineer, that's a good thing. :)

But the benefits don't stop there.  Shipping themes of user stories on a six-week schedule in a form the customer can immediately use (as opposed to the Chinese Water Torture method of shipping every two weeks) provides the customer with a coherent view of how multiple features fit together and helps develop the relationship with the customer.

The customer sees visible progress; this inspires confidence; this results in more progress.

None of this supersedes the Agile value of an on-site customer.  Again, in our experience, it was valuable to layer this practice on top of the other Agile values and top adapt them all to work cohesively together.

Ship often.  Make shipping as dull as a smooth airplane flight.  Have happy developers and happy customers.

===== Strut Your Successes =====

Eclipse milestone builds are accompanied by a document labelled "New and Noteworthy" from all of the core Eclipse teams.  This document is akin to the "sprint demo" in Scrum--an opportunity to strut what's new and exciting.

No matter the form this takes, we found it important to publicize our successes and tell our customers what clear benefits they would obtain by adopting the latest release.

In my experience, the six-week milestone release interval of Eclipse can be a much better granularity for producing New and Noteworthy documents and/or sprint demos.  There simply is more that can be demoed.  And since the new features are grouped together into themes, it is much easier to demo a theme of related functionality every six weeks (and to tell a compelling story around a theme) than around a few isolated user stories every two.

===== Conclusion =====

In this article, I have reviewed four key Eclipse Way practices that in my experience work very nicely in tandem with practices from Scrum and XP.  In a previous job, these practices synergized into a process enabling our team to consistently deliver quality software with new valuable features, on time, every six weeks, without stress.  I have delivered this as a case study because I believe that this success can be replicated, and in combination with best practices from XP and Scrum can result in high-value software delivered consistently on-time, at or under budget.
Data in Motion

User Tools

Site Tools

Page Tools
