Scala's version fragility make the Enterprise argument near impossible December 7, 2011

An attribute of Scala is that the Scala compiler generates fragile byte-code.  This means that all the code in an executable (JAR or WAR) must be compiled with the same library and compiler versions.  If you're using Scala 2.9.1 in your project, you must compile against a version of Lift that's compiled against Scala 2.9.1 and all the Lift dependencies must also be compiled against that version of Scala.  This is a reason that Lift has precious few Scala library dependencies.  It's also a reason that Lift is sprawling... there are a lot of modules in Lift that need to be cross-compiled and rather than having an external ecosystem of modules, we need to make them part of Lift to make sure they are all compiled against all the versions of Scala that Lift supports.

The by-product of this issue is that it's nearly impossible to test Lift or a Lift-based app against a pre-release of Scala.  This is one of the many reasons that there's almost always a "critical bug fix" release of Scala a few weeks after any major release.  The first chance that the broader community gets to use the ecosystem of Scala libraries and frameworks is after a release and after the cascading compilation/test/release 

A year+ back, during the 2.8 development cycle, some of us got together on the "Fresh Scala" project.  Basically, the idea was to have continuous integration of the major libraries in Scala-land and weekly milestones so that the community and the ecosystem could give better feedback on new versions of Scala.  Due to a lack of volunteer time, we never got the CI system built and here we are a year and a half later with the same Scala version fragility issues, but one thing has changed.

Scala is no longer exclusively an academic and community effort.  Scala the language and some associated libraries are now part of an enterprise-targeted suite from TypeSafe.  TypeSafe is a huge (20+ heads) venture funded company that is supposed to be bringing Scala to the enterprise.  But Scala's version fragility creates two huge costs for complex enterprise-type systems:

1. All dependencies must be compiled against the exact same version of Scala and every referenced Scala library which makes in-house builds when there are multiple teams amazingly complex or time-consuming
2. External libraries (like Lift) cannot have multiple layers of dependencies because you need to match the exact version of every library and Scala version all the way down

As a library builder, the version fragility issue means that our library is not nearly as good as it could be and we can't service our community the way we want.

In terms of servicing the community, we cannot have predictable releases of Lift because we try to time our releases such that Lift will be released soon after a new version of Scala is released.  Given that Scala release schedules are not predictable and given that we need to wait for the dependencies (e.g., ScalaCheck and Specs) to be upgraded, we can't answer simple questions like "when will Lift support Scala 2.8.2?"

More importantly, there are few external Lift modules and they are generally not available except for specific versions of Scala and Lift and there are always issues when there's a version mismatch.  Modules like Rogue and Reactive Web are available for a small subset of Lift/Scala version combinations.  There are dozens of folks who have smaller modules for Lift, but none of them are cross compiled for the last year's worth of Scala and Lift releases.  This basically means you can use Rogue (something I'd suggest) or you can use Reactive Web (something else I'd suggest), but you can't use them in the same project because there's no guarantee that they are compiled against the same versions of Lift and Scala, nor is there any guarantee that they will be compiled against the versions of Scala and Lift that you want to use.

So, the Scala library and framework ecosystem is about as big as it can get given the Scala version fragility issue.  Because its not possible to have multiple layers of library dependencies in Scala, there will either be monolithic projects like Lift or islands of projects like ScalaZ.  And at this point, Scala does not have enough native libraries and frameworks to make it enterprise-worthy.  Yes, you can use Java libraries, but as others have found, the cost of wrapping Java libraries in Scala is often not worth the cost/effort.

What have I done about the issue?  Over the years, I've raised the issue privately and it has not been addressed.  I have tried to organize a project to address part of the issue, but haven't had the volunteer uptake for the project.  I have had discussions with Paul Phillips about what could be done and Paul, Jevgeni and I even worked out a system that could be integrated into the Scala compiler to address the issue in 2010 when we were at the JVM Language Summit.  The best answer so far is the Migration Manager, a closed source tool that may address some of the issues (I haven't tested it because I'm not going to use any non-OSS software for compiling or testing Lift to avoid any legal issues.)

What we need:

1. Blog about the issue and raise awareness about the version fragility issue and how it has impacted you
2. Next time you're on a TypeSafe "what would you pay us money for?" call, please tell them that a vibrant Scala ecosystem is critical to wider adoption and the version fragility issue is significantly impacting both internal development efforts and ecosystem growth.  In order for your company to consider buying anything commercial from TypeSafe, you need broader internal adoption of Scala and that comes from reducing the transaction costs of using Scala including eliminating the version fragility issue in the open source parts of Scala.