Functional Languages will Rule (but not this year) May 27, 2011
Okay, to be fair, I have authored a few functional languages... a spreadsheet called Mesa and another spreadsheet called Mesa 2... and yet another spreadsheet called Integer. (As an odd historical note, Mesa and Integer are the only spreadsheets I know of that support higher order functions, but I digress.) But until I basked a while in the glory of programming language theory, I didn't understand what functional meant or why it was different or better.
So, five years later, I still think that functional languages will rule... the functional programming paradigm... one that stresses immutable data structures, composition, and pattern matching will provide a better way to write complex systems whether those systems are business systems, interactive applications, games, or even system level... writing those systems using functional languages will provide a better foundation, better stability, and ultimately lower total cost of ownership.
Let's take a tour of the landscape to see where each of the major functional languages is and why I think it hasn't become popular.
Erlang
Erlang is a tremendously powerful language for building high availability switching/routing kinds of applications. Erlang has powerful pattern matching that can even match and extract bit-wise data. Erlang's execution model and execution engine are purpose-build for low-latency, distributed message processing. The Erlang runtime abstracts system level stuff like I/O into messages and every part of the system deals with messages. Erlang embodies the Actor paradigm and each Actor (or process in Erlang parlance) runs with its own heap, so garbage collection pauses are generally inperceptable.
Many powerful systems (most of Ericsson's switches, RabbitMQ, Riak, Facebook's chat system, etc.) are written in Erlang. Erlang is perfect for writing scalable, distributed, message-heavy (rather than compute heavy or logic heavy) applications. Erlang's message-passing works the same between processes/Actors in the same address space and distributed across address spaces. This, combined with Erlang's OTP and Erlang's coding philosophy (things will fail, expect it) result in some amazingly scalable, resilient systems.
So, why is Erlang not part of the mainstream? Why aren't developers flocking to it?
First, it's ugly. I never really understood how aesthetics impact developers, but beauty on the developer's screen does lead to better results (okay, Java is an exception here...)
Second, Erlang is not something you can "parachute" into. Most Erlang coding is by convention (it's unityped and the compiler provides no type checking). This means that if you grow up with an Erlang team and it's a good team, you'll build great things (Basho anyone?), but it's not a casual language.
Third, Erlang is not good for compute-intensive code... its runtime is slow relative to the JVM or C++.
Fourth, Erlang is all immutable all the time. This is hard to grasp and work with, especially coming from a Java or Ruby background. And without state monads and other pieces that the Haskell folks built into the libraries, the only way to carry state around your application is on the stack via recursion within processes/Actors and message passing. While Actors/processes feel like coarse grained objects, building large networks of Actors/processes to carry state is a whole lot harder than defining some classing in Java or Ruby and interacting with those classes.
I apologize to the Erlang lovers in the world. I'm not trying to bash it... I think Erlang is a thing of beauty, but its beauty is beneath the skin. But Erlang's strengths are to a great degree rate limiting factors on Erlang adoption. Erlang is tremendously opinionated and that leads to excellence or failure but very little in between.
Haskell
Speaking of opinionated, Haskell certainly is. Haskell is a thing of beauty in the way that Karate is a thing of beauty. It requires practicing and rehearsal and focus and discipline and mathematics. But once you've got your black belt in Haskell, there's nothing you can't do.
Haskell seamlessly integrates all the best concurrency paradigms. Want STM, yeah, Haskell's got it. Want Actors, yeah, you can do those. There are half a dozen more floating around. Haskell, like Erlang, has its own runtime that does not rely on OS threads and can more accurately and granularly control scheduling.
Haskell guarantees no side effects in code. This means the same parameters fed to a function will always return the same value and anything that violates that (e.g., getting the current time) is cordoned off into the IO Monad. This leads to many interesting possibilities in terms of scheduling actual code execution (at the last possible moment), being able to calculate dependencies and split independent tasks across CPUs with little or no intervention on the coder's part.
Haskell code is concise and Haskell's type system is powerful, but Haskell's type inference is even more powerful, so types are generally for documentation rather than for the compiler. For the Python lovers in the audience, Haskell, too, has meaningful whitespace.
Haskell has more backends than any other language I know. It compiles down to native code, LLVM, and C as well as JavaScript (there are probably a few more back ends out there.) And Haskell has its own package distribution system and general ecosystem.
Haskell will likely never get Python/Ruby popular. Why? It's super-hard until it's super-easy. The learning curve is difficult. No, the syntax isn't hard and making it through some of the excellent books on Haskell is totally doable in less than 21 days. But mastering Haskell takes a lot of work and planning and designing Haskell apps takes a lot of work. This contrasts with spitting out a Ruby app or knocking together a Java app. That's not the Haskell way and in the long run, I'm betting Haskell programs are easier to maintain and enhance, but getting over the short run issues are a huge barrier.
OCaml
I don't have much to say about OCaml because every time I try to get into it, I don't get excited. I don't know why, but I've never been able to cotton to it, even with substantial coaching from Paul Snivley (sorry Paul, I keep failing you).
F#
I've spent a little time with F# and learned a lot about F# libraries at the Functional Programming Exchange.
F# is the most mainstream of functional languages. Microsoft did something amazingly bold by including it with VisualStudio.Net, but that makes a ton of difference.
F# is polished, supported, and has a great IDE. If it weren't for the fact that I'd have to boot Windows to use it every day, I'd be spending a lot more time with F#. I also think in 5 years, F# will be an accepted part of the corporate development landscape and it seems that Microsoft is making a real commitment to it, not to mention the folks at Thought Works. I don't know if F# will ever be Python popular, but right now, it's got the best chance.
Scala
I saved the best for last. Five years ago, I thought that Scala would rule the world. And last year, it seemed to be close... it almost broke into the Tiobe top 20 languages, but then it dropped back. As I was contemplating a business built around Scala and Lift, I had to ask myself, why the drop in market popularity?
I've been one of Scala's biggest fan-boys for nearly 5 years and I continue to love the language, love the community (especially the Lift part of the community), and think that Scala has a long future in the same way that Smalltalk and Eiffel have.
Scala is a tour de force of concepts:
- Solving the multiple inheritance diamond problem with traits and linearization
- Pattern matching and customized extraction via unapply/patterns as partial functions
- Implicit parameters and conversion with result in Scala's unmatchedly awesome collections library
However, Scala has many challenges that make it less than ideal for the "mainstream developer" (one standard deviation on either side of the mean)... the developers who are spitting out Java code by the ton.
First, Scala tooling is weak... the IDEs, debuggers, etc. tools are not there. Yes, IntelliJ 10.5 is usable. But doing Java coding side-by-side with Scala coding demonstrates just how lame the Scala tooling is. While TypeSafe may have $3M to pour into tooling, the tooling problem has been a persistent one for the 5 years I've been doing Scala and I no longer believe it can be done correctly. Now, you may ask why one needs good tooling, after all, I wrote Lift with Emacs (although I've had brief detours though Eclipse, JEdit, NetBeans and most recently IntelliJ.) But it's different coding a coherent Scala program from scratch with Emacs than working with an enterprise style app that has 300 "POJO" model classes (or even 300 case classes). Further, the mainstream needs the tools with the built-in patterns.
Second, Scala is not opinionated enough (opinionation is a tough balance.) You want to code ScalaZ style... be Scala's guest. You want to code J/EE style... be Scala's guest. Lift style... sure, if it makes you happy. Scala can conform to whatever style you want. But without guard rails (or Rails), most of the mainstream coders are lost. Not having the Scala patterns built into the IDE (even Textmate has common Ruby patterns built in) makes it harder for the mainstream coder to "know what to do." Rails is opinionated and has led to an opinionated culture and that has led to great success. J/EE is opinionated (thanks in large part to Bob Pasker who imposed his opinions and will on WebLogic, WebLogic users and the J/EE standards). Spring is opinionated. Scala so much less so. It hasn't been until this year that I have come around to the position that Scala's lack of opinionation is a weakness, not a strength. But, thinking about one of Martin's "Scala isn't hard" posts, he talked about Legos... do you want a prepackaged toy or do you want Legos so you can build any toy you want. But the Lego site is about prebuilt toys these days. The strength of Scala's early flexibility is a serious limiting factor as Scala tries to go mainstream. This is true of Lift as well.
Third, Scala has the blessing and curse of Java compatibility. The blessing is a tremendously stable runtime (the JVM) and a huge collection of high quality libraries. The curse is that a class of developers treat Scala as a "better" Java or a "different" Java or "the thing my boss made me do and I just want to do" Java. Scala tooling is compared to Java tooling and that's a huge strike against Scala. Scala code can be written like Java code and you get almost no benefit from Scala, in fact sometimes (due to the tooling issues and the compile-time issues [Scala is about 10x slower per line of code to compile than Java]) an actual negative. Scala as Java suggests approaching problems like you approach them in Java and that leads to no material gain. But it's not every group that has someone who develops Scala-friendly coding styles and can then teach those to other team members. The team cost of this is high, especially in mainstream teams.
Scala works really well for top-tier teams and that's where you see most of Scala's penetration. You see teams like Novell Vibe and Foursquare and the UK Guardian doing stellar things with Lift and Scala. There were similar teams 20-30 years ago that did stellar things with Lisp and Smalltalk. Eiffel enabled grade A teams on Wall Street to outperform teams using C++ and Objective-C in the late 80s and early 90s. When the team has purpose and discipline, it can do tremendous things with Scala. But I have seen the benefits of Scala in top-tier teams turn into gradients of sub-optimality in mid-pack teams, just as mid-pack teams have rarely managed to succeed with Smalltalk, Lisp or Eiffel. (As a side-note, I get a lot of Eiffel ads that pop up in GMail when I'm reading Scala-related posts... I wonder if the Eiffel folks are trying to poach from the Scala market... but I digress.)
So, what could be done? If Scala tooling evolved into a system with Java-parity (including debuggers and JRebel support that doesn't break when you change functions around) and with a faster compiler, that'd be a start. There needs to be a "Scala way." It's not ScalaZ (although I think ScalaZ is beautiful and well thought out) because ScalaZ requires the thought work of Haskell. It's not Lift... I made the same mistake with Lift that Scala made... While Lift is opinionated about security and interactive apps, it's not a simple blue print that coders can follow to success. It's not Akka... Akka connects to everything and is generally an mish-mash of paradigms that kinda sorta hold together (watch Jonas' compatibility slide, it's some of everything, but like Lift and Scala, Akka can be almost whatever you want it to be.) In order to be successful, Scala needs a "way" with tooling support and a migration path from J/EE and Spring.
Scala needs to do to Spring what Spring did to J/EE. But with IntelliJ and Eclipse, the mainstream gain that could be gotten from Scala just isn't there. Most mainstream developers will use the database or message queue for their concurrency and distributed app needs. Spring makes these things possible for mainstream developers, so I don't see how Scala could make that an order of magnitude easier/better. For clustered apps, Akka is a nice back-end and for interactive apps, Lift is a nice front end. But these kinds of apps are the ones that make the news (Quora, Foursquare, Novell Vibe), but the ones that make bread and butter money are going to be J/EE and Spring apps (with some Flash or iOS or Android clients thrown in for good measure.)
Am I going anywhere? Nope, there's a good living and a lot of personal and technical satisfaction for me in the Lift community. I expect Lift will grow and thrive and attract more stellar successes, just as NextStep did in the 90s.
I also expect that Scala will continue to be a great niche language that has an energized community and continues to push the boundaries of computer languages and programming language theory.
But I no longer think that Scala is the break-out functional language. I just hope some language comes along in the next few years that breaks functional languages into the computing mainstream and runs on Linux and OS X as well as Windows.