Scala use is less good than Java use for at least half of all Java projects September 6, 2011

So, I made a post about agreeing with a post about Scala being "hard" for a large portion of Java developers.  This post caused a fair amount of "discussion" much of which misinterpreted my post.  I am making a post that I hope will be more clear in terms of the who, why, and my motivation.

First, about me.  My name is David Pollak.  I've been active in the Scala community since November 2006.  I am the founder of the Lift web framework project.  Here are the important bullet points:

• I have been writing Scala code continuously for longer than anyone outside of EPFL. [update - @propensive and Bill Venners have been in the Scala community longer than I have.]
• I started the first Scala conference in 2008, the Scala Lift Off, and continue to run that conference to this day.
• I have written more lines of Scala code (more than 250K) than almost anyone on the planet.
• I have written a popular book introducing Scala called Beginning Scala.
• I have had more than 10,000 interactions related to Scala and Lift on the Scala and Lift mailing lists over the last almost 5 years (that means I've interacted with a lot of developers.)
• I founded the Lift web framework project and have written a substantial portion of Lift including designing Lift's core APIs.  Lift is one of the most popular Scala-based frameworks and the first broadly known Scala library.
• I have code reviewed more than 500,000 lines of Scala code in the last year.
• I make my living consulting, mainly on Scala and Lift related projects and have broad exposure to lots of projects.
• I have given more than 25 presentations on Scala and Lift over the last 5 years.
• I have taught hundreds of people to use Scala and Lift in small groups.
• I introduced Scala, either directly or indirectly, to half of the big-name Scala users listed on TypeSafe's home page.

All of the above is to say, "I have substantial experience coding Scala, teaching Scala, and introducing Scala into a wide variety of environments.  I have interacted with enough Scala users and prospective Scala users to have a broad base of data to draw from in my analysis of Scala success factors."  This does not mean that I think I'm right and everyone else is wrong.  It does, however, mean that people who use ad hominem rhetoric to argue against my posts have very little validity or value in their posts. [Update: re-reading @fogus's post... I mis-interpretted it.  I think @fogus and I reach the same conclusions. I stand by my assertions that I'm not perpetuating a meme. However, I apologize for harshing on @fogus and his post!]

With that as my setup, let me give you my conclusion:

This should be no more controversial than:

While I used "Scala is hard for some developers" in my last blog post, I'm being a lot more precise here.  I did not say that I find Scala difficult.  I have however, observed a lot of developers who find Scala difficult and I've outlined my reasoning here and here.  For that class of developer, the value of Scala is outweighed by the costs of Scala.

Discussion

One of the things I'm particularly bad at is spelling.  My brain just doesn't remember the spelling of words.  When I was the editor of my college newspaper, one of the reporters got up in my face about being lazy because I couldn't spell (this was 1985... before spell checkers.)  I wrote a integral on the blackboard (this was before whiteboards) and asked her to solve it.  She said, "That's math... that's hard... I'm talking about something as simple as spelling."  Well, for me, I can solve integrals in my sleep and I can't spell to save my life.  Different people have different skills.

Different people value different things.  There is a class of people who love computers and love to code.  I'm in that class.  There is a class of people who would not put coding in their list of top 5 things they enjoy doing.

There is a large set of developers who have chosen development as their career who lack some combination of innate ability and motivation.  There are schools that foster this mentality.  No amount of blogging and blustering will change this.

For those who lack a combination of the innate ability to code and the interest in improving themselves, Scala is a liability.  If there are too many Scala liabilities floating around (i.e., failed Scala projects), Scala will cease to grow and that's a seriously suboptimal situation for people like me that have invested more than $600,000 in the Scala ecosystem.  More on motivation at the end of the post.

One of the key arguments against my "Virgina" post was that we must take the Java developer pool as it is because Scala is not better enough to cause a material improvement in the overall quality of the pool.  Quoting me:

I agree with Paul Snively that Scala is learnable.  That's exactly why I've been actively promoting Scala for years.  The issue is that Scala is learnable by a certain class of people.  That class does not include those that don't want to learn it and those that don't have the ability to learn Scala (just as there are those that can code PHP but could not code Java.)

In fact, I have been saying there is a class of developers for whom Scala is not appropriate for nearly four years... perhaps longer, but that's the oldest post I could find.  My position at the time was not dis-similar from the position that many are taking today: train or fire the developers who are less productive with Scala.  I have also been pretty clear that there's a class of Java-only developer that does not succeed with Scala for at least three years.  So, my position has been pretty consistent for many years.

What has changed is that I've realized that there's a vast quantity of development shops where the developers show up, have a few meetings, write a few lines of code and go home.  I've had experience with three instances of that kind of company over the last year.  One adopted Scala, is struggling, but trying to do the right thing despite problems with Java written in Scala, recruiting developers, and internal resistance.  One made the decision not to adopt Scala (although there's a little Scala that the VP Eng doesn't know about in production, but it's maintained by one person for whom Scala is vastly better at the task than Java.)  One made the decision to roll back from Scala back to Java because the institutional cost of replacing half their development staff, sending 25% of the remaining staff to expensive courses, and foregoing outsourcing parts of the project outweighed the value that Scala was bringing to the top 3 developers (although it was a very tough and close call.)

We live in a world where the average developer writes 3,250 lines of code per year (about 20 per day).  This is going into Eclipse, pressing the "give me pattern X" and filling in the blanks, then going to a few meeting and calling it a day.  We cannot fire all these developers.  We cannot train these developers to be better.  This is the Center of the Mean.  This is developer who may be the butt of a Dilbert cartoon.  But you know what, that's who uses Java.  And you know what else, that developer lacks some combination of the innate ability and will to get better.  Not only that, but all the way up that developer's management chain, there is not the ability or will to change the situation.  We cannot move this mountain... or more specifically, Scala is not better enough than anything else to allow shops to fire 50% of their less productive developers.

So, the better courses of action would be to:

• Focus on the kinds of developers and projects that will get a 3x or more out of Scala; or;
• Improve Scala for the COTM developer (this is not going to happen as long as Scala is primarily a research driven language)

Motivation

So, "what," you ask, "is your motivation in writing a series of pieces that make it seem like Scala is not successful and is hard?"

I have been a Scala supporter and fan since the time I encountered the language nearly five years ago.  There are few people who have moved the cause of Scala along more than I have and I have a strong vested interest in seeing Scala and Lift continue to succeed.

Scala has seen some remarkable success stories.  The quotes and articles and general sense of awesomeness around Scala is both tremendous and well deserved.  Scala is a remarkable language and a language that has no equal in computing today in terms of its versatility for solving a broad range of simple to complex problems.

But, Scala is not a cure-all.  Scala is successful in places that hire excellent developers.  Scala allows those developers a higher multiplier on their ability than Java or most other languages.  But in less-than-expert hands or worse, in the hands of those that resent Scala, it's less good than Java.  It leads to team strife and discord that, given the team type, often manifests itself in passive-aggressive ways where ship-dates slip and ultimately the language is blamed.

In order for Scala to continue to grow, there must continue to be Scala successes with a minimum of Scala failures.  This means that being open and honest about Scala's strengths and weaknesses is imperative so that the right places choose Scala and Lift.

Scala's growth means that Scala must be adopted by the kinds of developers that have the right intersection of ability and desire to build amazing things with Scala.

We must accept that Scala's not going to displace Java for green-screen, CRUD, DB front end applications.  Scala's value for doing ORM (sorry Max... Squeryl is really amazing) is way less than Scala's value for doing real-time, distributed, concurrent applications.  But most of the world is doing ORM, CRUD, green-screen, fill-in-the-form-and-update-the-database kind of stuff.  That's where most of the developers and money is.  While the cool kids are building massively mumble, event-driven, mega-hyper-super-collosal-data, real-time, buzz-word-buzz-word stuff, most of the developers are doing the boring stuff of moving data in and out of the database and Oracle/SQL offers a perfectly reasonable concurrency model.

As a community, we must accept Scala's weaknesses.  We must recruit developers who are going to take advantage of Scala's strengths (which are numerous.)  We must also actively discourage folks from using Scala is there's a reasonable likelihood of failure.  Having a "Scala is hard" or "Scala is for really good developers" brand is a lot better than having a "Scala is risky and as often as not, leads to failure" brand.

My motivation in writing these posts is to raise this kind of awareness in the brains of developers.  I'd rather see 5,000 new Scala projects in the next year where 4,000 are successful than see 50,000 new projects where 10,000 are successful.  The developers who have the ability and inclination to be successful with Scala will likely read my posts and claim I don't know what I'm talking about after experimenting with and succeeding with Scala.  The rest of the developers will repeat the conclusion ("Scala is too hard") and choose not to use it.  And that outcome will likely lead to a lot more Scala success percent-wise and until Scala changes to be nice to COTM developers, that's a much better path to success for Scala and Lift.

Oh... and all you wicked smart people who are pushing the boundaries (or think you will) with data size, event frequency and real-time stuff, you'll find Scala to be a dream come true and there will be nothing like it that you've ever used (okay, except maybe Haskell).  So, come, build your cool thing on Scala and succeed.