Moving a team from Scala to Golang

Share the joy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

Scala has long been part of the CrowdStrike stack, the primary language in fact. I helped lead the adoption of Scala as we first started to develop our applications back in 2012. In fact it was one of pros for my decision making process of wanting to come to CrowdStrike. Several of the early developers were interested in adopting it as well so it seemed to be a nice fit.

I had come from a company called Gravity, who were also heavy Scala users. It was the primary language there. I was used to it, enjoyed it, saw the power of it and thought I could prevent some of the issues I saw with Scala as CrowdStrike grew. We were doing high scale analytics, batch jobs over Hadoop and our Chief Architect (hi Bissel!) was doing lambda architecture before it was what the cool kids were doing.

A recent quote from one of our senior engineers prompted me to finally write this post describing why we’re transitioning most of our stack to Go and why new services default to Go by developer choice.

Instead of waiting until the end of the this post I should clarify that Scala will not be leaving our stack completely. In fact it will complement where Go does not shine. Scala is a big part of our machine learning / analytics stack. It’s interop with java projects we use, and its ability to provide a nice DSL that our analysts can use still make Scala a solid choice. It’s becoming more of a specialized tool vs the core development language.

I’m going to take you through this from the lens of a Technical Director. A lens where you need to scale a company from the early days of 5 engineers to 200+ engineers as the business grows. It’s about having a maintainable code base where you can have people cross projects easily and get new hires up to speed rapidly.

I remember when I first saw the potential issues of scaling Scala at Gravity back in 2009/10ish. It was close to the end of the day when we had a major issue reported in production that was affecting our large customers. Several of us started investigating and were able to track the source of the issue. The only problem was we had no idea what the code was doing at first. We came across a strange symbol we hadn’t seen in our projects before. The spaceship operator <|*|> . Someone said out loud “what the hell is that?”.  There was some other implicit magic going on that wasn’t immediately apparent. A CMD+B to traverse into the method yielded nothing in our IDE as it couldn’t find the symbol (IDE’s have improved here since). A quick googling of “<|*|>” yielded nothing as well. We were stumped and didn’t have sources pulled down. [1]

Screenshot 2015-11-21 11.20.00

The developer who wrote the code was unreachable on vacation so we had to figure it out. We noticed a new library was being included called scalaz, hours later we had tracked down the mystery symbol and grok’d what was going on, made the fix and life was good again. That blip turned a fix that should have taken minutes into a fix that took hours. That was the point I started seeing the split in our engineering team.

Scala is a powerful language, it comes from academic roots and gives enough flexibility that you can easily start writing “write-once” type code.  Scala developers typically travel down two paths: You have the “it’s a better java ” camp you have the “I (heart) Applicative Functors” camp.

Screenshot 2015-11-21 12.07.04

The “it’s a better java” camp like the terseness of Scala, and  the standard features that make Scala generally more enjoyable than Java. There’s functional programming in their new Scala code but it’s not the main focus. The “I (heart) Applicative Functors” camp really takes to their new functional world and begin to expand their knowledge deeper down that path or they bring their already functional backgrounds from places like Haskell.

Based on my experiences you start to split down these camps and have excellent programmers in each so you can’t say one side is superior to the other. On the semi-functional side you potentially have more generalists working across languages, or those who may not want to learn lambda calculus theories to work with an API server.

As an example, this is a code sample from a project we had one of our more advanced Scala developers start:

Some of you may look at that and say awesome, but some will say WTF is that? There were thousands more lines like above. This was to be something the whole team could work on but half the team didn’t want anything to do with it. The developer who wrote it is a brilliant person but the fact it divided half the team was a problem. Luckily this was caught in code review and rejected based on our internal guidelines. It never made it out to production.

As you’re scaling an engineering team this split becomes more apparent when trying to get new hires up to speed. Scala has a lot of rough edges already around getting a build environment, SBT pain, IDE environment pain , release upgrade pain, slow build times, add on top of that a heavy dose of functional concepts required for proficiency and the ramp up time grows and dev output slows down. Now there’s more upfront training from existing developers required which slows things down as well. SBT is also a real sore spot. There’s always that one person who actually knows what the heck SBT is doing and can debug everyone’s issues. I’m aware that Scala is not scalaz and they can be mutually exclusive, but I’ve seen issues with or without it.

It’s also not a question of Scala being “too hard”. I’ve never had someone not be able to learn the language. It’s about the investment. You make an investment in something with the hope of it paying off somehow down the line. Whether that’s faster time to market, higher performance/lower cost,  or increased stability. There are specialized places I’ve seen that happen with Scala but not in the general case. We discussed as we were about to scale up if we wanted to invest in more docs, guidelines, example projects, etc… but the reality was we didn’t think that investment would come back vs Go.

This isn’t unique to companies I’ve worked for. Twitter has gone through the same growing pains, as well as other companies I’ve talked to at conferences and people I know working with Scala. It’s a common theme, in fact. While you can have very high performing small teams going with Scala, trying to grow and engineering organization > 50 is an uphill battle. If you’re already invested in the JVM, Java8 is a solid choice that borrows some of the concepts of Scala to make Java easier to work with.

Other references from larger teams seeing issues:

Yammer   http://codahale.com/downloads/email-to-donald.txt

Is LinkedIn Moving off of Scala? https://www.quora.com/Is-LinkedIn-getting-rid-of-Scala

Former Twitter Platform VP heard saying Java8 may have been a better choice if available when they made the choice of Scala for similar reasons https://www.quora.com/Is-Twitter-getting-rid-of-Scala

You can also see a trend at Twitter where the latest OSS projects released are in Java (Heron, DistributedLog, etc..) In fact a telling line in the Hero release reads: “It is written in industry-standard languages (Java/C++/Python) for efficiency, maintainability, and easier community adoption.

Is Scala on it’s way out? (article):

https://www.linkedin.com/pulse/scala-way-out-owen-rubel

 

and a funny Tweet about the subject:

Screenshot 2016-02-17 07.52.50

That’s where Go enters the picture. One of Go’s reasons for existence is to make developers more productive, limit the number of ways you can do something and have a very opinionated view of the world at the compiler level. I pushed back on Go adoption internally for a while. I worried about splintering even more, having another language for people to learn as we already had quite a few technologies in play. We have an internal policy about looking at adopting new technologies when you have at least 3 people willing to support it at 3am if there’s a production issue. After much prod’ing (thanks Sean Berry) and getting past that 3 number, I dug in to the Go world and saw it solved a lot of issues I had with Scala at the organizational scaling level.

Fast build times, small binaries, one file, built in formatting, great tooling, built in test framework, race detector, visual profilers, a nice concurrency model? Wow, sold! We did a sample project in Go that was successful, then another, then another, expanded out the number of developers we had on Go and it started to become the language of choice people wanted to write in. You can jump into any Go project and know immediately what it’s doing. Do I miss immutable types and some of the great features of Scala? Sure do, but I think the maintainability side of the story is too great to overlook with Go. We’ve seen faster ship times, better stability and better test coverage being written.

One of the other benefits of Go was widening the pool of backgrounds we can hire. We can take someone from any language background and have them ramped up on Go in weeks. With the Scala side there’s the JVM learning curve, the Java world of containers, black magic of JVM tuning, profiling tools,  etc…

New developers we’ve hired are ramped up in weeks vs months (we have lots of services that operate at extremely high scale, across several divisions).  We now have the majority of our services written in Go and one of the last holdouts to move to Go just wrote his first project and said to me afterward “Wow, I read through that library once and I knew exactly what it was doing , I’ve read the Scala version of that library four times and I still have no idea what it does, I can see why you guys like it so much”. That was one of our senior engineers who’s previously worked for one of the largest web properties in the world. This process was a complete bottoms up initiative from our development team who pushed for the move to Go.

We now process hundreds of thousands of messages per second and Terabytes of data per day with our GoLang services. Some try to equate Go’s simplicity with weakness, I’ve seen the opposite. You can do some pretty powerful stuff in Go. There is power in simplicity. The error handling, while seemingly annoying at first, actually has lead to more robust error handling and stability in application code. You can’t just throw something and hope it gets caught somewhere.

Google is also a major investor and having that technical alignment opens up access to additional resources that we can leverage.

I’m not here to bash Scala, or ScalaZ (I love ValidationNel! ) but more to give some real world context of Scala in a production environment over 7 years with two companies. I still use Scala and love hacking in Scalding.  Some of our more ambitious projects coming up will most likely be Scala based but I’m just not as sold on it anymore as the core language when trying to scale a fast growing engineering team. There are always exceptions and if your team really loves Scala you can make it work, and some companies are.

Go sits in a place that happens often… when you need small’ish, high performance services. Where you’re doing light transformations, shuffling data around and putting APIs in front of data or supporting systems.

Go just makes it too easy.

 

If you’re interested in hearing more or chatting, follow me on Twitter https://twitter.com/jimplush

…and we’re hiring 🙂

 

Updates:

** Marius, one of the twitter Scala gurus, also seems to be getting keen on Go.

screenshot-2016-09-24-08-58-56 screenshot-2016-09-24-08-58-42

 

 

68747470733a2f2f7261772e6769746875622e636f6d2f676f6c616e672d73616d706c65732f676f706865722d766563746f722f6d61737465722f676f706865722e706e67

——

[1] Code Reviews would have solved this issue as someone else would have stumbled on that symbol and asked for more clarity and understanding. That would have help introduced the library to the team easier than a surprise. Unfortunately, at Gravity we didn’t have required code reviews in place. We do have code reviews in place at CrowdStrike. However, given not every person is on every code review there’s no guarantees.

 

 

 


Share the joy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

13 Comments

  1. Interesting post…
    I’m start to learn scala 4 years ago, but I think scalaz and friends are not my friends…

    I think it’s some place where it’s useful but the results are very creepy for a less experienced developer. So I agree with this part but I think scala is one of the most powerful language and you can write nice functional and reactive application without creepy codes 😀
    Creepy codes created by a developer not by the language, I think you can write creepy code in go lang too. I love scala syntax, and most case I love the operator overloading, I love the functional programming style and I think if you use in the right way your code is clean (cleaner than java).

    Why go? Why not nodejs or D ?

    • +1
      Nothing is perfect, and the scala problem seems to be complexity … as a trade for it’s powerfulness !
      I think a good response to that is strong, community driven, scala best practices like :
      https://github.com/alexandru/scala-best-practices

      Thanks for this post anyway Jim !

    • Basically anything that makes code creepy is either impossible or easily detected as Code Smell in Go (e.g. unused variables are not allowed by the compiler, and if you go around that with _, you have something you can grep for to get rid of that stuff. Same thing with e.g. empty interfaces). I used to work with Lisp before (and I think that for the stuff most people use Scala for, Clojure most often is a much better choice, if only for consistent syntax!)

      Talking about Clojure, Go has the same main reason to use it: simplicity. Simplicity is a huge advantage in a business situation, and is the one thing you cannot bolt on a programming language.

  2. Nice summary. I’ve seen similar experiences at my previous company. I’m actually dabbling a bit in Go now and enjoying it so far.

    thanks for sharing

  3. I wonder if you ever considered Kotlin? Sounds like it would hit a sweet spot by giving you a lot of the power that Scala gives you without a lot of its baggage while keeping you on the JVM.

  4. Well written article Jim, golang is definitely worthy way forward in terms of microservices. In all trueness Scala was bit early in the business though and it was the time when dev teams were considering how to reduce verbosity of java but not loosing the JVM aspect of it.

  5. Michal Kunikowski

    December 22, 2015 at 1:42 am

    What about other alternatives like Clojure, Kotlin, Elixir?
    Go is not functional language. I hate for (i = 0; < n; i++ ) style programming 🙂

  6. We’ve had the exact same issues and we’ve been using Scala in major roles since 2009.

    scalaz is a great tool, but it leads to write only code and concentration of knowledge into that half of the developers who can read it. It’s not that it’s impossible to learn, it’s that it everything just takes 3-4x as long to debug.

    I still enjoy writing Scala (I’m firmly on the left side of that graph) because it is much faster than writing the equivalent Java. But the compile times are annoying, and library bitrot is also annoying. We’ve basically internalized all external libraries that aren’t ScalaZ, Akka, etc so that we can keep them up to date with the latest Scala releases.

    I personally enjoy writing GoLang far more than writing Scala these days. It still has a long way to go (the stdlib is kind of … brittle) and I miss java.util.concurrent.* on a daily basis, but at least I can read what someone else wrote, and I can get services up within a couple of hours.

    Would I write our codebase in Scala today? Unlikely. Do I regret it? Not really. There’s something to be said for JVM compatibility and a wide selection of high quality libraries.

  7. I experienced the same problem even with a small team.
    Specially one man team. When a developer leave the new one takes months to produce a production code again.

  8. Interesting. Can you elaborate on the technical benefits achieved by making this move? I mean, by ditching Scala you ditched JVM with all the benefits. But perhaps there are technical details in the your requirements that benefit from Go more than from JVM?

  9. “Wow, I read through that library once and I knew exactly what it was doing , I’ve read the Scala version of that library four times and I still have no idea what it does,”

    This phenomenon is the hallmark of imperative, statement-based programming going back to the 1950s. It is also indicative of limited exposure to programming and programming languages. The ability to step-through a program, including the sequencing of assignment statements, conditional statements, and iterating through a loop gives a program a familiar recipe-like feel, which is attractive to beginners and journeyman programmers (and hence the attraction of lowest-common-denominator tech choices for large groups).

    But there is a dark side. Peter Landin noted in 1966 that Algol 60 programmers tried to avoid “explicit sequencing”. The basic problem is that statements don’t compose, and the sequencing introduces gaps for bugs to fill. Conditional statements have a similar problem, where the hierarchy of nested predicates split up the logic. I believe that Go does automatic bounds checking (Tony Hoare must be relieved), but in general, loops also have a “separation” problem where the control structure is not the same as the data structure.

    It takes a lot of discipline to overcome these problems. And time is the enemy, as large numbers of programmers trudge through the code base making changes. An increased testing and code review regimen can help mitigate this, but necessarily means adding more programmers.

    Nitpick – you might want to reword “immutable types”.

  10. I am increasingly of the opinion that the problem domain that Scala most thoroughly solves is modeling complex systems in as expressive manner as possible. What separates Scala from the pack of programming languages is its rich type system . Scala’s lifeblood is in its ability to model nearly any data structure or control structure that is needed to work within your problem domain.

    Much is made of the purity of Scala’s functional features, but what really seperates it from the pack of modern program languages is its SmallTalk-esque object oriented purity that allows the creation of powerful behavior driven abstractions.

    There’s quite a bit of power there – also quite a lot of responsibility. If you do not require that power the juice probably isn’t worth the squeeze. For lower level systems programming where you are dealing with fairly simple abstractions it isn’t really worth it.

    In my own academic work I’m using Scala for domain modeling and machine learning, but utilizing Go for lower level work.

  11. Thanks for sharing your experiences with scala & go !

Leave a Reply

Your email address will not be published.

*

© 2016 Jim Plush: Blog

Theme by Anders NorenUp ↑