Our whole company is excited about the results from our Going Reactive 2016 survey. So, I asked the source (and my teammate buddy), Oliver White, to give us an overview.
[TRANSCRIPT]
Tonya Rae Moore: Hi, and welcome to what I think is our fourth Typesafe AMA podcast with our experts. My name is Tonya Rae Moore, and I am a Product Marketing Specialist with Typesafe. I write some of our blogs, I record our webinars, I do this podcast, and in general I talk about our Reactive Platform. Today I am super super super excited to talk to Oliver White. Oliver is my teammate at Typesafe, and he is actually the person who brought me on to Typesafe. So we go way back in the Java world. And I'm very excited to actually work with him. It's been super fun so far. So Oliver, hi.
Oliver White: Hello. I've heard a lot about this podcast, Tonya. I'm really happy to finally join.
Tonya Rae Moore: Yeah, I'm really excited to finally have a project that's mine instead of yours. You are all over our Typesafe stuff.
Oliver White: And here I am now.
Tonya Rae Moore: I know. When worlds collide. We've crossed the streams. Can you tell me a little bit about your career path and what led you to Typesafe?
Oliver White: Let me start the day I was born. No. In 2010, I joined a company called ZeroTurnaround that makes the JRebel and XRebel tools. Very good stuff for Java developers. During my time at ZeroTurnaround just over four years, I developed a content and research brand within ZT that was called RebelLabs. And one of the things that I really enjoyed doing at RebelLabs was reaching out to the community and getting data and sentiment analysis on tools, methodologies, best practices and so on. So the content angle is something I've been focused on for many years now. And I joined Typesafe at the beginning of 2015 and it's been an incredible year, the culmination if which I have to say is Going Reactive 2016, the survey we just released and what we're talking about today.
Tonya Rae Moore: Yeah, that's really exciting. I know that you have been busting your butt ever since I joined in July basically to work on this. Can you tell me a little about what the idea was behind the survey? Were there gaps of information you wanted to fill? What were you interested in that got this survey off the ground?
Oliver White: Of course there's always gaps to fill. And the more we talk to people and learn their opinions and their practices in the real world, the more we know. So in this survey, we set out to look into the state of Reactive systems in the JVM world, and to understand a little bit more about the journey of Going Reactive, as we call it. Adopting Reactive systems that are loosely coupled but highly scalable. Resilience against failures. They know how to self-heal. Failures are met with elegance rather than a total disaster. And you end up with an overall consistently responsive system no matter what bad stuff is happening to it. And these systems we see arising in more places, not just the Twitters, Netflixes, and Amazons out there, but we see them every day among our own clients and also the wider JVM community.
Tonya Rae Moore: Yeah, and the little guys are really starting to pick it up at this point, too. Right?
Oliver White: Yeah, and not only smaller enterprises, but also major mainstream enterprises that you wouldn't necessarily associate with cutting edge system architecture, at least. But now that we see them, and these are companies like Walmart, UniCredit Bank, and a bunch of them on our case studies list: Samsung, Intel. This is going mainstream. And the point of the survey was to actually put some numbers behind it, get some data. And I'm proud to say that we had over 3,000 respondents, which was an excellent population for a survey of this size.
Tonya Rae Moore: Yeah. So what can you tell me about this? Who responded? Did you crowdsource? Who were the respondents? What happened?
Oliver White: We did, of course, reach out to our internal contacts at Typesafe. But in addition, we had coverage on Voxxed.com, through the O'Reilly Media Newsletter, TheServerSide.com, lots of social sharing. So we did try our best to get as broad of an audience as possible. Naturally, our customers and developers that are familiar with our stuff are probably more likely to respond to the call for the survey. That said, we have just a 2 percent margin of error at a 95 percent confidence interval. So we generally stayed away from drawing any conclusions that fell within that margin of error in either direction.
Tonya Rae Moore: Yeah, it's always hard on surveys. People are often nervous about the numbers, where you get them from, and how you can manipulate them. So it's good that you tried to spread it around as much as you possibly could.
Oliver White: Right. Well, my goal was 350,000 respondents. But I fell just a bit short and I didn't have another 2 years to get the word out to collect that. But I always want to have as many respondents as possible.
Tonya Rae Moore: You should've promised beer.
Oliver White: Well, they can have a beer and do the survey. I thought that was assumed!
Tonya Rae Moore: Booth opportunity, booth opportunity! All right, so tell me what you found in your survey.
Oliver White: I guess to put it in a nutshell, we can bring it down to three major conclusions. And we'll dig into the numbers behind those in a little bit. The first one is we have evidence from this survey that Reactive system adoption is going mainstream. The second point is that this adoption is being especially driven by two key trends. The first is microservices-based architectures. Yay, everyone's favorite buzzword: microservices. And fast data systems in which we're looking at data in motion. Streaming data rather than offline batch mode data. And a lot of that is revolving around Apache Spark. And we'll talk a little bit about that, too. And the third conclusion is that microservices and fast data system users are rallying around this specific group of tools and technologies. There's a list of around 10 specific technologies that show significant increases in adoption when we look at just these subsets of the overall population. And we'll talk a little bit about that, too.
Tonya Rae Moore: Okay. Well, the one that's most interesting to me is that Reactive is going mainstream. This is no longer just a buzz term. Like this is happening now, people. So what did you find out about that?
Oliver White: I invite everyone to obviously read the report, because there's a little bit too much to talk about in the podcast. But the summary of the Going Reactive section in which we ask people about how they feel about the trend, where are they in the journey, how close are they to developing and deploying Reactive systems in production and so on.
The first interesting statistic is that 83 percent of all respondents say that Reactive is something that demands attention. It's no longer a fringe concept that's only enjoyed by a handful of early adopters. Reactive system architecture and the concepts behind Reactive systems, message driven, resilient, elastic, responsive. There are things that people are finding that they want. And how they're getting there, whether they call it Reactive or not, is not the point. Just a little bit over 4 out of 5 respondents believe that Reactive systems is a topic that demands attention. Following closely after that is this prediction for enterprise adoption of Reactive systems. 80 percent of respondents believe that most successful enterprises will have adopted Reactive systems by the year 2018. That's two years away.
Tonya Rae Moore: Wow, that's quick.
Oliver White: Well, that's a high expectation. This is asking about most successful enterprises, the respondents' opinions on who they consider to be successful. 80 percent of them believe that they'll have fully adopted Reactive systems by 2018.
Tonya Rae Moore: Wow.
Oliver White: To continue, 43 percent of respondents are already researching or prototyping Reactive systems. So they've started the journey. They're looking at reports like this, researching and starting projects, which is great to see. And 34 percent, just over one-third, are actually building and deploying production systems. And I'll refer to this 34 percent of all respondents, so 1,040 people, as our power users in some of the other areas.
Tonya Rae Moore: Wow, so Reactive is no longer a fringe group. They've gone Billboard Top 100. This is huge. So how do you read microservices into this?
Oliver White: All right, so one of the things we did is look at the phase of journey among the respondents. So for the sake of simplicity we can break it into three stages. The first is just learning. Perhaps this survey is the first that they've actually started to look into Reactive. Then we have the researching and prototyping group that I mentioned before, that 43 percent. And then we have the power user group that are building and deploying Reactive systems and production.
So when we asked respondents whether they're working with microservices-based architectures, one-third of all users said that they were. However, if we look at these groups along the phase of journey going from just learning to ultimately running production systems, we see microservices adoption rise significantly. It goes from 16 percent, to 28 percent, to 50 percent, the final group being the power users. So this shows an orders of magnitude I guess you could say increase by the phase of journey. So the closer people are getting to launching into production, the more likely they are to be working with microservices-based architectures, as many as half of them.
Tonya Rae Moore: I'm just really impressed. I had no idea even working for this company that this was gaining so much momentum so quickly.
Oliver White: I can't say we expected this, but the numbers don't lie and this is what we've seen. So while we were looking at these significant increases appearing in certain groups, we turned this to the tools and technologies that we listed in the report in the survey.
There were 62 different tools and technologies to choose from. I'm sure there could've been many more, but we ran out of steam after 62. And this covered developer tools, infrastructure, DevOps tools, as well as big data, fast data, computation storage tools. So when we look at the microservices-based architectures group, the people that indicate that their system is using a microservices-based architecture, we see, again, very significant increases in a specific toolset. And I'll run through those very quickly. [ACA], Apache Cassandra, Apache Kafka, Apache Spark, Play Framework, Amazon EC2, Docker and Apache Mesos. I won't go too deep into this in the podcast, so you'll have to check out the full report for this. But long story short, significant increases in usage from the average respondent to the microservices-based architectures group. And this specific toolset is what they selected.
Tonya Rae Moore: So what did you see from these findings? What did you see about fast data and why Spark?
Oliver White: In earlier this year -- I was going to say in 2015. It's still 2015.
Tonya Rae Moore: We were just doing the content calendar before we started this podcast, so we are so far into 2016. We've got to finish this year!
Oliver White: I'm glad I'm not saying 2017. So earlier this year, in fact, my first project at Typesafe was to consolidate the results of an Apache Spark survey that we did in which over 2,000 people responded. One of the major drivers that we see among our customers is the embrace of data in motion, like I said. Not this offline batch processing, but a stream-based architecture where data is coming in, going out, and being spread around distributed systems running lots of different tools in order to get value from that data. Apparently, Apache Spark is the Apache Foundation's most popular project of all time. I heard this. I hope I'm not wrong. So when we talk about data in motion, we're referring to Apache Spark as one of the components in there. But generally it's fair to say that looking at Apache Spark users and the tools that they're embracing and the methodologies, we see the connection to Reactive systems.
So again, we looked at the phase of journey from just learning to building and deploying. And we saw, again, a steady increase. Not as significant as microservices, but a steady increase going from 13 percent who are just learning, 21 percent adoption of Spark between the group that is researching and prototyping, and 28 percent adoption for those that are building and deploying Reactive systems in production. And the average for Spark was 22 percent usage out of the total 3,060 respondents.
Tonya Rae Moore: That's pretty significant.
Oliver White: Yeah, that's a nice rise. Again, we look at the tools that have significant increases in use when we look at Apache Spark users. And we see this interesting list that very much parallels the specific tools preferred by microservices-based architectures. We see, again, ACA, Amazon EC2, Apache Cassandra, Docker, Hadoop, Kafka and Mesos.
Now I'll spend just a second on Hadoop and where does Hadoop fit in the picture. So a lot of people hear something along the lines of Spark is killing Hadoop. And it's important to note that Hadoop MapReduce is actually the component that Spark is being used to replace in many cases. Hadoop as an ecosystem is very well established and there's lots of different projects and uses. So what we see is that 61 percent of Hadoop users are using Spark. which is great because you see this full embrace of Spark in the Hadoop ecosystem. And when we look the other way, we see that 46 percent of Spark users are using Hadoop, which is also a high number. But it also shows that Spark has been firmly established outside of the Hadoop ecosystem, and there's a lot of interesting things coming up there. But the main point is that we see this significant increase in use of these specific tools again, and that's the conclusion of the fast data section.
Tonya Rae Moore: That's fascinating that we can even call out not what's happening and what trends are occurring, but what specifically is being used as well. I'm really only coming to the hard data analysis side of marketing and I think it's fascinating. And I think that this is amazing, Oliver. I think what you did was great work.
Oliver White: Thank you. It took a very long time and I needed a lot of help along the way. So I was glad to have a good team to help me out at times.
Tonya Rae Moore: If only I knew someone on that team. All right, well we're going to wrap it up here. It's almost the Christmas season as we record this podcast. So I'm going to let Oliver actually go shout-out to his lovely wife Babs, hang out with his family and take a vacation. Oliver, it was a pleasure to share this quarter at Typesafe with you. I really enjoyed it.
Oliver White: Well believe me, the pleasure was all mine, Tonya.
Tonya Rae Moore: You're just glad you had someone to type your reports for you. Check out Typesafe.com for the webinar and more info on our survey findings, Going Reactive. Go Reactive 2016. Woo!
Oliver White: All right, thanks everyone.
Tonya Rae Moore: Thanks, Oliver. You can also tweet at us. Tweet @theotown for Oliver, @tanyaraemoore for me, and @typesafe for Typesafe. I'll also be checking the Typesafe podcast hashtag for feedback and suggestions on improvements. Thanks to everyone who listed. And by the time you're listening to this, I hope you had a great holiday.
