Play Framework Grid Deployment with Mesos
Play Framework is now being deployed across large grids and it’s working great, but these large deployments add to the complexity of delivering updates. To reduce that complexity, we worked with Mesosphere to describe how to use Mesos for deploying Play apps across a grid. First a little background...
Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. It can run Hadoop, MPI, Chronos, Spark and other applications on a dynamically shared pool of nodes. Prominent users of Mesos include Twitter, Airbnb, MediaCrossing, Xogito and Categorize.
Mesos leverages features of the modern kernel – “cgroups” in Linux, “zones” in Solaris – to provide isolation for CPU, memory, I/O, file system (in progress), rack locality, etc. The big idea is to make a large collection of heterogeneous resources appear as if they were one big computer. In that sense, Mesos is similar to Google’s “Borg” and “Omega” projects for warehouse-scale computing.
Chronos and Marathon are frameworks based on Mesos, and both are used to manage the use of other frameworks. On the one hand, Chronos serves as a distributed “cron” that runs tasks on a Mesos cluster – any task that can be defined by a shell command line – and it includes support for dependency graph, retry logic, no single point of failure, etc. For example, one might have a Chronos task run a Hadoop job periodically. On the other hand, Marathon serves as a distributed “init” or “upstart” for highly-available, long-running services – and it includes service discovery, service constraints, metrics, event subscription, etc. For example, one might run a number of web app instances as a Marathon app. Taken together, Chronos and Marathon provide building blocks for creating new distributed apps. Both have browser-based UIs, REST APIs, command line support, etc.
Given that you have Mesos running as the kernel for your data center, Marathon is the “upstart” daemon. As a simple example, imagine that you want to run 10 instances of a Play! application somewhere within your cluster, and that each of the Play! apps requires 1 CPU and 1 GB of memory. You submit this request to Marathon, which creates 10 Mesos tasks – with each representing one instance of a Play! application. These instances get executed on the slaves.
If one of these processes dies for whatever reason, it gets restarted somewhere in the cluster. The image below depicts a Mesos/Marathon cluster that runs both Solr as well as a Play! application. The left image shows the initial cluster after launching instances. The right image shows the cluster after Marathon has recovered from a machine failure.
Marathon provides a REST API for starting, stopping, and scaling services. There is a browser-based GUI and also a Ruby command line client. It can run in highly-available mode by running multiple Marathon instances. Marathon was written in Scala, by the team that developed Chronos, and a public code repo is available on GitHub.
Play Framework is an open source web framework that helps developers quickly build modern web & mobile apps using Java and Scala. Play features a containerless deployment model that supports Continuous Delivery and reduces runtime overhead. To create a distribution of a Play application for deployment simply run:
This compiles the application and pulls all of the applications dependencies into a self-contained zip bundle that can easily be deployed through Marathon. By including the runtime dependencies of the application in the bundle you can avoid the “it works on my machine” problem that plagues container deployed applications. An optional by recommended approach to generating a Play app’s distribution zip is to use a Continuous Integration system such as Jenkins. Simply use the sbt plugin for Jenkins and have it run the “dist” task automatically.
In order to run our Play application via Marathon / Mesos, we need to upload it to a location that is accessible from the Mesos cluster…
Installation of Mesos and Marathon
Mesos is available as a package on a wide variety of servers, and Marathon, as a standalone Jar, is also quite portable. We'll cover installation in detail on Ubuntu and OS X -- for both Mesos and Marathon -- before touching on available packages and options for other platforms.
Installation on Ubuntu
On Ubuntu, we can leverage the prebuilt packages and Upstart to bring everything up together in working order.
#### Install Mesos and Zookeeper distro_version="12.10" #### Change on 12.04 or 13.04 curl -fL "http://downloads.mesosphere.io/master/ubuntu/$distro_version/mesos_0.14.0_amd64.deb" --output /tmp/mesos.deb sudo aptitude install -y unzip default-jre-headless zookeeper-bin zookeeperd sudo dpkg -i /tmp/mesos.deb #### Install Marathon sudo mkdir -p /opt/marathon sudo curl -fL "http://downloads.mesosphere.io/marathon/marathon-0.0.6-SNAPSHOT-jar-with-dependencies.jar" --output /opt/marathon/marathon.jar sudo chmod ug+rx /opt/marathon/marathon.jar sudo curl -fL "http://downloads.mesosphere.io/marathon/marathon.conf" --output /etc/init/marathon.conf
A reboot will refresh your service configuration and bring up the whole ensemble. You can reload the service configuration and start each service manually, too:
sudo initctl reload-configuration sudo start zookeeper sudo start mesos-master sudo start mesos-slave sudo start marathon
Installation on OS X
Installation on OS X is easy, too.
#### Install Mesos curl -fL "http://downloads.mesosphere.io/master/osx/mesos-0.14.0.pkg" --output ~/Downloads/mesos.pkg sudo installer -package ~/Downloads/mesos.pkg -target / #### Install and configure Zookeeper brew install zookeeper sudo cp -a /usr/local/etc/zookeeper/zoo_sample.cfg /usr/local/etc/zookeeper/zoo.cfg #### Install Marathon sudo mkdir -p /opt/marathon sudo curl -fL "http://downloads.mesosphere.io/marathon/marathon-0.0.6-SNAPSHOT-jar-with-dependencies.jar" --output /opt/marathon/marathon.jar
We need to run four services altogether. Running each one in its own terminal
is a reasonable choice for development mode.
#### Start Zookeeper zkServer start-foreground #### Start the master /usr/local/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 #### Start the slave /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos #### Start Marathon java -Xmx512m -Djava.library.path=/usr/local/lib -cp /opt/marathon/marathon.jar mesosphere.marathon.Main --zk_hosts localhost:2181 --master zk://localhost:2181/mesos
Hints for alternative installations
Mesos is available as a package on a variety of distros.
The Upstart scripts for Marathon will work on RedHat and CentOS, too. Finding an appropriate source for Zookeeper on those distros is a matter of selecting a Hadoop distribution and adding their repository.
Deploying a Play App
Once you have the Mesos and Marathon system up and running and have created a zip distribution of a Play app, you will need to upload the zip file to an http server that can be reached from the Marathon server. One option is an Amazon Web Services S3 bucket. Jenkins also can provide a way for Marathon to get the zip over http.
Marathon and Mesos provide a web UI through which one can manage, monitor and shutdown applications.
Marathon’s 23rd Century UI
Launching a new application
The app is running
The HTTPie tool provides an easy way to send JSON to marathon for launching a Play application from the console.
http -v POST http://localhost:8080/v1/apps/start \ id=play_nice cpus=2.0 mem=512 instances=1 \ cmd='bash foo-*/start -Dhttp.port=$PORT' \ uris:='["http://downloads.typesafe.com/tmp/foo-1.1-SNAPSHOT.zip"]'
By enabling verbose mode, we can see how Marathon’s simple HTTP API works under the hood:
Polling Marathon’s endpoints API shows us where the app is running and a quick check of the logs confirms. We can reach our Play app on port 31000.
Service Discovery and Load Balancing
Now that our app is up and running, we need a way to send outside traffic to it. If we're running multiple apps, they also need a way to find each other. A simple way to do this is to run a TCP proxy on each host in the cluster, and transparently forward a static port on localhost to the hosts that are running the app. That way, clients simply connect to that port, and the implementation details of discovery are completely abstracted away.
Marathon assigns each app a unique port on startup and keeps track of the hosts and ports of app instances across the cluster. This information can be queried at /v1/endpoints, and Marathon ships with a simple shell script called haproxy_cfg that turns it into a config file for HAProxy, a lightweight TCP/HTTP proxy.
To generate an HAProxy configuration from Marathon, use the haproxy_cfg script:
./bin/haproxy_cfg localhost:8080 > haproxy.cfg
To reload the HAProxy configuration without interrupting existing connections:
haproxy -f haproxy.cfg -p haproxy.pid -sf $(cat haproxy.pid)
The configuration script and reload could be triggered frequently by Cron, to keep track of topology changes. If a node goes away between reloads, HAProxy's health check will catch it and stop sending traffic to that node.
Learn more about creating and using Play app distributions in the Play docs:
Apache Mesos web site:
GitHub repo, email list, docs, etc., for Marathon:
Mesos USENIX 2011 paper: