Run a Mesos / Marathon / ZooKeeper Cluster on Docker

Run a Mesos / Marathon / ZooKeeper Cluster on Docker

Overview

This is a quick post that was inspired by this one but updated to use images available straight from Docker Hub. This is essentially the tutorial that I wish I had seen before I went through the effort to make it all work. Hopefully it will help someone out there.

NOTE: I have created a docker-compose file that runs this all in one command. See https://github.com/tellmejeff/mesos-marathon.

Mesos, Marathon, and Zookeeper

What I'm building today is just a simple little cluster that allows you to schedule tasks on available resources. Mesos does the job of managing the available resources, Marathon is the tool for scheduling the tasks on the Mesos cluster, and Zookeeper is used by Mesos to syncrhonize the configuration of the cluster so that it has redundancy and can recover if one of the hosts in the cluster goes down. Mesosphere has a vagrant project that will let you spin up some virtual machines to do this, but I found that this was a little resource intensive and wouldn't work in my lab cloud environment that was itself a virtual machine. I found a post on doing it using Docker, and it was a little out of date, but by tweaking the images used and the actual commands, I found I was able to get the cluster up and running pretty quickly.

Docker Required

This does require that you have Docker installed, but fortunately that is generally pretty easy with whatever package management tool you use for a Linux distro. It should also work on Windows, although I haven't tested it. I used a Fedora 27 instance running on Digital Ocean for this tutotiral.

Steps

Step 1:: Get the IP of your server and put it into a HOST_IPenvironment variable. If you have multiple network interfaces on your machine, it doesn't really matter which one you use, as long as they are consistent between the containers. I recommend using the one that you can point a web browser to, which will likely be your main eth0 interface.

Note: If you are doing this on a cloud instance like I was, chances are you don't actually have the ability to launch a browser directly on the server (or trying to tunnel X through ssh would be really slow), so you will likely need to open some firewall ports if you want to see the UI interface for Mesos and Marathon. Mesos will be listening on port 5050, and Marathon on port 8080.


$ HOST_IP=10.11.31.7

Step 2: Start the ZooKeeper container.


$ docker run -d \
-p 2181:2181 \
-p 2888:2888 \
-p 3888:3888 \
zookeeper

Step 3: Start the Mesos Master. This container uses the host network, listens on port 5050, and has a few options set for Mesos, including pointing at ZooKeeper. Unfortunately, the mesos-master image does not have a latest tag (of this writing) so I pinned it to 1.4.1.


$ docker run --net="host" -p 5050:5050 \
-e "MESOS_HOSTNAME=${HOST_IP}" -e "MESOS_IP=${HOST_IP}" \
-e "MESOS_ZK=zk://${HOST_IP}:2181/mesos" -e "MESOS_PORT=5050" \
-e "MESOS_LOG_DIR=/var/log/mesos" -e "MESOS_QUORUM=1" \
-e "MESOS_REGISTRY=in_memory" -e "MESOS_WORK_DIR=/var/lib/mesos" \
-d mesosphere/mesos-master:1.4.1

Step 4: Start Marathon. This container listens on port 8080 and is configured to point at ZooKeeper.


$ docker run -d -p 8080:8080 mesosphere/marathon \
--master zk://${HOST_IP}:2181/mesos \
--zk zk://${HOST_IP}:2181/marathon

Step 5: Start the Mesos Slave. It actually uses the same image as the master (there is a separate slave image, but I suspect it is pretty much the same, just with a different entrypoint) and sets some options for mesos and points to ZooKeeper. Again, the image is pinned to 1.4.1 because the Docker Hub doesn't have a latest tag set for this image.


$ docker run -d --name mesos_slave_1 -e "MESOS_LAUNCHER=posix" \
-e "MESOS_SYSTEMD_ENABLE_SUPPORT=false" -e "MESOS_WORK_DIR=/tmp" \
--entrypoint="mesos-slave" -e "MESOS_MASTER=zk://${HOST_IP}:2181/mesos" \
-e "MESOS_LOG_DIR=/var/log/mesos" -e "MESOS_LOGGING_LEVEL=INFO" \
mesosphere/mesos-master:1.4.1

Step 6: Go to the mesos homepage. It should be at http://<HOST_IP>:5050, substituting HOST_IP with the IP address you specified in the Docker commands. You will get a page that looks like this (without the tasks of course):

mesos

Step 7: Go to Marathon's homepage to start a job. This is the fun stuff, why we are doing this. For this tutorial, we are going to only do something really simple, but there is a lot of different things that Marathon (which really is powered by something called DC/OS that has a nice JSON REST api) can do,including running Docker images in containers. Marathon should be at http://<HOST_IP>:8080, substituting HOST_IP with the value you passed to the Docker commands. You will see a page that looks like this:

marathon

Now, let's create an application by selecting Create Application. This will bring up a modal where we fill out the details of the application. Really, you only need to fill in an id (just a unique identifier for the application - valid values are letters, numbers, dashes, periods and slashes) and the command to run. In this case, I'm just scheduling a task that will echo "Hello World" and append it to a file:

hello_world_marathon

Then simply select Create Application at the bottom, and away we go. When I do this, it will deploy the task and then complain that it gets delayed, but it will actually run it. To see the output in action, we need to connect to the mesos slave container where the task is actually run. To do this, we run this command:


$ docker exec -it mesos_slave_1 /bin/bash

When we started the slave, we gave it a specific name, and this Docker command indicates that we wish to connect interactively to this container and run the given command, which happens to be a shell. Now you will see a prompt inside the container, and we just want to tail the file we specified in our command. You will see that Hello World appears every few seconds. Why does it repeat it? Because an application should be always running, but ours does one thing and then quits. By default, Marathon starts one instance of an application, and if for some reason the instance dies (or finishes), Marathon automatically starts another one for us, and therefore we keep seeing the text added to the file:

hello_world_output

Conclusion

That is all there really is to it. Now of course when you run production traffic on this, you might want to run these tools outside of a container, but for demonstration and local development purposes, this will work pretty nicely. Soon I will dive into how to run a Docker container on this.

Related Article