Introduction to Apache Mesos
--
What is Mesos?
A distributed systems kernel
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments.
Project Features * Scalability to 10,000s of nodes * Fault-tolerant replicated master and slaves using ZooKeeper * Support for Docker containers * Native isolation between tasks with Linux Containers * Multi-resource scheduling (memory, CPU, disk, and ports) * Java, Python and C++ APIs for developing new parallel applications * Web UI for viewing cluster state
Apache Mesos is a centralised fault-tolerant cluster manager. It’s designed for distributed computing environments to provide resource isolation and management across a cluster of slave nodes.
Mesos joins multiple physical resources into a single virtual resource
Four major components of Mesos:
ZooKeepers
Mesos masters
Mesos slaves
FrameworksZooKeeperApache ZooKeeper is a centralised configuration manager, used by distributed applications such as Mesos to coordinate activity across a cluster.Mesos uses ZooKeeper to elect a leading master and for slaves to join the cluster.Mesos mastersA Mesos master is a Mesos instance in control of the cluster.A cluster will typically have multiple Mesos masters to provide fault-tolerance, with one instance elected the leading master.Mesos slavesA Mesos slave is a Mesos instance which offers resources to the cluster.They are the ‘worker’ instances - tasks are allocated to the slaves by the Mesos master.FrameworksOn its own, Mesos only provides the basic “kernel” layer of your cluster. It lets other applications request resources in the cluster to perform tasks, but does nothing itself.Frameworks bridge the gap between the Mesos layer and your applications. They are higher level abstractions which simplify the process of launching tasks on the cluster.
### System Requirements
Mesos runs on Linux (64 Bit) and Mac OS X (64 Bit). Make sure your hostname is resolvable via DNS or via /etc/hosts to allow full support of Docker’s host-networking capabilities, needed for some of the Mesos tests. When in doubt, please validate that /etc/hosts contains your hostname.
It is always better to run Mesos in multimaster mode or HA mode where you will be running more than 1 mesos master.
Mesos Master Hardware requirements:-
Minimum
Nodes: 1
OS: Ubuntu
Processor: 4 cores
Memory: 32 GB RAM
Hard disk space: 120 GBRecommended
Nodes: 3
OS: Ubuntu
Processor: 4 cores
Memory: 32 GB RAM
Hard disk space: 120 GB
There are many mixed workloads on the masters, for example Mesos replicated log and ZooKeeper. Some of these require fsync()ing every so often, and this can generate a lot of very expensive random I/O.
Solid-state drive (SSD)
RAID controllers with a BBU
RAID controller cache configured in writeback modeMesos Slave/Agent Hardware requirements:-
Minimum
Nodes: 1
OS: Ubuntu
Processor: 2 cores
Memory: 16 GB RAM
Hard disk space: 60 GBNodes: 6
OS: Ubuntu
Processor: 2 cores
Memory: 16 GB RAM
Hard disk space: 60 GB
The agent nodes must also have: A /var directory with 10 GB or more of free space. This directory is used by the sandbox for both Docker and Mesos Containerizer. Network Access to a public Docker repository or to an internal Docker registry.
Port Needed
5050 Mesos master nodes
5051 Mesos agent nodes
8080 MarathonOne of the key factor in working with Microservice infrastrucutre Cluster Manager is Service Discovery. Mesos comes with mesos-dns but I reccomend to use Consul with Mesos.
