Docker Monitoring

It is been a while since I got my first docker application (Zenoss) running. Since I am a big fan of monitoring, this blog will talk about bit more about docker monitoring. All the stuff here are from docker documentation, other’s blogs or online training videos. So I would not claim any copyright of this blog. ( It is open source anyway :))

When comes to monitoring, the business requirements and deployment scenarios are the key things to understand. According to my understanding, I will split into 3 use cases.

  1. Public cloud deployment
  2. Private Cloud deployment ( in house )
  3. Hybrid cloud deployment

Before diving into these 3 scenarios, let’s look at some basics about docker statistics and best practise, my best practise.

Docker Statistics

 

My Best Practises

Label your container

Allocate resource to docker containers

 

Public Cloud deployment

 

 

 

Clean up docker containers and images

For some reason, docker is not that friendly to disk space and different applications may have different issues in terms of cleaning up old stale docker containers and images. When docker directory is filling up, your application would die. So keep an eye on the disk utilisation of docker partition is very critical. You shall monitor this by either scripting or third party monitoring tool.

Try the below commands, they are tested on CentOS 7 with docker: 1.5.0 , build a8a31ef

  1. docker images | grep “<none>”

find out the image ID, then run docker rmi $imageID

2.  docker rm `docker ps -a | grep Exited | awk ‘{print $1 }’`

3.  If your application is running, it is safe to use the following as well, do not run this when your application service is not

running, a good practice is to take a snapshot if you are uncertain about the results.

docker rmi `docker images -aq`

Change the default docker0 interface address

I was deploying Zenoss core 5, as you may know, zenoss core 5 is packed into a docker container. so the deployment is to get the docker installed, then install the Zenoss Control Center, then download the proper docker image. To me, while it simplifies the deployment of Zenoss Core 5 itself and ease the backup/restore, it actually introduce some overheads to manage the system.

One of the issues I met was on default docker0 interface, by default, the interface has an IP address: 172.17.42.1, but I have a few subnets which are in 172.17 range, so I had some trouble to reach those devices because of default routing. So I have to change the default address.

Steps to change the docker0 interface address:

I run Zenoss on CentOS 7, so the steps here only cover CentOS.

1. Stop Serviced ( control center process )

systemctl stop serviced

2. Stop docker

systemctl stop docker

3. change the file /etc/sysconfig/docker

default: DOCKER_OPTS=”-s btrfs –dns=172.17.42.1″

change to: DOCKER_OPTS=”-s btrfs –bip=192.168.10.254/24″

192.168.10.254 is an example here, you need to pick up the range suits your environment.

4. run the command below :

ip link set dev docker0 down

brctl delbr docker0

iptables -t nat -F POSTROUTING ( flush the nat table is very important, you may still have the old entries which was using the default 172.17 range, which will cause network issues )

5. bring up the docker service and the new address should be assigned to docker0

systemctl start docker

6. You might want to inject another rule into iptables to allow the docker0 and other virutal interfaces to be able to communicate with external world.

iptables -A INPUT -d 192,168.10.0/24 -j ACCEPT

If you still have trouble, you need to check carefully of your iptables, local policy routing etc.

You can also check the docker advanced network here:

https://docs.docker.com/articles/networking/#customizing-docker0