Docker Troubleshooting

Docker Troubleshooting

Disk Usage (MergeDir)

If you run out of disk usage and it's not the data from your volumes, maybe a container is writing data to the overlay file system. This data is placed in the merge directory. Try the following snippets to show the size of the merge directory for all the containers.

while read -r name mergedir _; do
    size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"
    size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"
    echo "${name} ${mergedir} ${size:=0} ${size_hr}"
done <<<"$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t

The same as a one-liner.

while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t

This is how the output could be displayed.

Example

root@server:~# while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t
NAME                              MERGE_DIR                                                                                         SIZE_B  SIZE_MB
/zabbix-db_data_pgsql-1           /var/lib/docker/overlay2/93e67764c8b6f74e5686a5421c2aa3e79e8a11e81eb5869fbb13ad13ccb46130/merged  0       0.00
/zabbix-zabbix-server-1           /var/lib/docker/overlay2/9e1ecf3b598501ff76c28264c73cd34bf9697fdf7605ad467f8b39a7f8f36fca/merged  63604   62.11
/zabbix-zabbix-web-nginx-pgsql-1  /var/lib/docker/overlay2/4942fc913e4546c7090fba3614c9718607f7b0274abbc50ea6feed252efed58a/merged  179516  175.31
/zabbix-postgres-server-1         /var/lib/docker/overlay2/d7918d5dea06a3df43c847c91368341e56bda720a68cd6a7be6296dcb9a97f27/merged  272720  266.33
/zabbix-zabbix-web-service-1      /var/lib/docker/overlay2/eebdcc8ee3b0b2fc871db6c32a8988a52b4b200996ff28aa015e2bec8f039eb0/merged  754740  737.05

About Namespaces

https://www.redhat.com/sysadmin/container-namespaces-nsenter

Network

Enter Namespace

container_id="mycontainerid"
container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")"
nsenter -n -t "${container_pid}"
exit

After executing nsenter all commands that support namespaces see the network interfaces like they are in the container. Example commands: ip, tcpdump. With exit the namespace can be left to see the network interfaces from the host system again.

tcpdump in Container

tcpdump has to be installed on the host system and not in the container.

container_id="mycontainerid"
container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")"
nsenter -n -t "${container_pid}" tcpdump -nn -i eth0

Logs

Add Local Timestamp to Container Logs

If a container is logging without a timestamp or with an unknown time zone, you can add a timestamp while watching the logs.

docker logs mycontainer -n 0 -f | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L'

In case you want to catch the stderr logs, too

docker logs mycontainer -n 0 -f 2>&1 | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L'