Both sides previous revisionPrevious revisionNext revision | Previous revision |
apps:docker:troubleshooting [2025-01-29 12:45] – network namespace Manuel Frei | apps:docker:troubleshooting [2025-02-04 17:09] (current) – add xargs data Manuel Frei |
---|
====== Docker Troubleshooting ====== | ====== Docker Troubleshooting ====== |
| |
| ===== Disk Usage (MergeDir) ===== |
| |
| If you run out of disk usage and it's not the data from your volumes, maybe a container is writing data to the overlay file system. This data is placed in the merge directory. Try the following snippets to show the size of the merge directory for all the containers. |
| |
| <code bash> |
| while read -r name mergedir _; do |
| size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)" |
| size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')" |
| echo "${name} ${mergedir} ${size:=0} ${size_hr}" |
| done <<<"$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t |
| </code> |
| |
| The same as a one-liner. |
| <code bash> |
| while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t |
| </code> |
| |
| This is how the output could be displayed. |
| <code - Example> |
| root@server:~# while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t |
| NAME MERGE_DIR SIZE_B SIZE_MB |
| /zabbix-db_data_pgsql-1 /var/lib/docker/overlay2/93e67764c8b6f74e5686a5421c2aa3e79e8a11e81eb5869fbb13ad13ccb46130/merged 0 0.00 |
| /zabbix-zabbix-server-1 /var/lib/docker/overlay2/9e1ecf3b598501ff76c28264c73cd34bf9697fdf7605ad467f8b39a7f8f36fca/merged 63604 62.11 |
| /zabbix-zabbix-web-nginx-pgsql-1 /var/lib/docker/overlay2/4942fc913e4546c7090fba3614c9718607f7b0274abbc50ea6feed252efed58a/merged 179516 175.31 |
| /zabbix-postgres-server-1 /var/lib/docker/overlay2/d7918d5dea06a3df43c847c91368341e56bda720a68cd6a7be6296dcb9a97f27/merged 272720 266.33 |
| /zabbix-zabbix-web-service-1 /var/lib/docker/overlay2/eebdcc8ee3b0b2fc871db6c32a8988a52b4b200996ff28aa015e2bec8f039eb0/merged 754740 737.05 |
| </code> |
| |
===== About Namespaces ===== | ===== About Namespaces ===== |
container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")" | container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")" |
nsenter -n -t "${container_pid}" tcpdump -nn -i eth0 | nsenter -n -t "${container_pid}" tcpdump -nn -i eth0 |
| </code> |
| |
| ===== Logs ===== |
| |
| ==== Add Local Timestamp to Container Logs ==== |
| |
| If a container is logging without a timestamp or with an unknown time zone, you can add a timestamp while watching the logs. |
| <code bash> |
| docker logs mycontainer -n 0 -f | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L' |
| </code> |
| |
| In case you want to catch the stderr logs, too |
| <code bash> |
| docker logs mycontainer -n 0 -f 2>&1 | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L' |
| </code> |
| |
| <code - Example> |
| root@server:~# docker logs mycontainer -n 0 -f 2>&1 | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L' |
| [2025-02-04T16:55:01+01:00] 2025-02-04T15:55:01+0000 INFO main.py:24 send request for url http://server/ping. |
| [2025-02-04T16:55:01+01:00] 2025-02-04T15:55:01+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong |
| [2025-02-04T16:55:03+01:00] 2025-02-04T15:55:03+0000 INFO main.py:24 send request for url http://server/ping. |
| [2025-02-04T16:55:03+01:00] 2025-02-04T15:55:03+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong |
| [2025-02-04T16:55:05+01:00] 2025-02-04T15:55:05+0000 INFO main.py:24 send request for url http://server/ping. |
| [2025-02-04T16:55:05+01:00] 2025-02-04T15:55:05+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong |
</code> | </code> |
| |