User Tools

Site Tools


apps:docker:troubleshooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
apps:docker:troubleshooting [2025-02-01 19:36] – add disk usage snippets Manuel Freiapps:docker:troubleshooting [2025-02-04 17:09] (current) – add xargs data Manuel Frei
Line 1: Line 1:
 ====== Docker Troubleshooting ====== ====== Docker Troubleshooting ======
  
-===== Disk Usage =====+===== Disk Usage (MergeDir) ===== 
 + 
 +If you run out of disk usage and it's not the data from your volumes, maybe a container is writing data to the overlay file system. This data is placed in the merge directory. Try the following snippets to show the size of the merge directory for all the containers. 
  
 <code bash> <code bash>
 while read -r name mergedir _; do while read -r name mergedir _; do
     size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"     size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"
-    size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.3f\n", s/1024 }')" +    size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')" 
-    echo ${name} ${mergedir} ${size:=0} ${size_hr}+    echo "${name} ${mergedir} ${size:=0} ${size_hr}"
 done <<<"$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t done <<<"$(docker inspect $(docker ps -qa) | jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t
 </code> </code>
Line 13: Line 15:
 The same as a one-liner. The same as a one-liner.
 <code bash> <code bash>
-while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.3f\n", s/1024 }')"; echo ${name} ${mergedir} ${size:=0} ${size_hr}; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t+while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t
 </code> </code>
  
 +This is how the output could be displayed.
 <code - Example> <code - Example>
-root@server:~# while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.3f\n", s/1024 }')"; echo ${name} ${mergedir} ${size:=0} ${size_hr}; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t+root@server:~# while read -r name mergedir _; do size="$([ -d "${mergedir}" ] && du -s "${mergedir}" | cut -f 1)"; size_hr="$(awk -v s="${size:=0}" 'BEGIN { printf "%.2f\n", s/1024 }')"; echo "${name} ${mergedir} ${size:=0} ${size_hr}"; done <<< "$(docker inspect $(docker ps -qa) |  jq -r 'map([.Name, .GraphDriver.Data.MergedDir]) | .[] | "\(.[0])\t\(.[1])"')" | sort -nk3 | awk 'BEGIN{ print "NAME","MERGE_DIR","SIZE_B","SIZE_MB" } { print $0 }' | column -t
 NAME                              MERGE_DIR                                                                                         SIZE_B  SIZE_MB NAME                              MERGE_DIR                                                                                         SIZE_B  SIZE_MB
-/zabbix-db_data_pgsql-1           /var/lib/docker/overlay2/93e67764c8b6f74e5686a5421c2aa3e79e8a11e81eb5869fbb13ad13ccb46130/merged  0       0.000 +/zabbix-db_data_pgsql-1           /var/lib/docker/overlay2/93e67764c8b6f74e5686a5421c2aa3e79e8a11e81eb5869fbb13ad13ccb46130/merged  0       0.00 
-/zabbix-zabbix-server-1           /var/lib/docker/overlay2/9e1ecf3b598501ff76c28264c73cd34bf9697fdf7605ad467f8b39a7f8f36fca/merged  63608   62.117 +/zabbix-zabbix-server-1           /var/lib/docker/overlay2/9e1ecf3b598501ff76c28264c73cd34bf9697fdf7605ad467f8b39a7f8f36fca/merged  63604   62.11 
-/zabbix-zabbix-web-nginx-pgsql-1  /var/lib/docker/overlay2/4942fc913e4546c7090fba3614c9718607f7b0274abbc50ea6feed252efed58a/merged  179516  175.309 +/zabbix-zabbix-web-nginx-pgsql-1  /var/lib/docker/overlay2/4942fc913e4546c7090fba3614c9718607f7b0274abbc50ea6feed252efed58a/merged  179516  175.31 
-/zabbix-postgres-server-1         /var/lib/docker/overlay2/d7918d5dea06a3df43c847c91368341e56bda720a68cd6a7be6296dcb9a97f27/merged  272720  266.328 +/zabbix-postgres-server-1         /var/lib/docker/overlay2/d7918d5dea06a3df43c847c91368341e56bda720a68cd6a7be6296dcb9a97f27/merged  272720  266.33 
-/zabbix-zabbix-web-service-1      /var/lib/docker/overlay2/eebdcc8ee3b0b2fc871db6c32a8988a52b4b200996ff28aa015e2bec8f039eb0/merged  754740  737.051+/zabbix-zabbix-web-service-1      /var/lib/docker/overlay2/eebdcc8ee3b0b2fc871db6c32a8988a52b4b200996ff28aa015e2bec8f039eb0/merged  754740  737.05
 </code> </code>
  
Line 52: Line 54:
 container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")" container_pid="$(docker inspect --format "{{ .State.Pid }}" "${container_id}")"
 nsenter -n -t "${container_pid}" tcpdump -nn -i eth0 nsenter -n -t "${container_pid}" tcpdump -nn -i eth0
 +</code>
 +
 +===== Logs =====
 +
 +==== Add Local Timestamp to Container Logs ====
 +
 +If a container is logging without a timestamp or with an unknown time zone, you can add a timestamp while watching the logs.
 +<code bash>
 +docker logs mycontainer -n 0 -f | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L'
 +</code>
 +
 +In case you want to catch the stderr logs, too
 +<code bash>
 +docker logs mycontainer -n 0 -f 2>&1 | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L'
 +</code>
 +
 +<code - Example>
 +root@server:~# docker logs mycontainer -n 0 -f 2>&1 | xargs -IL date +'[%Y-%m-%dT%H:%M:%S%:z] L'
 +[2025-02-04T16:55:01+01:00] 2025-02-04T15:55:01+0000 INFO main.py:24 send request for url http://server/ping.
 +[2025-02-04T16:55:01+01:00] 2025-02-04T15:55:01+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong
 +[2025-02-04T16:55:03+01:00] 2025-02-04T15:55:03+0000 INFO main.py:24 send request for url http://server/ping.
 +[2025-02-04T16:55:03+01:00] 2025-02-04T15:55:03+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong
 +[2025-02-04T16:55:05+01:00] 2025-02-04T15:55:05+0000 INFO main.py:24 send request for url http://server/ping.
 +[2025-02-04T16:55:05+01:00] 2025-02-04T15:55:05+0000 INFO main.py:29 Received answer with HTTP status code 200 and content: GET: pong
 </code> </code>
  
apps/docker/troubleshooting.1738434969.txt.gz · Last modified: 2025-02-01 19:36 by Manuel Frei