2024-08-16, 12:34 PM
And now with the Log. I copied it from Putty directly with "Copy all to Clipboard". So some lines are cut off.
I fully copied those, that might point to the problem. If there are some lines that I should post completely please tell me.
(After I finished, I remebered that I could have just downloaded the whole file... now its to late... Well I only started about 2 years ago with the whole Linux thing)
Especially this line might be the clue to the perpetrator:
I looked into it and found out, that OMV (extra VM in Proxmox) is restarting at 2:30am and not fully back online until 3:00am.
In the time until 2:30 Adguard as well as my Router is also restarting interfering with my network.
It could be, that my Media folder is unreachable during container setup. So I will change the cron to 3:10 and I will also add "sleep 60" to the end (so that the script stays active during "docker compose up -d").
I think changing the docker compose file to "restart: never" is out of question. As I intend to restart the server sometimes during daytime.
And again: Tomorrow I will know more
I fully copied those, that might point to the problem. If there are some lines that I should post completely please tell me.
(After I finished, I remebered that I could have just downloaded the whole file... now its to late... Well I only started about 2 years ago with the whole Linux thing)
Code:
Aug 16 00:35:33 youngdocker systemd[1]: Stopping Docker Application Container Engine...
░░ Subject: A stop job for unit docker.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit docker.service has begun execution.
░░
░░ The job identifier is 4443.
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.252755077Z" level=info msg="Processing signal 'terminated'"
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.486114817Z" level=info msg="ignoring event" container=b4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.507901907Z" level=info msg="ignoring event" container=d6XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.633672917Z" level=info msg="ignoring event" container=fbXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.634317967Z" level=info msg="ignoring event" container=e5XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.651180199Z" level=info msg="ignoring event" container=ffXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.651743968Z" level=info msg="ignoring event" container=d2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.687667896Z" level=info msg="ignoring event" container=42XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.739226153Z" level=info msg="ignoring event" container=e2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.750027587Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=d2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.750224389Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=d6XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.750298545Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=ffXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.752090347Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=fbXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.768063227Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=b4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.798294628Z" level=info msg="ignoring event" container=51XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.805013223Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=42XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.821763510Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=e5XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.821956706Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=e2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:33 youngdocker dockerd[92310]: time="2024-08-16T00:35:33.843460524Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=51XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.221585511Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:37131" dns-server="udp:10.1.0.1:53" error="read udp 172.25.0.>
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.221624333Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:53776" dns-server="udp:10.1.0.1:53" error="read udp 172.25.0.>
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.221793978Z" level=warning msg="[resolver] connect failed" error="dial udp [XXXXX___IPv6_local_DNS___XXXXX]:53: connect: network is unreachable"
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.221841097Z" level=warning msg="[resolver] connect failed" error="dial udp [XXXXX___IPv6_local_DNS___XXXXX]:53: connect: network is unreachable"
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.400576594Z" level=info msg="ignoring event" container=b6XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:35 youngdocker dockerd[92310]: time="2024-08-16T00:35:35.420955779Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=b6XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.678719671Z" level=info msg="ignoring event" container=01XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.699809753Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=01XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.719311783Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:41680" dns-server="udp:10.1.0.1:53" error="read udp 172.25.0.>
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.719348323Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:60409" dns-server="udp:10.1.0.1:53" error="read udp 172.25.0.>
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.719992456Z" level=warning msg="[resolver] connect failed" error="dial udp [XXXXX___IPv6_local_DNS___XXXXX]:53: connect: network is unreachable"
Aug 16 00:35:36 youngdocker dockerd[92310]: time="2024-08-16T00:35:36.720058582Z" level=warning msg="[resolver] connect failed" error="dial udp [XXXXX___IPv6_local_DNS___XXXXX]:53: connect: network is unreachable"
Aug 16 00:35:37 youngdocker dockerd[92310]: time="2024-08-16T00:35:37.888346938Z" level=info msg="ignoring event" container=41XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:37 youngdocker dockerd[92310]: time="2024-08-16T00:35:37.907650406Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=41XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:40 youngdocker dockerd[92310]: time="2024-08-16T00:35:40.720206362Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:45828" dns-server="udp:8.8.8.8:53" error="read udp 172.25.0.8>
Aug 16 00:35:40 youngdocker dockerd[92310]: time="2024-08-16T00:35:40.720313848Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.25.0.8:39034" dns-server="udp:8.8.8.8:53" error="read udp 172.25.0.8>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.365445413Z" level=info msg="Container failed to exit within 10s of signal 3 - using the force" container=b10a8cb22e84c9fa7afff8c8d7ba8c2ab4033b88acff53276114a9ecd11fb>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.403430494Z" level=info msg="ignoring event" container=b1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.424959877Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=b1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.438649295Z" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=54fe20ba61330530b815b807afcb94b497157451c09f159dc315372d2871>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.455241168Z" level=info msg="Container failed to exit within 10s of signal 28 - using the force" container=684fac9f01cfbfbf98cb275eb993a11aef28fb6ed5be0a09e39ea2f70559>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.496823495Z" level=info msg="ignoring event" container=68XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/task>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.516984338Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=68XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.576681065Z" level=error msg="stream copy error: reading from a closed fifo"
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.577413462Z" level=error msg="stream copy error: reading from a closed fifo"
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.596544043Z" level=info msg="ignoring event" container=54XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.599755276Z" level=warning msg="Health check for container 54XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX error: OCI runtime exec failed: exec failed: cannot exec in a stopped container: unknown"
Aug 16 00:35:43 youngdocker dockerd[92310]: time="2024-08-16T00:35:43.618617383Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=54XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX daemo>
Aug 16 00:35:44 youngdocker dockerd[92310]: time="2024-08-16T00:35:44.139449643Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Aug 16 00:35:44 youngdocker dockerd[92310]: time="2024-08-16T00:35:44.140545155Z" level=info msg="Daemon shutdown complete"
Aug 16 00:35:44 youngdocker systemd[1]: docker.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit docker.service has successfully entered the 'dead' state.
Aug 16 00:35:44 youngdocker systemd[1]: Stopped Docker Application Container Engine.
░░ Subject: A stop job for unit docker.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit docker.service has finished.
░░
░░ The job identifier is 4443 and the job result is done.
Aug 16 00:35:44 youngdocker systemd[1]: docker.service: Consumed 5min 58.032s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit docker.service completed and consumed the indicated resources.
-- Boot a130f49d66544bdd8708f30066cb6c79 --
Aug 16 00:37:14 youngdocker systemd[1]: Starting Docker Application Container Engine...
░░ Subject: A start job for unit docker.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has begun execution.
░░
░░ The job identifier is 155.
Aug 16 00:37:15 youngdocker dockerd[937]: time="2024-08-16T00:37:15.363140146Z" level=info msg="Starting up"
Aug 16 00:37:15 youngdocker dockerd[937]: time="2024-08-16T00:37:15.794664221Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Aug 16 00:37:16 youngdocker dockerd[937]: time="2024-08-16T00:37:16.248067416Z" level=info msg="Loading containers: start."
Aug 16 00:37:17 youngdocker dockerd[937]: time="2024-08-16T00:37:17.600813350Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 16 00:37:18 youngdocker dockerd[937]: time="2024-08-16T00:37:18.520363980Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:18 youngdocker dockerd[937]: time="2024-08-16T00:37:18.663961580Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:18 youngdocker dockerd[937]: time="2024-08-16T00:37:18.716062563Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:18 youngdocker dockerd[937]: time="2024-08-16T00:37:18.898107588Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:19 youngdocker dockerd[937]: time="2024-08-16T00:37:19.026728039Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:19 youngdocker dockerd[937]: time="2024-08-16T00:37:19.164193172Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:19 youngdocker dockerd[937]: time="2024-08-16T00:37:19.219447498Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:19 youngdocker dockerd[937]: time="2024-08-16T00:37:19.339838600Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Aug 16 00:37:22 youngdocker dockerd[937]: time="2024-08-16T00:37:22.358178676Z" level=error msg="stream copy error: reading from a closed fifo"
Aug 16 00:37:22 youngdocker dockerd[937]: time="2024-08-16T00:37:22.364623813Z" level=error msg="stream copy error: reading from a closed fifo"
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.016881446Z" level=error msg="failed to start container" container=54XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/mnt/Medien\" to rootfs at \"/media\": mount /mnt/Medien:/media (via /proc/self/fd/6), flags: 0x5000: no such device: unknown"
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.016983996Z" level=info msg="Loading containers: done."
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.093654436Z" level=info msg="Docker daemon" commit=f9522e5 containerd-snapshotter=false storage-driver=overlay2 version=27.1.2
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.096379873Z" level=info msg="Daemon has completed initialization"
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.270296268Z" level=info msg="API listen on /run/docker.sock"
Aug 16 00:37:24 youngdocker systemd[1]: Started Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has finished successfully.
░░
░░ The job identifier is 155.Especially this line might be the clue to the perpetrator:
Code:
Aug 16 00:37:24 youngdocker dockerd[937]: time="2024-08-16T00:37:24.016881446Z" level=error msg="failed to start container" container=54XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX error="failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/mnt/Medien\" to rootfs at \"/media\": mount /mnt/Medien:/media (via /proc/self/fd/6), flags: 0x5000: no such device: unknown"
.I looked into it and found out, that OMV (extra VM in Proxmox) is restarting at 2:30am and not fully back online until 3:00am.
In the time until 2:30 Adguard as well as my Router is also restarting interfering with my network.
It could be, that my Media folder is unreachable during container setup. So I will change the cron to 3:10 and I will also add "sleep 60" to the end (so that the script stays active during "docker compose up -d").
I think changing the docker compose file to "restart: never" is out of question. As I intend to restart the server sometimes during daytime.
And again: Tomorrow I will know more