[Help]: Restore data from old installation

[Gilgahmesh]

Controller Version

5.15.6.7

Describe Your Issue or Question

Hello,

I am using a Synology NAS with docker.
I had to uninstall the docker image and container of the omada controller as well as Container Manager.
Beforehand I made a copy of the docker folder containing all omada controller data.
Unfortunately, I didn't know I have to stop the controller before I do the backup so the controller was running while I've copied the controller data.
After the reinstall of Container Manager I've redownloaded the image and reinstalled the container for the omada controller. After that I've copied the old data of the docker folder back to the new one.
Unfortunately, after that I wasn't able to access the controller. The image is running but when I try to open the controller with its ip it only says "ERR_CONNECTION_REFUSED" "Website is not availabe" (in German it's "Die Website ist nicht erreichbar").
Does anyone has an idea if there is any way I could restore the data of my old installation?
Or did I screw up and I have to do all the settings again?

Thanks

Expected Behavior

Use settings of old installation

Steps to Reproduce

1. Backup folder "docker" while container is running
2. Copy backed up data back after reinstallation

How You're Launching the Container

With Synology NAS and the Container Manager

Container Logs

.

MongoDB Logs

No response

Additional Context

No response

[mbentley]

Can you provide both the container logs and MongoDB logs when you try to run the container with the restored data? That will tell us why it is not starting.

[Gilgahmesh]

Yes, sure - I hope these are the correct ones.
I made a fresh install to clean the logs from old data.
server.log
mongod.log

This is the command how I install the image and container:
docker run -d --name=omada-controller
-e PUID=
-e PGID=
-e TZ=Europe/Berlin
-e MANAGE_HTTP_PORT=8088
-e MANAGE_HTTPS_PORT=8043
-e PORTAL_HTTP_PORT=8088
-e PORTAL_HTTPS_PORT=8043
-e SHOW_SERVER_LOGS=true
-e SHOW_MONGODB_LOGS=false
-e SSL_CERT_NAME="tls.crt"
-e SSL_KEY_NAME="tls.key"
-v /volume1/docker/omada/data:/opt/tplink/EAPController/data
-v /volume1/docker/omada/logs:/opt/tplink/EAPController/logs
-v /volume1/docker/omada/work:/opt/tplink/EAPController/work
--net host
--restart always
mbentley/omada-controller

[mbentley]

OK, so you are definitely running into a corrupt MongoDB. The only thing to really try would be to stop the controller container and then manually try to run a repair command:

docker run -it --rm \
  -u 508:508 \
 -v /volume1/docker/omada/data:/opt/tplink/EAPController/data  \
  --entrypoint /opt/tplink/EAPController/bin/mongod \
  mbentley/omada-controller \
    --repair \
    --dbpath /opt/tplink/EAPController/data/db \
    --bind_ip 127.0.0.1 \
    --port 27217

Once that runs, try to start your controller in a container again.

If that doesn't work, the data can't be repaired. You'll need to reinstall from a clean installation unless you previously enabled autobackups.

[Gilgahmesh]

Do I need to let the command finish before I restart the container or should I start the container while the command is running?
This is what I get:
"2024-12-21T07:50:17.774+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27217 dbpath=/opt/tplink/EAPController/data/db 64-bit host=f9279af22335
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] db version v3.6.8
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] git version: 8e540c0b6db 93ce994cc548f000900bdc740f80a
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] allocator: tcmalloc
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] modules: none
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] build environment:
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] distarch: x86_64
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] target_arch: x86_64
2024-12-21T07:50:17.795+0000 I CONTROL [initandlisten] options: { net: { bindIp : "127.0.0.1", port: 27217 }, repair: true, storage: { dbPath: "/opt/tplink/EAPC ontroller/data/db" } }
2024-12-21T07:50:17.811+0000 I STORAGE [initandlisten] exception in initAndList en: IllegalOperation: Attempted to create a lock file on a read-only directory: /opt/tplink/EAPController/data/db, terminating
2024-12-21T07:50:17.811+0000 I CONTROL [initandlisten] now exiting
2024-12-21T07:50:17.811+0000 I CONTROL [initandlisten] shutting down with code: 100"
Unfortunately, I still can't open the controller.

[Gilgahmesh]

That's the output I get for ls:
total 8
drwxr-xr-x+ 1 1026 100 126 Dec 17 22:37 .
drwxr-xr-x+ 1 1026 100 24 Sep 29 20:11 ..
drwxr-xr-x+ 1 1026 100 156558 Dec 21 12:47 db
drwxr-xr-x+ 1 1026 100 128 Nov 5 14:44 device-firmware
drwxr-xr-x 1 1026 100 62 Sep 27 09:28 html
drwxr-xr-x+ 1 1026 100 38 Sep 29 20:35 keystore
-rwxr-xr-x+ 1 1026 100 9 Dec 21 12:46 LAST_RAN_OMADA_VER.txt
-rw-r--r-- 1 1026 100 4 Dec 21 12:47 mongo.pid
drwxr-xr-x+ 1 1026 100 0 Sep 29 20:35 pdf

[mbentley]

OK, so what you're seeing there is that the directory, the subdirectories, and files are owned by user ID 1026 and group ID 100. In the container, these are controlled by the PUID and PGID environment variables and the container defaults to 508:508. If you didn't specify those variables before when you were running your container, you can fix the ownership with a chown -R 508:508 /volume1/docker/omada/data/ and then try to re-run the repair docker command again.

If you did previously use 1026 and 100, you will need to modify the repair docker command by changing -u 508:508 to -u 1026:100.

[Gilgahmesh]

Ah, I see.
Yes, I did use 1026 and 100 when installing the container. That's the PUID and PGID I got from my synology when using the id command.
So I changed the repair command as you suggested and now I've got a much longer output:
`2024-12-21T12:20:15.628+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27217 dbpath=/opt/tplink/EAPController/data/db 64-bit host=67ca21035652
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] db version v3.6.8
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] allocator: tcmalloc
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] modules: none
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] build environment:
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] distarch: x86_64
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] target_arch: x86_64
2024-12-21T12:20:15.645+0000 I CONTROL [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27217 }, repair: true, storage: { dbPath: "/opt/tplink/EAPController/data/db" } }
2024-12-21T12:20:15.672+0000 I - [initandlisten] Detected data files in /opt/tplink/EAPController/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-12-21T12:20:15.689+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2024-12-21T12:20:15.689+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=5399M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-12-21T12:20:17.232+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:232181][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 557056: block header checksum of 1734310245 doesn't match expected checksum of 2686118591
2024-12-21T12:20:17.232+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:232487][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: {557056, 4096, 2686118591}: (chunk 1 of 4): 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 99 66 69 6c 65 3a 69 6e 64 65 78 2d 33 30 30 2d 38 35 30 31 33 30 32 31 39 30 38 37 33 37 32 32 34 38 36 2e 77 74 00 80 c3 30 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 5f 69 64 5f 22 2c 20 22 6e 73 22 20 3a 20 22 6f 6d 61 64 61 2e 73 65 72 76 69 63 65 74 79 70 65 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 37 32 37 36 33 35 30 35 32 2c 73 69 7a 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 30 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 30 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 33 30 34 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70
2024-12-21T12:20:17.232+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:232749][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: {557056, 4096, 2686118591}: (chunk 2 of 4): 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 99 66 69 6c 65 3a 69 6e 64 65 78 2d 33 30 31 2d 38 35 30 31 33 30 32 31 39 30 38 37 33 37 32 32 34 38 36 2e 77 74 00 80 c3 5c 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 73 69 74 65 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 73 69 74 65 5f 69 64 22 2c 20 22 6e 73 22 20 3a 20 22 6f 6d 61 64 61 2e 73 65 72 76 69 63 65 74 79 70 65 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 30 31 38 30 38 30 38 30 38 30 38 30 38 30 38 30 38 30 38 30 38 30 38 30 38 30 63 66 63 30 38 30 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 37 32 37 36 33 35 30 31 35 2c 73 69 7a 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 30 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 38 34 31 32 31 36 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 33 30 35 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b
2024-12-21T12:20:17.232+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:232976][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: {557056, 4096, 2686118591}: (chunk 3 of 4): 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 9d 66 69 6c 65 3a 69 6e 64 65 78 2d 33 30 32 2d 2d 32 34 35 30 33 35 36 30 38 30 31 36 37 30 37 37 35 31 36 2e 77 74 00 80 c3 4f 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 5f 69 64 5f 22 2c 20 22 6e 73 22 20 3a 20 22 6f 6d 61 64 61 5f 64 61 74 61 2e 64 70 69 63 6c 69 65 6e 74 74 72 61 66 66 69 63 73 74 61 74 38 64 61 69 6c 79 5f 32 30 32 34 31 32 32 30 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 37 33 34 35 36 38 32 32 39 2c 73 69 7a 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 30 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 30 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 31 30 39 39 39 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70
2024-12-21T12:20:17.233+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:233197][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: {557056, 4096, 2686118591}: (chunk 4 of 4): 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 9d 66 69 6c 65 3a 69 6e 64 65 78 2d 33 30 33 2d 2d 32 34 35 30 33 35 36 30 38 30 31 36 37 30 37 37 35 31 36 2e 77 74 00 80 c3 71 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 6f 6d 61 64 61 63 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 6f 6d 61 64 61 63 5f 69 64 5f 31 22 2c 20 22 6e 73 22 20 3a 20 22 6f 6d 61 64 61 5f 64 61 74 61 2e 64 70 69 63 6c 69 65 6e 74 74 72 61 66 66 69 63 73 74 61 74 38 64 61 69 6c 79 5f 32 30 32 34 31 32 32 30 22 2c 20 22 62 61 63 6b 67 72 6f 75 6e 64 22 20 3a 20 74 72 75 65 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 37 33 34 35 36 38 32 32 39 2c 73 69 7a 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 30 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 30 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 31 31 30 30 30 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f
2024-12-21T12:20:17.234+0000 E STORAGE [initandlisten] WiredTiger error (0) [1734783617:234069][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2024-12-21T12:20:17.234+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1734783617:234101][1:0x7f36ab099ac0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2024-12-21T12:20:17.234+0000 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2024-12-21T12:20:17.234+0000 F - [initandlisten]

***aborting after fassert() failure

2024-12-21T12:20:17.259+0000 F - [initandlisten] Got signal: 6 (Aborted).

0x55ba11dea5fe 0x55ba11de9eef 0x55ba11de9f8c 0x7f36ab35f420 0x7f36ab19c00b 0x7f36ab17b859 0x55ba10555fdd 0x55ba102fdace 0x55ba10692429 0x55ba103080a8 0x55ba10308408 0x55ba103141d6 0x55ba1075b644 0x55ba1075bb8c 0x55ba10758776 0x55ba1073ee68 0x55ba106b71ab 0x55ba10638355 0x55ba10690cfe 0x55ba106912f0 0x55ba106914f4 0x55ba107015b6 0x55ba1068ca3f 0x55ba106544e2 0x55ba106545cf 0x55ba10636ff7 0x55ba1060a438 0x55ba105ef8f9 0x55ba107d3386 0x55ba105c9659 0x55ba105ccb83 0x55ba1055a99d 0x7f36ab17d083 0x55ba105b70ee
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55BA0FADF000","o":"230B5FE","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BA0FADF000","o":"230AEEF"},{"b":"55BA0FADF000","o":"230AF8C"},{"b":"7F36AB34B000","o":"14420"},{"b":"7F36AB159000","o":"4300B","s":"gsignal"},{"b":"7F36AB159000","o":"22859","s":"abort"},{"b":"55BA0FADF000","o":"A76FDD","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55BA0FADF000","o":"81EACE"},{"b":"55BA0FADF000","o":"BB3429"},{"b":"55BA0FADF000","o":"8290A8","s":"__wt_err"},{"b":"55BA0FADF000","o":"829408","s":"__wt_panic"},{"b":"55BA0FADF000","o":"8351D6"},{"b":"55BA0FADF000","o":"C7C644","s":"__wt_block_extlist_read"},{"b":"55BA0FADF000","o":"C7CB8C","s":"__wt_block_extlist_read_avail"},{"b":"55BA0FADF000","o":"C79776","s":"__wt_block_checkpoint_load"},{"b":"55BA0FADF000","o":"C5FE68"},{"b":"55BA0FADF000","o":"BD81AB","s":"__wt_btree_open"},{"b":"55BA0FADF000","o":"B59355","s":"__wt_conn_dhandle_open"},{"b":"55BA0FADF000","o":"BB1CFE","s":"__wt_session_get_dhandle"},{"b":"55BA0FADF000","o":"BB22F0","s":"__wt_session_get_dhandle"},{"b":"55BA0FADF000","o":"BB24F4","s":"__wt_session_get_btree_ckpt"},{"b":"55BA0FADF000","o":"C225B6","s":"__wt_curfile_open"},{"b":"55BA0FADF000","o":"BADA3F"},{"b":"55BA0FADF000","o":"B754E2","s":"__wt_metadata_cursor_open"},{"b":"55BA0FADF000","o":"B755CF","s":"__wt_metadata_cursor"},{"b":"55BA0FADF000","o":"B57FF7","s":"wiredtiger_open"},{"b":"55BA0FADF000","o":"B2B438","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55BA0FADF000","o":"B108F9"},{"b":"55BA0FADF000","o":"CF4386","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55BA0FADF000","o":"AEA659"},{"b":"55BA0FADF000","o":"AEDB83","s":"ZN5mongo11mongoDbMainEiPPcS1"},{"b":"55BA0FADF000","o":"A7B99D","s":"main"},{"b":"7F36AB159000","o":"24083","s":"__libc_start_main"},{"b":"55BA0FADF000","o":"AD80EE","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.6.8", "gitVersion" : "8e540c0b6db93ce994cc548f000900bdc740f80a", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.302+", "version" : "#72806 SMP Thu Sep 5 13:42:01 CST 2024", "machine" : "x86_64" }, "somap" : [ { "b" : "55BA0FADF000", "elfType" : 3, "buildId" : "28879BCB86C4B9AF7CEC8E6482F331A095E338D0" }, { "b" : "7FFFA0BE7000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "DCCBDF10DCF3622A6162B4877DDABA2072EA7AF0" }, { "b" : "7F36ABD7B000", "path" : "/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "F129B7D409F6F63A822673139BE40F369F3E4B13" }, { "b" : "7F36ABD5F000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "101D8983A254F1B26637242E10B80ECAE42EFF2D" }, { "b" : "7F36ABD54000", "path" : "/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "74CAFD320059B4D17CF89FD6379A5105FF55840F" }, { "b" : "7F36ABCF9000", "path" : "/lib/x86_64-linux-gnu/libyaml-cpp.so.0.6", "elfType" : 3, "buildId" : "B9EC1711A8AAD2FE980B735F13E186BEFE9FA4DB" }, { "b" : "7F36ABCEE000", "path" : "/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "9199246C7429052FBE1F97DA6FE0B0CFBED2DE3F" }, { "b" : "7F36ABC5F000", "path" : "/lib/x86_64-linux-gnu/libboost_program_options.so.1.71.0", "elfType" : 3, "buildId" : "CB2EB21D65AF49F187F384ABED5D46D66E998195" }, { "b" : "7F36ABC3F000", "path" : "/lib/x86_64-linux-gnu/libboost_filesystem.so.1.71.0", "elfType" : 3, "buildId" : "956195B5ABA44A072A6242BF2399D9EB0B6163E7" }, { "b" : "7F36ABA48000", "path" : "/lib/x86_64-linux-gnu/libtcmalloc.so.4", "elfType" : 3, "buildId" : "BC036BA2C0C0B4976A51184F1E9F712CCB8611C7" }, { "b" : "7F36ABA2C000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "08F526F3424A323F041D27D040659E7FD07A8BE5" }, { "b" : "7F36AB755000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "FF0FF411AA0EB5DE40716E6CB22752F4687818FD" }, { "b" : "7F36AB6C2000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "AD5124AE9AE4CF0FAC48A89F0458C56EDE195C11" }, { "b" : "7F36AB6BC000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25372F43DBCC661AA02020D0365C948E89F6E612" }, { "b" : "7F36AB4D8000", "path" : "/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "FCD5414020056F1D95855868E00F3A7D479630FD" }, { "b" : "7F36AB389000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "8D2573EFF281739F0E2B0EB710C860CE0B7261CF" }, { "b" : "7F36AB36E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8296A8EF14673927602C39FF755ABFE97732DC6F" }, { "b" : "7F36AB34B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9A65BB469E45A1C6FBCFFAE5B82A2FD7A69EB479" }, { "b" : "7F36AB159000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "0702430AEF5FA3DDA43986563E9FFCC47EFBD75E" }, { "b" : "7F36AB0E6000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "CEF782225A118AF34A28083235A6DA2AA59B7E82" }, { "b" : "7F36AB0C7000", "path" : "/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "08DCE9B46ABF8D1E0E1DEB17E94F8EA84FC449B9" }, { "b" : "7F36ABDD3000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DB0420F708B806CF03260AADB916C330049580B7" }, { "b" : "7F36AB09E000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "0E13B0F27B3BC79A2828C11199CE4336A348AF37" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x3E) [0x55ba11dea5fe]
mongod(+0x230AEEF) [0x55ba11de9eef]
mongod(+0x230AF8C) [0x55ba11de9f8c]
libpthread.so.0(+0x14420) [0x7f36ab35f420]
libc.so.6(gsignal+0xCB) [0x7f36ab19c00b]
libc.so.6(abort+0x12B) [0x7f36ab17b859]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55ba10555fdd]
mongod(+0x81EACE) [0x55ba102fdace]
mongod(+0xBB3429) [0x55ba10692429]
mongod(__wt_err+0xA5) [0x55ba103080a8]
mongod(__wt_panic+0x38) [0x55ba10308408]
mongod(+0x8351D6) [0x55ba103141d6]
mongod(__wt_block_extlist_read+0x94) [0x55ba1075b644]
mongod(__wt_block_extlist_read_avail+0x2C) [0x55ba1075bb8c]
mongod(__wt_block_checkpoint_load+0x226) [0x55ba10758776]
mongod(+0xC5FE68) [0x55ba1073ee68]
mongod(__wt_btree_open+0xD4B) [0x55ba106b71ab]
mongod(__wt_conn_dhandle_open+0x295) [0x55ba10638355]
mongod(__wt_session_get_dhandle+0xCE) [0x55ba10690cfe]
mongod(__wt_session_get_dhandle+0x6C0) [0x55ba106912f0]
mongod(__wt_session_get_btree_ckpt+0x154) [0x55ba106914f4]
mongod(__wt_curfile_open+0x56) [0x55ba107015b6]
mongod(+0xBADA3F) [0x55ba1068ca3f]
mongod(__wt_metadata_cursor_open+0x72) [0x55ba106544e2]
mongod(__wt_metadata_cursor+0x4F) [0x55ba106545cf]
mongod(wiredtiger_open+0x2027) [0x55ba10636ff7]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xCE8) [0x55ba1060a438]
mongod(+0xB108F9) [0x55ba105ef8f9]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E6) [0x55ba107d3386]
mongod(+0xAEA659) [0x55ba105c9659]
mongod(ZN5mongo11mongoDbMainEiPPcS1+0x613) [0x55ba105ccb83]
mongod(main+0xD) [0x55ba1055a99d]
libc.so.6(__libc_start_main+0xF3) [0x7f36ab17d083]
mongod(_start+0x2E) [0x55ba105b70ee]
----- END BACKTRACE -----
`
Connection to the controller is still not working, unfortunately.
So I guess I have to do a fresh install right?

[mbentley]

Yeah, unfortunately. The db isn’t recoverable. Make sure to setup auto backups in the controller software to help in future scenarios.

[Gilgahmesh]

I definitely will - thanks a lot for your help!