Skip to content

Meilisearch complains about incorrect "read-only file system" and exhibits unexpected behaviour #368611

@MatthiasvB

Description

Describe the bug

I'm trying to have meilisearch store its data in a different (encrypted) location. It fails

Steps To Reproduce

Steps to reproduce the behavior:

  1. overwrite config.systemd.services.meilisearch.environment.MEILI_DB_PATH
  2. witness the service failing
Dez 27 19:27:07 m-testserver systemd[1]: Started MeiliSearch daemon.
░░ Subject: A start job for unit meilisearch.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit meilisearch.service has finished successfully.
░░ 
░░ The job identifier is 125438.
Dez 27 19:27:07 m-testserver meilisearch[378580]: 2024-12-27T18:27:07.704954Z ERROR meilisearch: error=Read-only file system (os error 30)
Dez 27 19:27:07 m-testserver meilisearch[378580]: Error: Read-only file system (os error 30)
Dez 27 19:27:07 m-testserver systemd[1]: meilisearch.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit meilisearch.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Dez 27 19:27:07 m-testserver systemd[1]: meilisearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit meilisearch.service has entered the 'failed' state with result 'exit-code'.

Expected behavior

Meilisearch should accept its new storage location

Additional context

There is some more weirdness for which I don't have an explanation. The default storage location is /var/lib/meilisearch. However, on my NixOS, that's actually just a symlink to /var/lib/private/meilisearch. I have looked through the meilisearch nix code and found no indication of why this might be. Could this be a more general mechanism?

While debugging, I deleted this symlink. Upon trying to restart meilisearch, it is recreated, even though this is the unit config of meilisearch now:

❯ sudo systemctl cat meilisearch.service
[sudo] password for user: 
# /etc/systemd/system/meilisearch.service
[Unit]
After=network.target
Description=MeiliSearch daemon

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/xpwb8kynidlxik543bhwjnbj2x2nbcf0-glibc-locales-2.40-36/lib/locale/locale-archive"
Environment="MEILI_DB_PATH=/mnt/data/meilisearch"
Environment="MEILI_DUMP_DIR=/mnt/data/meilisearch/dumps"
Environment="MEILI_ENV=production"
Environment="MEILI_HTTP_ADDR=127.0.0.1:3548"
Environment="MEILI_LOG_LEVEL=INFO"
Environment="MEILI_MAX_INDEX_SIZE=107374182400"
Environment="MEILI_NO_ANALYTICS=true"
Environment="PATH=/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin:/nix/store/w8pnfazxqwmrqmwkb5zrz1bifsd8abxl-findutils-4.10.0/bin:>
Environment="TZDIR=/nix/store/dfsiy184c8z6gf5x6cqfyahxv9g16828-tzdata-2024b/share/zoneinfo"
DynamicUser=true
EnvironmentFile=/run/keys/arcus-meili-env
ExecStart=/nix/store/yvw55vrhicgdwxamzvqvr9k21gdigxpx-meilisearch-1.11.1/bin/meilisearch
StateDirectory=meilisearch

[Install]
WantedBy=multi-user.target

The error is the same whether the specified db-dir already exists or not. I've taken care to migrate the existing files with permissions intact - no change. Though, notably, when the data is there but permissions are incorrect the error changes to "permission denied"

Metadata

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.63, NixOS, 24.11 (Vicuna), 24.11.20241130.62c435d`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - nixpkgs: `/nix/store/ljk3wvwqqzbmz0pys49ii18qmsi5aibc-source`

Notify maintainers

@Br1ght0ne @happysalada


Note for maintainers: Please tag this issue in your PR.


Add a 👍 reaction to issues you find important.

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions