Etcd - failed to open database ... mmap too large

[venkataramanam]

We encountered a strange Etcd issue after restoring the Etcd pvc data from a backup on a Kubernetes cluster. The storage provider is NetApp. The etcd members fail to open the database with the error "mmap too large). Please see the error details below...

{
    "level": "panic",
    "ts": "2024-05-14T15:43:47.578595Z",
    "caller": "backend/backend.go:189",
    "msg": "failed to open database",
    "path": "/data/etcd/xyzxyz/member/snap/db",
    "error": "mmap too large",
    "stacktrace": "go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\tgo.etcd.io/etcd/server/v3/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\tgo.etcd.io/etcd/server/v3/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/etcdserver.newBackend\n\tgo.etcd.io/etcd/server/v3/etcdserver/backend.go:55\ngo.etcd.io/etcd/server/v3/etcdserver.openBackend.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/backend.go:76"
}

After a bit of investigation, a workaround was found. The workaround was to slightly increase the mounted PVC size and restart the Etcd cluster. After that Etcd comes up fine. The act of storage provider expanding seems to make the boltdb happy and not report the mmap too large. Not sure if this is an issue on the storage provider or it is something that can be addressed within Etcd. If you have seen any such before, please let us know what you did to overcome the error. This does not happen with any other storage provider that we have used till now and only happens with NetApp. If there is anything on the mount options that can be customized to overcome, that would be great.

[ahrtr]

• What's the etcd version?
• What's the size of the etcd's db file (locates at ${data_dir}/member/snap/db) and what's the PVC size when you saw the "mmap too large" error?
• What's the new PVC size to workaround the issue?

[venkataramanam]

What's the etcd version?

Etcd 3.5.13

What's the size of the etcd's db file (locates at ${data_dir}/member/snap/db) and what's the PVC size when you saw the "mmap too large" error?

It was around 18.5 Mb
-rw-rw----. 1 1001160000 1001160000 18481152 May 30 06:55 db

What's the new PVC size to workaround the issue?

The original PVC size was 10Gi and just bumping up the size by an additional small amount of 1Mi did the trick. Like the new size as 10240Mi (Gi, Mi as in Gigibytes or Mebibytes). Something might have changed on the file system when the PVC expanded that caused the Etcd to start without failing to open the database.

I was thinking if the file system was incorrectly reporting the file size being large, which in turn is making the boltdb fail with mmap too large, but the file listing did not seem to report large size. This is on amd64 architecture.

[ahrtr]

I was thinking if the file system was incorrectly reporting the file size being large

Exactly, I was thinking the same, and it's the only reason I can think of for bbolt to return an error "mmap too large".

It was around 18.5 Mb
-rw-rw----. 1 1001160000 1001160000 18481152 May 30 06:55 db

If you checked the db size right after you saw the issue, it might be a huge size.