MongoDB updated this morning to 7.0.1, now wont restart

Question

We have a pod running a MongoDB instance. It was automatically upgraded this morning to 7.0.1 and now wont restart citing;

initandlisten","msg":"Wrong mongod version","attr":{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCo │
│ mpatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"5.0\" }. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value '5. │
│ 0'; expected '6.0' or '6.3' or '7.0'

Where would I update this Compatibility version? The pod isn’t running at this point.

Or can I fix the version of MongoDB on the pod to a prior version.

Bobby · Accepted Answer

Hi there,

Based on the output that you’ve shared it seems that MongoDB was upgraded to a newer version without the featureCompatibilityVersion being upgraded in stages.

When upgrading MongoDB across multiple major versions, it’s recommended to upgrade one major version at a time and update the featureCompatibilityVersion in between.

Here’s what you can do to recover:

Always make sure to backup your data before performing any upgrade operations.

1. Downgrade MongoDB:

First, downgrade your MongoDB to the previous stable version you were using (e.g., 5.0).

As you mentioned that it is a MongoDB pod, I would assume that you are using Kubernetes, you can change the image version in your MongoDB deployment configuration.

For instance:

spec:
  containers:
  - name: mongodb
    image: mongo:5.0

After making the change, apply the updated configuration:

kubectl apply -f your-deployment-file.yaml

Once the pod is running with the downgraded version, verify that you can connect to your MongoDB instance.

2. Set the featureCompatibilityVersion:

Connect to your MongoDB instance and set the featureCompatibilityVersion:

mongo

db.adminCommand({ setFeatureCompatibilityVersion: "5.0" });

3. Upgrade Step-by-step:

Now, when you’re ready to upgrade:

Upgrade to MongoDB 6.0 and update featureCompatibilityVersion to “6.0”.
Then, if needed, upgrade to MongoDB 6.3 and update featureCompatibilityVersion to “6.3”.
Finally, upgrade to MongoDB 7.0 and set featureCompatibilityVersion to “7.0”.

Always make sure to backup your data before performing any upgrade operations.

Preventing Automatic Upgrades:

To prevent unexpected upgrades in the future, always pin your MongoDB version in your Kubernetes configurations (or whatever method you use to deploy). For instance, instead of using mongo:latest as your image, use mongo:5.0 or whatever specific version you intend to use.

Let me know how it goes!

Best,

Bobby

adaba956bf124818aa9c374fe0d796 · Answer

“First, downgrade your MongoDB to the previous stable version you were using (e.g., 5.0).”

I’ve been using MongoDB in latest versions all the time. The latest update from DO broke our staging machines and I don’t know what version I should downgrade to.

I’ve tried all various older version, but none worked, meaning the K8s pod always ran into CrashLoopBackOff with the error “Wrong mongod version” … "UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion … Invalid feature compatibility version value, expected ‘5.0’ or ‘5.3’ or '6.0. … If the current featureCompatibilityVersion is below 5.0, see the documentation

I have no idea how to fix this and when an update from DO is rolled out to our prod machines they will break, as well.

For the prod deployments I tried to pin the version via image: mongo:6.0 but that lead to the same issues as with the staging machines that use latest, so I had to revert to use the latest mongo latest.

Any suggestions what to do?

Kind regards Marcel

This is one of our K8s yaml files for deployment:

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: signals-mongo-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: signals-mongo
    serviceName: 'signals-mongo'
    template:
        metadata:
            labels:
                app: signals-mongo
        spec:
            containers:
                - name: signals-mongo
                  image: mongo
                  imagePullPolicy: Always
                  volumeMounts:
                      - name: signals-mongo-db
                        mountPath: /data/db
                        subPath: data
    volumeClaimTemplates:
        - metadata:
              name: signals-mongo-db
          spec:
              accessModes:
                  - ReadWriteOnce
              resources:
                  requests:
                      storage: 500Mi
              storageClassName: do-block-storage-retain