MongoDB Replication

If you open up your terminal and launch a mongod instance,without any options, you are using a standalone server. There is one copy of your data. It's an quick and easy way to get started but it's extremely dangerous in a production environment.

A server crash could make your data unavailable. Hardware issues could force you to move your data. Replication keeps several identical copies of your data on multiple servers. If something happens to one of your servers, you have another one ready to take its place and keep your application running and your data safe.

In MongoDB, replication is achieved by creating a replica set. A replica set is composed of multiple servers. One of them is defined as the primary. This is the server taking client requests. The other servers are called secondaries. They have a copy of your primary's data. Your primary goes down, one of the secondaries becomes the primary, and your data is still available.

Setting up a replica set

First, let's create three data directories for each node. Open up a terminal window and type:

mkdir -p ~/data/rs{1,2,3}

Or this on Windows:

md c:\data\rs1 c:\data\rs2 c:\data\rs3

Next, run the following commands in three different terminal windows:

mongod --replSet introRep --dbpath ~/data/rs1 --port 27017 
mongod --replSet introRep --dbpath ~/data/rs2 --port 27018 
mongod --replSet introRep --dbpath ~/data/rs3 --port 27019 

Note: Replace ~/data/rs1 with c:\data\rs1 on Windows. Same for the other two.

Let's decompose these commands:

We create three different mongod instances.

  • --replSet

We are saying to our mongod instance that it is part of a replica set. Each host in the same replica set must have the same set name. Here, I chose introRep

  • --dbpath

Defines where this mongod instance will store its data. We created those directories earlier. Each set is assigned a different directory.

  • --port

Self explanatory, the sets can't run on the same port. 27017 is the default port for mongod. I use 27018 and 27019 for the other two.

Great, now you have three separate mongod processes running.

Configuration

At this point, our mongod processes do not know about each other. We need to create a configuration with the list of each of the members of our replica set and inform on of our mongod processes.

Open up a new terminal window and connect to one of the mongod processes by specifying a port:

mongo --port 27017

Next, we'll create a configuration document. The document will have two keys:

  • _id : Name of the replica set. For our example, introRep

  • members: Array of documents. Each document has an _id key and a host key. It represents every set in our replica set.

Here is how it will look:

config = {
... _id: "introRep",
... members: [
... {_id: 0, host: "localhost:27017"},
... {_id: 1, host: "localhost:27018"},
... {_id: 2, host: "localhost:27019"}
... ]}
{

Store this in a variable and pass it to the rs.initiate() method to initiate a replica set:

rs.initiate(config)

Great, localhost:27017 will notify the other members of the new configuration. After that, they will elect a primary.

Replica set status

Your replica set is now fully functional. If you launch all three sets with mongo and the appropriate --port flag, you will see one prefixed with introRep>PRIMARY>, the other two will have introRep>SECONDARY> as prefix.

To have more information about your replica set, you can use the rs.status() helper:

introRep:PRIMARY> rs.status()
{
    "set" : "introRep",
    "date" : ISODate("2017-08-15T15:37:14.734Z"),
    "myState" : 1,
    "term" : NumberLong(1),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "optimes" : {
        "lastCommittedOpTime" : {
            "ts" : Timestamp(1502811433, 1),
            "t" : NumberLong(1)
        },
        "appliedOpTime" : {
            "ts" : Timestamp(1502811433, 1),
            "t" : NumberLong(1)
        },
        "durableOpTime" : {
            "ts" : Timestamp(1502811433, 1),
            "t" : NumberLong(1)
        }
    },
    "members" : [
        {
            "_id" : 0,
            "name" : "localhost:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 949,
            "optime" : {
                "ts" : Timestamp(1502811433, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2017-08-15T15:37:13Z"),
            "electionTime" : Timestamp(1502810832, 1),
            "electionDate" : ISODate("2017-08-15T15:27:12Z"),
            "configVersion" : 1,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "localhost:27018",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 613,
            "optime" : {
                "ts" : Timestamp(1502811433, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1502811433, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2017-08-15T15:37:13Z"),
            "optimeDurableDate" : ISODate("2017-08-15T15:37:13Z"),
            "lastHeartbeat" : ISODate("2017-08-15T15:37:13.925Z"),
            "lastHeartbeatRecv" : ISODate("2017-08-15T15:37:14.472Z"),
            "pingMs" : NumberLong(0),
            "syncingTo" : "localhost:27017",
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "localhost:27019",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 613,
            "optime" : {
                "ts" : Timestamp(1502811433, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1502811433, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2017-08-15T15:37:13Z"),
            "optimeDurableDate" : ISODate("2017-08-15T15:37:13Z"),
            "lastHeartbeat" : ISODate("2017-08-15T15:37:13.925Z"),
            "lastHeartbeatRecv" : ISODate("2017-08-15T15:37:14.559Z"),
            "pingMs" : NumberLong(0),
            "syncingTo" : "localhost:27018",
            "configVersion" : 1
        }
    ],
    "ok" : 1
}

If you look in the members array, you will see our three sets listed. On my machine, localhost:27017 has been elected primary. We can see this with the stateStr set to PRIMARY. The other two are set to SECONDARY.

Congratulations! Your successfully set up a replica set with MongoDB!!

 

Part2:

We have our replica set configured. We have each node running with the following commands:

mongod --replSet introRep --dbpath ~/data/rs1 --port 27017 
mongod --replSet introRep --dbpath ~/data/rs2 --port 27018 
mongod --replSet introRep --dbpath ~/data/rs3 --port 27019 

You can connect to each instance with mongo --port portNumber

For now, you'll need to connect to your primary node. If you have followed the previous article, you should see a introRep:PRIMARY> at the beginning of each line in your shell. In my case, the primary is on port 27017, but it could be a different one for you.

Playing around

Let's write some documents to our primary. We will write 10000 simple documents, like so:

introRep:PRIMARY> use test
switched to db test

introRep:PRIMARY>  for( i = 0; i < 10000; i++){ db.example.insertOne({count: i})}
{
    "acknowledged" : true,
    "insertedId" : ObjectId("599994478d005c672de000ed")
}
introRep:PRIMARY> db.example.count()
10000

Great, we now have our primary populated. The concept of replication means that your data is replicated identically in your secondaries. Let's connect to one of our secondaries. First, let's run isMaster(). This command acts as a condensed rs.status(). You will see informations such as which node is your primary and on which node you're connected:

introRep:PRIMARY> db.isMaster()
{
    "hosts" : [
        "localhost:27017",
        "localhost:27018",
        "localhost:27019"
    ],
    "setName" : "introRep",
    "setVersion" : 1,
    "ismaster" : true,
    "secondary" : false,
    "primary" : "localhost:27017",
    "me" : "localhost:27017",
    "electionId" : ObjectId("7fffffff0000000000000002"),
    "lastWrite" : {
        "opTime" : {
            "ts" : Timestamp(1503237420, 1),
            "t" : NumberLong(2)
        },
        "lastWriteDate" : ISODate("2017-08-20T13:57:00Z")
    },
    "maxBsonObjectSize" : 16777216,
    "maxMessageSizeBytes" : 48000000,
    "maxWriteBatchSize" : 1000,
    "localTime" : ISODate("2017-08-20T13:57:08.312Z"),
    "maxWireVersion" : 5,
    "minWireVersion" : 0,
    "readOnly" : false,
    "ok" : 1
}

I know I'm not on a secondary, and I know where I am. Let's create a connection on localhost:27018:

introRep:PRIMARY> secondary = new Mongo("localhost:27018")
connection to localhost:27018

introRep:PRIMARY> secondaryDB = secondary.getDB("test")
test

introRep:PRIMARY> secondaryDB.example.find()
Error: error: {
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk"
}

We use the Mongo() constructor to instantiate a connection. Then, we store our database test in a variable to query it. But, as you can see, we end up with the error "not master and slaveOk=false".

Why is this? Well, secondaries may not have the most up to date data when you query it. Therefore, by default, read requests will be refused. If you accidentally connect to one of your secondaries, this will protect your application from having outdated data.

To allow queries on a secondary, we must tell Mongo that we are okay with reading from the secondary, like so :

introRep:PRIMARY> secondary.setSlaveOk()

Careful: The method is used on the connection, not the database. Let's try to query it again:

introRep:PRIMARY> secondaryDB.example.find()
{ "_id" : ObjectId("5999943d8d005c672ddfd9de"), "count" : 0 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9df"), "count" : 1 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9ec"), "count" : 14 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9eb"), "count" : 13 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9e1"), "count" : 3 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9f1"), "count" : 19 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9e4"), "count" : 6 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9e0"), "count" : 2 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9e9"), "count" : 11 }
{ "_id" : ObjectId("5999943d8d005c672ddfd9ea"), "count" : 12 }

...

Let's try to write to our secondary now:

introRep:PRIMARY> secondaryDB.example.insert({"count": 10001})
WriteResult({ "writeError" : { "code" : 10107, "errmsg" : "not master" } })
introRep:PRIMARY> secondaryDB.example.count()
10000

We have an error and the write failed. The secondary only accepts writes that it gets through replication.

New primary

One last interesting thing about replication. If your primary goes down, a secondary is elected as the new primary. Let's see it in action by stopping our primary:

db.adminCommand({"shutdown": 1})

You'll see some error messages because the instance we were connected to lost its connection. But don't worry, the shell won't crash. Now, run the isMaster() method we use before on the secondary:

>secondaryDB.isMaster()
{
    "hosts" : [
        "localhost:27017",
        "localhost:27018",
        "localhost:27019"
    ],
    "setName" : "introRep",
    "setVersion" : 1,
    "ismaster" : false,
    "secondary" : true,
    "primary" : "localhost:27019",
    "me" : "localhost:27018",
    "lastWrite" : {
        "opTime" : {
            "ts" : Timestamp(1503238614, 1),
            "t" : NumberLong(3)
        },
        "lastWriteDate" : ISODate("2017-08-20T14:16:54Z")
    },
    "maxBsonObjectSize" : 16777216,
    "maxMessageSizeBytes" : 48000000,
    "maxWriteBatchSize" : 1000,
    "localTime" : ISODate("2017-08-20T14:16:54.375Z"),
    "maxWireVersion" : 5,
    "minWireVersion" : 0,
    "readOnly" : false,
    "ok" : 1
}

Look at that! Our primary is no longer localhost:27017, but localhost:27019. Now, I can write directly to the new primary.

Conclusion

A few things to remember about replication:

 

This article taken from Damien Cossets blog

  • Clients ( your app for example ) can send a primary node the same operations you could send on a standalone server. You can write, read, build indexes ...

  • By default, you can't read from secondaries. This is a security check. MongoDB can't promise you that the data on your secondaries will be up-to-date with the primary. If you don't care about your data being the most up-to-date, you can use setSlaveOk() to read from the secondary.

  • Clients can't write to secondaries. Secondaries only accept write through replication.

  •