Explore topic-wise InterviewSolutions in .

YES, there is very MUCH a simpler way of achieving this without having to do this programmatically. The $unwind operator deconstructs an array field resulting in a document for each ELEMENT. 

Consider user “John” with multiple addresses

would result in 3 documents, one for each of the addresses

The $GRAPHLOOKUP LOOKS like this for a query from the EMPLOYEES collection where “manager” is the self-referencing field

The value of as, which is “Reporting Structure” in this case is the name of the array field which contains the documents traversed in the $graphLookup to reach the output document.   

For the following documents in the employee collection,

“Reporting Structure” for each output document would look like this

Recursive queries can be performed within a collection using $graphLookUp which is an aggregate pipeline stage.

If a collection has a self-referencing field like the classic example of MANAGER for an employee, then a QUERY to get the entire REPORTING structure for manager “David” would look like this

For the following documents in the employee collection,

Output of the above $graphLookup operation would result in the following 3 documents returned

The hierarchy starts with “David” which is specified in startWith and there on the DATA for each of the members in that reporting hierarchy are fetched recursively 

MongoDB has the db.collection.explain(), cursor.explain() and explain command to provide INFORMATION on the query PLAN. The results of explain contain a lot of information, key ones being

• rejectedPlans, other plans considered by the database (if any)
• winningPlan, the plan SELECTED by the query optimizer for execution
• executionStats, gives detailed information on the winning plan
• IXSCAN stage, if the query planner selects an index, the explain result will include this stage. This is one of the key THINGS to look for when ANALYZING the query plan for performance optimization

When "FAST reads" are the single most important criteria, EMBEDDED documents can be the best way to model one-to-one and one-to-many relationships.

Consider the example of certifications awarded to an employee, in the below example the certification data is embedded in the employee document which is a denormalized way of storing data

In a normalized form, there would be a reference to the employee document from the certificate document, example

Embedded documents are best used when the entire relationship data needs to be frequently retrieved together. Data can be retrieved via single query and hence is much faster.

Note: Embedded documents should not grow unbounded, otherwise it can slow down both read and write operations. Other factors like consistency and frequency of data change should be considered before making the final design decision for the application.

The $addToSet operator should be used with the $each modifier for this. The $each modifier allows the $addToSet operator to add multiple values to the array field.

Example, start ups are tagged as per the technology skill that they excel in

Now the start up needs to be updated with additional skills

The resultant document after update()

Note: There is no particular ORDERING of elements in the modified set, $addToSet does not guarantee that. Duplicate items will not be added.   

• Compound indexes not only support queries that match all the INDEX FIELDS, they also support queries on the index PREFIXES as well.

Consider the following compound index

The index prefixes are

Query plan will USE this index if the query has the following fields

• accountholder
• accountHolder and  accountNumber
• accountholder and accountNumber and currency
• Ordering is very IMPORTANT, the order of fields in the queries should match the order of fields in a compound index (left to right) for the index to be used

MongoDB provides database profiler which captures and stores detailed information about commands executed on running instance. Captured details INCLUDE CRUD operations, administrative commands and configuration commands. The data collected by the profiler is stored in the system.profile collection in the admin database.

By default, the profile is turned off. We can enable it and set to different profiling levels based on the requirements. It provides 3 profiling levels:

• 0 – Profiler OFF, no data collected, Default.
• 1 – Profiler ON, data collected for operations taking time lower than slowms.
• 2 – Profiler ON, data collected for all operations.

To capture the SLOW running queries, we can start the profiler with profiling level 1 or 2 and Default slow operation threshold is 100 milliseconds. We can CHANGE this threshold by specifying a slowms option.

Eg: To enable profiler which captures all queries slower than 50ms below command should be used:

When we create any collection in MongoDB, a UNIQUE index on the _id field is created automatically. This unique index prevents applications from inserting multiple same DOCUMENTS with the same value for the _id field. This is forced by the system and we cannot drop this index on the _id field. Moreover, in replica sets the unique _id values are used in the oplog to reference documents to update.

In a sharded CLUSTER if we do not have unique _id values across the sharded collection chunk migrations MAY fail as when documents migrate to another shard, any identical values will not be inserted to receiver shard. In such cases, we should code application such that it ensures uniqueness on _id for given collection across all shards in a sharded cluster.

If we use _id as the shard key, this uniqueness of values will automatically be forced by the system. In such case chunk ranges will be assigned to SINGLE shard and then the shard will force uniqueness on the values in that range.

In sharded cluster we may have a database which has sharded as well as non sharded collections. While the sharded collections are spread across all the shards, all the unsharded collections are stored on a single shard known as a primary shard. Every database in the sharded collection has its own primary shard. When we create any new database, mongos pick the shard with the least amount of data in cluster and marks it as a primary shard.

 If there is a need to change the primary shard we can do so by using the movePrimary command. This migration may take significant time to complete. We should not access any collections associated with MIGRATING database until the process completes. Also, the migration of primary shard should be done at the LEAN time as it may impact the performance of the OVERALL cluster.

Eg. To MIGRATE the primary shard of accounts database to Shard0007 below command should be used.

db.adminCommand( { MOVEPRIMARY : "accounts", to : "shard0007" } )

MongoDB provides several utilities for data movement activities like mongodump, mongoexport ETC. Mongodump is used to export the contents of the collection in an external file in the BSON(binary) format. The contents exported by this method can then be used by mongorestore command to restore in another database or different collection. Mongodump does not capture index data and only captures the data present in the backup. Since the contents are exported in binary format using this method we cannot use it for exporting to CSV file.

To export the contents in JSON or CSV format we can use the mongoexport command. The exported collection can then be RESTORED using mongoimport command. Since Mongoexport exports cannot export in BSON all the rich BSON data TYPES are not preserved while exporting the data. Due to this reason, mongoexport should be used with CAREFUL consideration.

Below is the command that can be used for the same.

Indexes are important to consider while designing the databases as they impact the performance of the APPLICATIONS. Without index, query will perform collection scan in which all the documents of the collection are scanned one by one to find the matching fields of the query. If we have indexed for a particular query, MongoDB can use the index to limit the number of documents scanned to execute the query as the indexes store the values of the query in ascending or descending ordered form.

While the indexes help in improving the performance of find operations for write operations like insert and update there can be a SIGNIFICANT NEGATIVE impact of adding indexes as with modifications with each write MongoDB WOULD need to update the indexes associated with the index also. This would be overhead on the system and we may end up with performance degradation.

 So while the find() will improve performance the operators updateOne and insertOne would degrade the performance as with every update or insert related indexes would need to be updated.

In MongoDB data set consistency is ensured by locking. In any database system long running queries DEGRADE the performance as requests and operations have to wait for a lock. Locking issues are INTERMITTENT and so need to be resolved immediately.

MongoDB provides US with tools and utilities to troubleshoot these locking issues. The serverStatus() command provides us a view of the system including the locking-related information. We should look for locks and a globalLock section of serverStatus() command for TROUBLESHOOTING locking issues.

We can use below commands to filter locking related information from the serverStatus output.

To get the approximate average wait time for a lock mode we can divide locks. timeAcquiringMicros by locks.acquireWaitCount.

To check the number of times deadlocks occurred locks.deadlockCount should be checked.

If the application performance is constantly degraded there might be concurrency issues, in such cases, we should look at globalLock.currentQueue.total. A high value INDICATES concurrency issues.

Sometimes globalLock.totalTime is high relative to uptime which suggests database has been in a lock state for a significant time.

MONGODB WiredTiger ensures data durability with journaling and checkpoints. Journals are write-ahead logs which checkpoints are point-in-time snapshots.

• Checkpoints:

In wiredTiger, with the start of each operation, a point-in-time snapshot is taken which presents a consistent view of in-memory data. WiredTiger then writes all snapshot data to the disk in a consistent way across all data files. This data on disk is durable and acts as a checkpoint in the data files. The checkpoint ensures all data files are consistent from the LAST checkpoint.

These checkpoints usually OCCUR every 60SEC so we have a consistent snapshot every 60sec of interval thus ensuring durability.

• Journal:

Journal is write-ahead logs which persist all data changes between 2 checkpoints. In case we require data between checkpoints for recovery these journal files can be used. These general files act as crash recovery files in case of interrupts. Once the system is BACK up these journal files can be replayed for recovery.

Concurrency in MongoDB is different for different storage engines. While WiredTiger uses document level concurrency control for write operations, MMAPV1 has concurrency at the collection level.

In WiredTiger locking is at document level due to which multiple clients can MODIFY documents at the same TIME and so uses optimistic concurrency control for most read and writes. Instead of exclusive locks, it uses only intent locks at the global, database and collection levels. In case the conflict is detected by storage engine between operations, one will incur a write conflict causing MongoDB to transparently retry that operation. 

SOMETIMES global “instance-wide” locks are required for global operations which involve multiple databases. Exclusive database lock incurs for operations such as DROPPING a collection.

MMAPv1 still uses collection level lock meaning if we have 3 applications running against the same collection, 2 would have to wait before FIRST application completes its own as it applies a collection level write lock.

The storage engine is the component that lies between the database and storage layer and is primarily responsible for managing data. MongoDB PROVIDES few choices of storage engines, enabling us to use best suited for our applications. Choosing the appropriate storage engine can significantly impact performance.

• WiredTiger Storage Engine (DEFAULT)

WiredTiger replaced MMAPV1 in 3.2 to become default storage engine. If we install MongoDB and do not specify any storage engine, wiredTiger is enabled. As it provides a document-level concurrency model, checkpointing, and compression it is suited for most WORKLOADS. It also supports encryption at rest in MongoDB Enterprise.

• In-Memory Storage Engine

Various applications REQUIRE predictable latencies which can be achieved by storing the documents in memory rather than on disk. In-Memory Storage Engine is helpful for such applications. It is available only in MongoDB enterprise edition.

• MMAPv1 Storage Engine (Deprecated as of MongoDB 4.0)

MongoDB started with MMAPv1 storage engine only but it was successful for a specific SUBSET of use cases only due to which it is deprecated from version 4.0.

If secondaries are FALLING behind they are experiencing REPLICATION lag, which is a delay in the application of oplog from primary to secondary. Replication lag can be a significant issue and can seriously affect MongoDB replica set deployments. Excessive replication lag makes a member ineligible to quickly become primary and increases the possibility of distributed read operations to be inconsistent.

We can check the replication lag by calling the rs.printSlaveReplicationInfo() method or by running the rs.status() command.

Possible causes of replication lag include:

1. Issues related to network between primary and secondary server. Sometimes the network becomes a bottleneck. We should ensure there is no packet loss or any other network related issues.
2. Disk issues on the secondary server. Sometimes there are different hardware on primary and secondary. If the primary is using SSD and secondary HDD, then disk flushes will be slow and secondary will lag behind. There might be other disk issues which can be checked by vmstat and iostat.
3. Sometimes there can be an issue due to concurrency issues as long-running operations on the primary can block replication on secondary. We should configure write CONCERN to require acknowledgement of replication to secondary. This prevents write operations from returning if replication cannot keep up with the write load.

MongoDB follows Role access control authorization model(RBAC). In this model, USERS are assigned one or more roles which provide them access to the database resources and operations. Apart from these role assignments users do not have any access to the resources. When we enable internal authentication it automatically enables client authorization. The authorization can be enabled by starting mongod with –auth OPTION or providing security.authorization setting in the config file.  

• Roles: Roles are groups of privileges, actions over resources that are granted to users over a given database. A role grants privileges to perform the specified actions on the resource. Each privilege is either specified explicitly in the role or inherited from another role or both.
• Actions: All operations and commands that a user can perform in MongoDB are called actions. Actions are performed on resources.
• Resources: Resources are any objects that HOLD a state in a database.
• Privilege: When a user performs an action on a given resource that constitutes privilege.

These group of privileges are roles that can be assigned to the user.

MongoDB provides several Build-in roles like Database User Roles, Database Administration Roles, Cluster Administration Roles, Backup and Restoration Roles, All-Database Roles, Superuser Roles But we can also create our own custom roles based on the requirement which are called User-defined roles.

For Example: Suppose we WANT to create a role to MANAGE all operations in the cluster we can create below user-defined role.

If we start Mongo server using mongod with the --auth option it will enable authorization but we will not be able to do anything, even listing databases using SHOW DBS will fail with an error "not authorized on admin to execute the command". This is because we have not authenticated yet to the database. But this is a NEW database with no users in this database, so how can we create new users with no authorization.

MongoDB provides localhost exception for creating the first user on the database without any authorization. With this first user, we can create other users with relevant access.

But there are a few considerations to it:

• localhost exception only applies for the first user in a database.
• localhost exception only applies when connected to the database VIA the localhost interface, MEANING on the same computer, if not we will not be able to UTILIZE localhost exception.

Loading EVERY document into RAM means that QUERY will not be using index efficiently and will have to fetch documents from disk to ram.

For using an index, the initial match in the find statement should either use index or index PREFIX.

Below query has b key to find that does not use any existing index or index prefix so it would have to fetch from disk.

In replication we have multiple copies of the same data in sync with each other. It is mainly useful for High availability purpose. While in sharding we divide our entire dataset in small chunks and distribute among several servers. Sharding is used where we have some SORT of bottleneck in terms of hardware or for getting the benefits of query parallelism. If our dataset is very small sharding would not provide many advantages but as the data grows we should MOVE to sharding.

Below are a few of the SITUATIONS where sharding is recommended over replication.

• If our MongoDB instance cannot keep up with the application's write load. We have exhausted RAM and CPU options for the server.
• Out data set is too big to fit in a single MongoDB instance. We have reached the disk limits for the server.
• To improve read performance for the application. By using targeted queries in the sharded cluster we can only view the data required skipping other data.
• The data set is taking too much time to backup and restore.

By breaking the dataset over shards will mean having more RESOURCES available to handle the subset of data it owns, and operations of moving data ACROSS machines for replication, backups, restores will also be faster.

In some cases, chunks can grow beyond the specified chunk size but cannot undergo a split. The most COMMON scenario is when a chunk represents a single shard key value. Since the chunk cannot split, it continues to grow beyond the chunk size, becoming a jumbo chunk. These jumbo chunks can become a PERFORMANCE bottleneck as they continue to grow, ESPECIALLY if the shard key value occurs with high frequency.

The addition of new data or new shards can result in data distribution imbalances WITHIN the CLUSTER. A particular shard may acquire more chunks than another shard, or the size of a chunk may grow beyond the configured maximum chunk size.

MongoDB ensures a balanced cluster using two processes: chunk splitting and the balancer.

Chunk split operations are carried out automatically by the system when any insert operation causes chunk to exceed the maximum chunk size. Balancer then migrates RECENTLY split chunks to new shards. But in some cases we may want to pre-split the chunks manually:

• If we have deployed a cluster using existing DATA, we may have large data and very few chunks. In cases, pre-splitting would be beneficial for even distribution.
• If the cluster is using hashed shard KEY, or we KNOW the distribution of our data very well, we can arrange for a distribution of data to be equilibrated between shards and pre-split the chunks.
• If we perform initial bulk load, all data would go to single shard and then those documents will migrate to other shards later doubling the number of writes. Alternatively, if we can pre-split the collection across the values avoiding the documents to be written twice.

To split the chunks manually we can use the split command with helper sh.splitFind() and sh.splitAt().

Example:

To split the chunk of employee collection for employee id field at a value of 713626 below command should be used.

We should be CAREFUL while pre-splitting chunks as sometimes it can lead to a collection with different sized chunks.

Shard key selection is based on the workload. Since the first query is being USED 90% it should be driving the selection for selection of shard key.

Combination of FIELDS from that query would make the best shard key. This ELIMINATES option b, c and d.

Option a and e uses a subset of fields from the most used workload and both can be shard key but option has more fields and so would be more suitable.

In MONGODB we can read from Primary as well as secondary members of the replica set. This behaviour can be CONTROLLED by us as we can define the desired Read preference to which clients route read operations to a member of the replica set. If we do not SPECIFY any real preference by default MongoDB will read from primary. There are situations when you would want to reduce the load on your primary by forcing APPLICATIONS to read from secondary.

Below are different MongoDB read preference modes:

• primary

This is the default mode. Applications read from the replica set primary.  

• primaryPreferred

In this mode, all applications read from primary but if the primary member is not available they start reading from secondary.

• secondary

All applications read from the secondary members of the replica set.

• secondaryPreferred

In this mode, all applications read from secondary but if any secondary member is not available they start reading from primary.

• NEAREST

In this mode, applications read from the member which is nearest to them in terms of network latency, irrespective of the member being primary or secondary.

Idempotence is the property of certain operations whereby they can be applied multiple times WITHOUT changing the result beyond the initial application. In MongoDB, oplog is idempotent meaning even if they are applied multiple times the same output will be produced. So if the server goes down and we need to apply oplogs there WOULD not be any inconsistency, even if it applies any logs that were ALREADY applied there will not be CHANGED in the database end state.

Also, there was a desire to have a new state of a document to be independent of the previous state. For these all operators that rely on the previous state to determine new value needed to be TRANSFORMED to see the actual values. For example, if an addition operation results in modifying the value from ‘21’ to ’30’, the operation should be changed to set value ‘30’ on the field. Replaying the operator multiple times should produce the same result.

MongoDB applies database operations on the PRIMARY and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. For each operation, there is separate oplog entry.

First, let’s check how many ROWS the query WOULD fetch by changing delete to find operation.

db.sample.find( { state : "WA" } )

This will give all the rows with the state is WA.

Now Ideally delete should remove all matching rows but query says deleteOne.

If the query would have said deleteMany then all the matching rows would have been deleted and there would be 3 oplog ENTRIES but deleteOne will remove first matching row. So 1 oplog entry will be generated with provided query

The oplog is operation logs that keep an update of all operations that modify the data stored in databases. We can define the oplog size while starting MONGODB by specifying the --oplog option. If we do not specify this option it will take the default values which is 5% of physical memory in case of wiredTiger. While the default value is sufficient for most workloads in some cases we may need to change the oplog size for the replica set.

OPlog size is CHANGED in a ROLLING manner, first, we change on all secondary and then a primary member of the replica set. To change oplog size

• First we need to CONNECT to any secondary member.
• Verify the size of current oplog by running below command on the local database.

use local

• Change the oplog size using ADMIN command replSetResizeOplog specifying a new size for oplog.

• Repeat the same process for other secondary members and then on the primary member of the replica set.

Every operation on the primary is logged in operation logs known as oplog. These oplogs are replicated to For a healthy replica set system, it is RECOMMENDED that all members are in sync with no replication lag. Data is first written on primary by the applications then replicated to secondary. This synchronization is important to MAINTAIN up-to-date copies of data on all members. Synchronization happens in 2 ways: initial sync and continuous replication.

• Initial Sync: When we add new member to replica set data from one member is copied to the new member. When we perform an initial sync, MongoDB copies all databases one by one except the local database. This is done by scanning all collections in the source database and inserting them on a new member. All indexes are also copied during the initial syn. There MIGHT be changes to the data set when initial sync happens. At the end of copy, the changes from already copied collections are applied USING oplog.
• Continuous Replication: After the initial sync the secondary members replicate data continuously. We can decide which member sync happens. The replication of secondary member from their sync source happens asynchronously. These replications happen using oplog.

The FIRST part of the query WOULD give all documents where y>=10. So we will have 2 documents i.e

Now the SECOND part of the query would update the value of Y for above 2 documents to 75, but we already have a document with value y:75, that will not be updated.

Finally, we will have one 1 document that will be updated by the provided query.

In MONGODB data is stored as JSON documents. These documents can have different sets of fields, with different data type for each field. For example, we can have a collection with number, varchar and array all as different documents.

It is not correct to say MongoDB is schemaless, in fact, schema plays an important role in the designing of MongoDB applications. MongoDB has a dynamic schema having database structure with collections and indexes. These collections can be created either implicitly or explicitly.

Due to the dynamic behaviour of the schema, MongoDB has several advantages over RDBMS systems.

Schema Migrations BECOME very easy as in traditional systems we had to use ALTER TABLE command after adding any column which could result in downtime. In MongoDB, such adjustments become transparent and automatic. For example, if we want to add CITY field to PEOPLE collection, we can add the attribute and resave, that’s it. Whereas in a traditional system we would have to run ALTER TABLE command FOLLOWED by REORG which would require downtime.

In MongoDB we have Built-in ROLES as well as custom roles. Built-in roles already have pre-defined access associated with them. We can assign these roles directly to users or groups for access. To run MONGOSTAT we WOULD require access to run the server status on the server.

Built-in role cluster monitor comes with required access for the same.

Custom roles or user-defined roles are the ones where we have to manually define access ACTIONS to a particular resource. MongoDB provides method db.createRole() for creating user-defined roles. These roles can be created in a specific database as MongoDB uses a combination of database and role name to uniquely define the role.

We will create a custom role mongostatRole that provides only the privileges to run mongostat.

First, we need to connect to mongod or mongos to the admin database with a user that has privileges to create roles in the admin as well as other databases.

Now we will create a desired custom role in the admin database.

This role can now be assigned to members of monitoring team.

only those DOCUMENTS that contain the field specified in the query. 

For the FOLLOWING documents in EMPLOYEE COLLECTION

{ _id: 1, name: "Jonas", linkedInProfile: null },   { _id: 2, name: “WILLIAMS” }

The query { linkedInProfile: { $exists: true } } will return only the employee “Jonas” 

This can be achieved in MongoDB using the $type operator. A null value, i.e., BSON type null has the type number 10.  Using this type number, only those documents can be retrieved whose value is null. 

Take the example of the below two documents in STARTUP collection

{ _id: 1, name: "XYZ Tech", website: null },   { _id: 2, name: “ABC Pvt Ltd” }

The query { website : { $type: 10 } } will retrieve only those documents where the website is null, in the above case it WOULD be the startup “XYZ Tech”

NOTE: The query { website : null } on the other HAND will match documents where the website is null or the documents where the website field does not EXIST. For the above collection data, this query will return both the startups.