KBank successfully migrate data from 16 databases with minimal downtime
MAKE, by KBank in Thailand, is a banking application featuring the categorization of transactions, receipt of monthly expenditure summaries, and a unique savings technology called "cloud pockets."
Nicha Rojsrikul, Innovation Engineer, KASIKORN Business Technology Group, presented at MongoDB.local Bangkok, sharing the app’s migration from self-hosted community edition of MongoDB database to MongoDB Atlas, the SaaS offering of MongoDB. A move prompted by the community edition reaching its end of life, and the manpower required to update and maintain it in-house. Atlas, by contrast, came with low effort upgrading, detailed metrics and a support team.
The team’s recommendations for migration were summed up as follows:
**
⁃ Use live migration
⁃ test with copy of production data first
⁃ Split or delay migration if necessary **
Here’s the step-by-step guide:
1) Know your system
MAKE had 16 MongoDB databases to migrate, and most of them were less than 10 gigabytes. Only two databases were larger than 100 GB. As a banking app, security was key, meaning the migration would have to run within the VPC network.
2) Do your research
a. Migration tools
MAKE considered 3 tools:
**
⁃
mongodump
and
mongostore
: a common-line tool for breaking up MongoDB could also be adapted to migrate data. Although simple, it involved downtime.
⁃
mongoimport
and
mongoexport:
generates human readable data, like JSON, CSB, or TSV, instead of binary backup. Simple to use, has an extra data filtering function, but is slower than mongostore.
⁃
mongomirror
: an official tool needing no downtime. If set up for VPC peering, either the peer's VPC CIDR block (or a subset) or the Security Group pf the peer VPC. **
MAKE choose mongodump and mongostore.
b. Verification tools
⁃
dbHash
: an application command for MongoDB that calculates checksum.
⁃
GoDriver
: having previously used GoServer, MAKE decided to update to the then latest GoDriver 1.10
3) Prepare for migration
a. Preparing target databases
MAKE had to create a VPC peering connection between MongoDB Atlas and Google Cloud Platform. After that, the MongoDB clusters – using higher than needed clusters for quicker migration.
b. Preparing the system
A simple upgrade of MongoDB Driver on MAKE’s system was followed by a regression test and performance test.
c. Preparing the scripts
To make the migration easier, a simple script was created that could run with different param files, enabling them to simply provide parameters such as file to read, database and number. This eliminates the need for 16 separate scripts, thus accelerating migration.
d. Testing the scripts
Nicha recommends testing with production data and gathering information like system metrics, CPU usage, memory usage, and storage usage for troubleshooting purposes along the way.
4) Migration and deployment
With 16 databases to migrate, and customers waiting to use the app, simplicity and minimum downtime were key. MAKE had two cloud engineers run migration on two separate databases simultaneously.
For large databases, such as the 126 GB log database and the 100GB transaction database, each was first duplicated on a higher performance node and then migrated from there to take advantage of the higher IOPS. Once all data had been migrated successfully, MAKE connected its service to the new database, tested the system, and proceeded to deployment.
Now, the MAKE banking app is running all its migrated services on the new MongoDB Atlas database.
