Schools in shutdown: How Sanoma Learning and MongoDB Atlas kept teaching as classes went virtual, supporting growth to 12 Million exercises daily
Enterprise IT is awash in buzzwords: scalability, performance, responsiveness. If you worry about databases, you’ll have words like these etched on your eyeballs. Sometimes, though, they stop being words and become critical realities — not to be rolled out over months of planning and testing, but things that must work, if not this hour, then the next. And not just for your company, but for everyone who relies on you.
This is where Bingel, the online learning platform for primary education, found itself on Friday the 13th, March 2020, the day Belgium went into lockdown. Bingel, part of the leading pan-European Sanoma Learning group of educational technology companies, had been having a good few years until then. It had been the first to deploy — and was easily the leader in — interactive online learning for Belgian primary school children. It was expanding into neighboring territories. Its colorful, adaptive, thoroughly gamified experience of exercises proved very popular with children, parents and teachers alike. Growth was good — very good, with 1.5 million exercises completed daily.
Then came Friday the 13th. The Belgian government had looked at the data for the COVID-19 pandemic, and reached the only possible conclusion: shut down, go home, stay home. Suddenly, there were no classrooms to go to. For Belgium's teachers, students and parents, they all went to the other familiar place they had come to trust and love over the past years, Bingel.
Becoming ‘critical infrastructure’
The company knew it had a role to play. Koen Rousseau, Digital Lead of Primary Education at Sanoma Learning, was leading the response. “Teachers had a million questions. We had to reach out and help. Over the weekend of the 14th and 15th, we opened up all our digital platforms, making everything freely available, reaching out to schools and saying, ‘We’re here, let’s do it together.’” By Sunday evening, there was nothing left to do but wait.
“Monday morning, 9 am, all hell broke loose,” says Rousseau. “We’d seen traffic skyrocketing since 8 am, but by 10 the first cracks started to appear. Parts of the system started to go down.”
One early sign came when his youngest daughter, who also had been using Bingel that morning, suddenly had trouble with the service and burst into the room yelling "Daddy, Bingel is broken, fix it! Rousseau knew he had to act quickly.
At first, the standard fallback procedure worked: “We switched it off and on again, and for a while it was running fine. Then at 1 pm it went down again.” Disk utilization reached saturation, with 12,000 queries per second and up to 96,000 connections. As the day progressed, the platform would crash three times. The magnitude of the problem — and the consequences — became clear. “I started to get calls from journalists. I once worked in the power industry, where I learned phrases like ‘critical infrastructure’ for things that, if you lose them, the nation’s in trouble. That afternoon, I realized that Bingel had become part of the nation’s critical infrastructure.” Rousseau needed a plan — and fast.
Fortunately, there was a plan. Not for this exactly, but the company had looked ahead to serving a lot more students. It just didn’t expect to have to do it overnight.
“We’d had a major crunch before, when we were much newer,” says Rousseau. “We’d started out with a very basic three-layer architecture, a front-end web server, some business logic and a plain relational database holding exercise resources.”
When the service was launched in 2011, there were low expectations. “We thought we were a bit too far into the future for our audience. It had so many new features, a digital currency to buy games, colorful islands you could leap between — it wasn’t like anything else.” But it turned out lots of children wanted the future to happen straight away. “Suddenly we had thousands upon thousands of children doing exercises at the same time, and the system couldn’t cope.”
Growth plans take a sudden turn
Bingel 2.0 was born. Out went the relational database. “It wasn’t a good match anyway. We had to map all the colorful stuff, all the rich experiences, into huge flat tables.” In came MongoDB with its document model. “It was the new kid on the block, but it was such a good match for our needs with its performance and scalability and accessibility, so we went for it.”
The company started off self-hosting MongoDB, in the days when there wasn’t another option, and it worked well for some time. “But we grew. We started with thousands of Belgian pupils a day using the service, but by the end of last year we had hundreds of thousands from across Europe, doing a million exercises a day. We were coping, but I wanted to release my dev team from worrying about database management, security and so on. So when MongoDB Atlas became an option, it was very interesting.”
At first, the planned migration followed the same path hundreds of other companies had taken. Conversations were held with the MongoDB Atlas team, numbers crunched, predictions made, lists and times for component moves and tests were drawn up. People were assigned tasks on paper. “If things had been normal,” says Rousseau, “then by late April we would be well along the way.”
On March 13, it became clear normal wasn’t going to be happening.
Rousseau picked up the phone to MongoDB and said: “Help!”
The Atlas ‘virtual war room’ for ‘unconventional migrations’
“It’s at times like these that you get to appreciate the difference between a supplier and a partner,” says Rousseau about what happened next. “I explained the position Bingel and I were in, and that immediately resonated with the MongoDB Atlas team. Within minutes there was a virtual war room. Data flow experts and architects and connectivity people were pulled out of other projects and made available. They started to make a plan, all within hours, all on Monday afternoon.”
By Tuesday, the non-production environments had been migrated to Atlas. Wednesday, the production environment followed suit. “The platform was being hammered on by hundreds of thousands of pupils and teachers, but even with that huge load, the critical database component was migrated to Atlas,” says Rousseau. “I’ve had a long history in IT, so I can say with authority that this was an unconventional migration.”
By Thursday, performance was rock-solid again. “We break records on a daily basis. We were peaking at 1.5 to 3 million exercises, but now we see 12 million, day after day after day. Things aren’t relaxing, but now it’s exciting, not frightening. Our devs love the new environment, they love the monitoring and performance management dashboards. They can see what's going on with the platform, how it interacts with the database, how we can find bottlenecks. It’s tremendous.”
The future, uncertain as it is, is once again on the table. “We’re continuing on our mission to reshape education. New technologies — AI, machine learning and analytics — will allow us to reach whole new levels of personalization. Now, we offer exercises based on pupils’ previous results. In the future, we may have content personalized based on how the pupil sees the world, where her passions are. For teachers, we want to reduce the admin overhead that comes with the job, give them immediate oversight with analytics and let them concentrate on their teaching. Everything is about making the most of what you’re good at, and letting the machines do the rest.”
"It’s at times like these that you get to appreciate the difference between a supplier and a partner."
Sanoma Learning is shaping the future of education. Learn more about their incredible work.