Best way to organise data - New to MongoDB

Nicolas_SAINT_SEVER · 2024-08-05T15:24:46.705Z

Hi all,

I’m very new to mongoDB and databases creation. I’m currently using Mongo Atlas and Compass to try and see if it can fit my needs but before, I’d like to be sure of how MongoDB works and what are the best practises.
In my case, I automated multiple scraping bots that retrieve data about products each day (one bot for one website). The idea here is to follow overtime newly listed products / unlisted products and of course, products’ available quantity and price evolution.
As I see it (and do it as of today), my bots scrap everyday the info I need, then I add a column/feature with “Date” and ultimately, I export the “daily data” as a csv. For some websites, I started a few weeks ago so I have around 25 csvs.

I split my goal in two steps. First, when I have enough data, I’d like to track changes with a dashboard for every website I scraped (so one dashboard for each website scraped everyday).
Then, when i’m satisfied with the first step, I’ll need to aggregate / homogenize my data so a same product from website A and website B is considered similar (I guess I’ll have a huge standardization work to do) and I can sum quantities and deal with prices (lowest, highest, mean, quartile…) and track them over time.

So, here are my questions : Should I create multiple databases, one for each website ? Or only one database with one collection for each website ? And in the latter, I’ll have multiple documents in each collection (equivalent to one file per day).
In both these situations, does it seem achievable to realize my “short term goal” (step one) and long term one (aggregation / consolidation of data) ?

I’m kind of lost myself so sorry if that’s not particularly clear.

Any help would be much appreciated.

Kind regards,

Have a nice day all,

Joel_Lord · 2024-08-08T14:54:49.154Z

Hi Nicolas, and welcome to the MongoDB Forums!

Data modeling really depends on multiple factors. Since the data is similar in nature, I would probably group everything in a single collection. That will let you do complex aggregation queries later on.

I also recommend taking a look at these two resources. They will provide you with the right foundation for data modeling, and you’ll be able to tackle future challenges that might come your way :).

MongoDB University Data Modeling Learning Path (free)
MongoDB Data Modeling and Schema Design book