Alternative to using $merge with time series export

Brett_Spratt · 2024-10-23T15:21:48.063Z

I am trying to write a pipeline to export our current time series data that is stored in a regular collection to a time series collection. I can do it as a one-off using $out for testing say one sensor at a time but I can’t do it for all the data at once as $out overwrites the output collection. I tried using $merge instead, but I get the error “lock for timeseries is not allowed”. Are there any alternatives? If not I will just do it by pulling into Python first then writing, which will be slower.

Mark_Smith · 2024-10-28T09:37:01.482Z

That’s interesting! I wasn’t aware of this limitation, but there is a lot of stuff going on with timeseries collections, so it’s definitely possible.

Let me check internally and see if I can get you a more authoritative answer.

Nishith_Atreya · 2024-10-28T14:45:45.762Z

Currently, we don’t support $merge on time-series collections. We are aware of this limitation and have it on our roadmap as something we want to support in the future. $out is currently the recommended way or custom script might make sense in your case.

For $out, you would have to transform the data in your regular collection into new desired TS schema in a temporary collection and then mongodump/restore can be used on the transformed collection created from the $out output.

You can find more information on the steps here – https://www.mongodb.com/docs/v5.3/core/timeseries/timeseries-migrate-data-into-timeseries-collection/.