Migrate Data into a Time Series Collection
On this page
To migrate data from an existing collection into a time series collection:
Create a New Time Series Collection
To create a new time series collection, issue the following command in the
mongosh:
db.createCollection( "weathernew", { timeseries: { timeField: "ts", metaField: "metaData", granularity: "hours" } } )
For more information on the preceeding command, see Create a Time Series Collection.
Transform Data (Optional)
Time series collections support secondary indexes on the field specified as the
metaField. If the data model of your time series data does not have
a designated field for your metadata, you can transform your data to
create one. To transform the data in your existing collection, use
$merge or $out to create a temporary collection
with your time series data.
Consider a collection with weather data of the following format:
{ "_id" : ObjectId("5553a998e4b02cf7151190b8"), "st" : "x+47600-047900", "ts" : ISODate("1984-03-05T13:00:00Z"), "position" : { "type" : "Point", "coordinates" : [ -47.9, 47.6 ] }, "elevation" : 9999, "callLetters" : "VCSZ", "qualityControlProcess" : "V020", "dataSource" : "4", "type" : "FM-13", "airTemperature" : { "value" : -3.1, "quality" : "1" }, "dewPoint" : { "value" : 999.9, "quality" : "9" }, "pressure" : { "value" : 1015.3, "quality" : "1" }, "wind" : { "direction" : { "angle" : 999, "quality" : "9" }, "type" : "9", "speed" : { "rate" : 999.9, "quality" : "9" } }, "visibility" : { "distance" : { "value" : 999999, "quality" : "9" }, "variability" : { "value" : "N", "quality" : "9" } }, "skyCondition" : { "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" }, "cavok" : "N" }, "sections" : [ "AG1" ], "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 } }
To transform this data, we issue the following command:
db.weather_data.aggregate([ { $addFields: { metaData: { "st": "$st", "position": "$position", "elevation": "$elevation", "callLetters": "$callLetters", "qualityControlProcess": "$qualityControlProcess", "type": "$type" } }, }, { $project: { _id: 1, ts: 1, metaData: 1, dataSource: 1, airTemperature: 1, dewPoint: 1, pressure: 1, wind: 1, visibility: 1, skyCondition: 1, sections: 1, precipitationEstimatedObservation: 1 } }, { $out: "temporarytimeseries" } ])
After you run this command, you have an intermediary
temporarytimeseries collection:
db.temporarytimeseries.findOne() { "_id" : ObjectId("5553a998e4b02cf7151190b8"), "ts" : ISODate("1984-03-05T13:00:00Z"), "dataSource" : "4", "airTemperature" : { "value" : -3.1, "quality" : "1" }, "dewPoint" : { "value" : 999.9, "quality" : "9" }, "pressure" : { "value" : 1015.3, "quality" : "1" }, "wind" : { "direction" : { "angle" : 999, "quality" : "9" }, "type" : "9", "speed" : { "rate" : 999.9, "quality" : "9" } }, "visibility" : { "distance" : { "value" : 999999, "quality" : "9" }, "variability" : { "value" : "N", "quality" : "9" } }, "skyCondition" : { "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" }, "cavok" : "N" }, "sections" : [ "AG1" ], "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 }, "metaData" : { "st" : "x+47600-047900", "position" : { "type" : "Point", "coordinates" : [ -47.9, 47.6 ] }, "elevation" : 9999, "callLetters" : "VCSZ", "qualityControlProcess" : "V020", "type" : "FM-13" } }
Migrate Data into a Time Series Collection
To migrate your data from an existing collection that is not of type
timeseries into a time series collection, use mongodump and
mongorestore.
Warning
When migrating or backfilling into a time series collection you
should always insert the documents in order, from oldest to newest.
In this case mongodump exports documents in natural
order and the --maintainInsertionOrder option for
mongorestore guarantees the same insertion order for
documents.
For example, to export the temporarytimeseries collection, issue the
following command:
mongodump --uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \ --collection=temporarytimeseries --out=timeseries
The command returns the following output:
2021-06-01T16:48:39.980+0200 writing weather.temporarytimeseries to timeseries/weather/temporarytimeseries.bson 2021-06-01T16:48:40.056+0200 done dumping weather.temporarytimeseries (10000 documents)
To import timeseries/weather/temporarytimeseries.bson into the new
collection weathernew, issue the following command:
mongorestore --uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \ --collection=weathernew --noIndexRestore \ --maintainInsertionOrder \ timeseries/weather/temporarytimeseries.bson
The command returns the following output:
2021-06-01T16:50:56.639+0200 checking for collection data in timeseries/weather/temporarytimeseries.bson 2021-06-01T16:50:56.640+0200 restoring to existing collection weather.weathernew without dropping 2021-06-01T16:50:56.640+0200 reading metadata for weather.weathernew from timeseries/weather/temporarytimeseries.metadata.json 2021-06-01T16:50:56.640+0200 restoring weather.weathernew from timeseries/weather/temporarytimeseries.bson 2021-06-01T16:51:01.229+0200 no indexes to restore 2021-06-01T16:51:01.229+0200 finished restoring weather.weathernew (10000 documents, 0 failures) 2021-06-01T16:51:01.229+0200 10000 document(s) restored successfully. 0 document(s) failed to restore.
Note
Ensure that you run the preceeding command with the
--noIndexRestore option.
mongorestore cannot create indexes on time series
collections.
If your original collection had secondary indexes, manually recreate them now.