Migrate Data into a Time Series Collection with Database Tools

On this page

Steps

Create a new time series collection.
(Optional) Transform your data.
Export your original collection.
Import your collection.

Use the following steps to migrate data from an existing collection to a time series collection with mongodump and mongorestore.

Steps

Create a new time series collection.

To create a new time series collection, issue the following command in the mongosh:

db.createCollection(
   "weathernew", {
      timeseries: {
         timeField: "ts",
         metaField: "metaData",
         granularity: "hours"
      }
   }
)

This example uses sample data for the timeField, metaField, and granularity. For more information on the preceeding command, see Create a Time Series Collection.

(Optional) Transform your data.

Time series collections support secondary indexes on the field specified as the metaField. If the data model of your time series data does not have a designated field for your metadata, you can transform your data to create one. To transform the data in your existing collection, use $out to create a temporary collection with your time series data.

Consider a collection with weather data of the following format:

db.weatherdata.insertOne(
   {
      _id: ObjectId("5553a998e4b02cf7151190b8"),
      st: "x+47600-047900",
      ts: ISODate("1984-03-05T13:00:00Z"),
      position: {
         type: "Point",
         coordinates: [ -47.9, 47.6 ]
      },
      elevation: 9999,
      callLetters: "VCSZ",
      qualityControlProcess: "V020",
      dataSource: "4",
      type: "FM-13",
      airTemperature: { value: -3.1, quality: "1" },
      dewPoint: { value: 999.9, quality : "9" },
      pressure: { value: 1015.3, quality: "1" },
      wind: {
         direction: { angle: 999, quality: "9" },
         type: "9",
         speed: { rate: 999.9, quality: "9" }
      },
      visibility: {
         distance: { value: 999999, quality : "9" },
         variability: { value: "N", quality: "9" }
      },
      skyCondition: {
         ceilingHeight: { value: 99999, quality: "9", determination: "9" },
         cavok: "N"
      },
      sections: [ "AG1" ],
      precipitationEstimatedObservation: {
         discrepancy: "2",
         estimatedWaterDepth: 999
      }
   }
)

Note

Choosing the right field as your time series metaField and grandularity optimizes both storage and query performance. For more information on field selection and best practices, see metaField and Granularity Best Practices.

The pipline below performs the following operations:

Uses $addFields to add a metaData field to the weather_data collection.
Uses $project to include or exclude the remaining fields in the document.
Uses $out to create a temporary collection called temporarytimeseries.

db.weather_data.aggregate([
{
   $addFields: {
      metaData: {
         "st": "$st",
         "position": "$position",
         "elevation": "$elevation",
         "callLetters": "$callLetters",
         "qualityControlProcess": "$qualityControlProcess",
         "type": "$type"
      }
   },
}, {
   $project: {
      _id: 1,
      ts: 1,
      metaData: 1,
      dataSource: 1,
      airTemperature: 1,
      dewPoint: 1,
      pressure: 1,
      wind: 1,
      visibility: 1,
      skyCondition: 1,
      sections: 1,
      precipitationEstimatedObservation: 1
   }
}, {
   $out: "temporarytimeseries"
}
])

After you run this command, you have an intermediary temporarytimeseries collection:

db.temporarytimeseries.findOne()
{
   "_id" : ObjectId("5553a998e4b02cf7151190b8"),
   "ts" : ISODate("1984-03-05T13:00:00Z"),
   "dataSource" : "4",
   "airTemperature" : { "value" : -3.1, "quality" : "1" },
   "dewPoint" : { "value" : 999.9, "quality" : "9" },
   "pressure" : { "value" : 1015.3, "quality" : "1" },
   "wind" : {
   "direction" : { "angle" : 999, "quality" : "9" },
   "type" : "9",
   "speed" : { "rate" : 999.9, "quality" : "9" }
   },
   "visibility" : {
   "distance" : { "value" : 999999, "quality" : "9" },
   "variability" : { "value" : "N", "quality" : "9" }
   },
   "skyCondition" : {
   "ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" },
   "cavok" : "N"
   },
   "sections" : [ "AG1" ],
   "precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 },
   "metaData" : {
   "st" : "x+47600-047900",
   "position" : {
      "type" : "Point",
      "coordinates" : [ -47.9, 47.6 ]
   },
   "elevation" : 9999,
   "callLetters" : "VCSZ",
   "qualityControlProcess" : "V020",
   "type" : "FM-13"
   }
}

Export your original collection.

To export your data from an existing collection that is not of type timeseries use mongodump.

Warning

When migrating or backfilling into a time series collection, always insert the documents in order, from oldest to newest. In this case, mongodump exports documents in natural order and the --maintainInsertionOrder option for mongorestore guarantees the same insertion order for documents.

For example, to export the temporarytimeseries collection, issue the following command:

mongodump
   --uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \
   --collection=temporarytimeseries --out=timeseries

The command returns the following output:

2021-06-01T16:48:39.980+0200  writing weather.temporarytimeseries to timeseries/weather/temporarytimeseries.bson
2021-06-01T16:48:40.056+0200  done dumping weather.temporarytimeseries
(10000 documents)

Import your collection.

To import your data into a timeseries collection, use mongorestore.

Important

Ensure that you run the mongorestore command with the --noIndexRestore option. mongorestore cannot create indexes on time series collections.

The following operation imports timeseries/weather/temporarytimeseries.bson into the new collection weathernew:

mongorestore
   --uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \
   --collection=weathernew --noIndexRestore \
   --maintainInsertionOrder \
   timeseries/weather/temporarytimeseries.bson