Migrate Data into a Time Series Collection
On this page
This page describes how to migrate data into a time series collection using an aggregation pipeline. Alternatively, to learn how to migrate data into a time series collection using MongoDB Relational Migrator, see Configure a Time Series Collection with Relational Migrator.
About this Task
Starting in MongoDB version 7.0, you can use the $out
aggregation stage to migrate data from an existing collection into a
time series collection. If your
collection stores data that you want to compare across time intervals,
use a time series collection to improve performance and storage. For more
information on the benefits of time series collections, see
Time Series Collections.
Before you Begin
Consider a weatherdata
collection that contains time and metadata information:
db.weatherdata.insertOne( { _id: ObjectId("5553a998e4b02cf7151190b8"), st: "x+47600-047900", ts: ISODate("1984-03-05T13:00:00Z"), position: { type: "Point", coordinates: [ -47.9, 47.6 ] }, elevation: 9999, callLetters: "VCSZ", qualityControlProcess: "V020", dataSource: "4", type: "FM-13", airTemperature: { value: -3.1, quality: "1" }, dewPoint: { value: 999.9, quality : "9" }, pressure: { value: 1015.3, quality: "1" }, wind: { direction: { angle: 999, quality: "9" }, type: "9", speed: { rate: 999.9, quality: "9" } }, visibility: { distance: { value: 999999, quality : "9" }, variability: { value: "N", quality: "9" } }, skyCondition: { ceilingHeight: { value: 99999, quality: "9", determination: "9" }, cavok: "N" }, sections: [ "AG1" ], precipitationEstimatedObservation: { discrepancy: "2", estimatedWaterDepth: 999 } } )
Steps
Optional. Create a metadata field.
If your collection does not include a specific field that contains all metadata, transform your data to define one.
The following pipeline stages add a metaData
field and use
$project
to include or exclude the remaining fields in
the document. In this example, the metaData
field becomes the
metaField
of the time series collection that you create.
Note
Choosing the right field as your time series metaField
optimizes
both storage and query performance. For more information on
metaField
selection and best practices, see
metaFields.
{ $addFields: { metaData: { st: "$st", position: "$position", elevation: "$elevation", callLetters: "$callLetters", qualityControlProcess: "$qualityControlProcess", type: "$type" } }, }, { $project: { _id: 1, ts: 1, metaData: 1, dataSource: 1, airTemperature: 1, dewPoint: 1, pressure: 1, wind: 1, visibility: 1, skyCondition: 1, sections: 1, precipitationEstimatedObservation: 1 } }
Create your time series collection and insert your data.
The example below performs the following operations:
Uses the
$addFields
aggregation stage to add ametaData
field to yourweather_data
collection.Uses the
$out
aggregation stage with thetimeseries
option to create aweathernew
time series collection in themydatabase
database.Defines the
metaField
of yourweathernew
collection as themetaData
field.Defines the
timeField
of yourweathernew
collection as thets
field.
Note
The timeField
of a time series collection must be a date type.
db.weather_data.aggregate( [ { $addFields: { metaData: { st: "$st", position: "$position", elevation: "$elevation", callLetters: "$callLetters", qualityControlProcess: "$qualityControlProcess", type: "$type" } }, }, { $project: { _id: 1, ts: 1, metaData: 1, dataSource: 1, airTemperature: 1, dewPoint: 1, pressure: 1, wind: 1, visibility: 1, skyCondition: 1, sections: 1, precipitationEstimatedObservation: 1 } }, { $out: { db: "mydatabase", coll: "weathernew", timeseries: { timeField: "ts", metaField: "metaData" } } } ] )
For the aggregation stage syntax, see
$out
. For a full explanation of the time series
options, see the Time Series Field Reference.
After you run this aggregation pipeline, you can use
findOne()
to view a document in your
weathernew
time series collection:
db.weathernew.findOne()
The operation returns the following document:
{ _id: ObjectId("5553a998e4b02cf7151190b8"), ts: ISODate("1984-03-05T13:00:00Z"), metaData: { st: "x+47600-047900", position: { type: "Point", coordinates: [ -47.9, 47.6 ] }, elevation: 9999, callLetters: "VCSZ", qualityControlProcess: "V020", type: "FM-13" }, dataSource: "4", airTemperature: { value: -3.1, quality: "1" }, dewPoint: { value: 999.9, quality: "9" }, pressure: { value: 1015.3, quality: "1" }, wind: { direction: { angle: 999, quality: "9" }, type: "9", speed: { rate: 999.9, quality: "9" } }, visibility: { distance: { value: 999999, quality: "9" }, variability: { value: "N", quality: "9" } }, skyCondition: { ceilingHeight: { value: 99999, quality: "9", determination: "9" }, cavok: "N" }, sections: [ "AG1" ], precipitationEstimatedObservation: { discrepancy: "2", estimatedWaterDepth: 999 } }
Next Steps
If your original collection had secondary indexes, manually recreate them now.
If your time series collection includes timeField
values before
1970-01-01T00:00:00.000Z
or after 2038-01-19T03:14:07.000Z
,
MongoDB logs a warning and disables some query optimizations that make
use of the internal clustered index. To regain query performance and
resolve the log warning, create a secondary index on the timeField
.