$out

You want to write 1 GiB of data as compressed BSON files to an S3 bucket named my-s3-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "s3": {
4
      "bucket": "my-s3-bucket",
5
      "filename": "big_box_store/",
6
      "format": {
7
        "name": "bson.gz"
8
      }
9
    }
10
  }
11
}

The s3.region is omitted and so, Atlas Data Federation determines the region where the bucket named my-s3-bucket is hosted from the storage configuration. $out writes five compressed BSON files:

1. The first 200 MiB of data to a file that $out names big_box_store/1.bson.gz.

   • The value of s3.filename serves as a constant in each filename. This value doesn't depend upon any document field or value.

   • Your s3.filename ends with a delimiter, so Atlas Data Federation appends the counter after the constant.

   • If it didn't end with a delimiter, Atlas Data Federation would have added a . between the constant and the counter, like big_box_store.1.bson.gz

   • Because you didn't change the maximum file size using s3.format.maxFileSize, Atlas Data Federation uses the default value of 200 MiB.

2. The second 200 MiB of data to a new file that $out names big_box_store/2.bson.gz.

3. Three more files that $out names big_box_store/3.bson.gz through big_box_store/5.bson.gz.

You want to write 90 MiB of data to JSON files to an S3 bucket named my-s3-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "s3": {
4
      "bucket": "my-s3-bucket",
5
      "region": "us-east-1",
6
      "filename": {"$toString": "$saleDate"},
7
      "format": {
8
        "name": "json",
9
        "maxFileSize": "100MiB"
10
      }
11
    }
12
  }
13
}

$out writes 90 MiB of data to JSON files in the root of the bucket. Each JSON file contains all of the documents with the same saleDate value. $out names each file using the documents' saleDate value converted to a string.

You want to write 176 MiB of data as BSON files to an S3 bucket named my-s3-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "s3": {
4
      "bucket": "my-s3-bucket",
5
      "region": "us-east-1",
6
      "filename": {
7
        "$concat": [
8
          "persons/",
9
          "$name", "/",
10
          "$uniqueId", "/"
11
        ]
12
      },
13
      "format": {
14
        "name": "bson",
15
        "maxFileSize": "200MiB"
16
      }
17
    }
18
  }
19
}

$out writes 176 MiB of data to BSON files. To name each file, $out concatenates:

• A constant string persons/ and, from the documents:

  • The string value of the name field,

  • A forward slash (/),

  • The string value of the uniqueId field, and

  • A forward slash (/).

Each BSON file contains all of the documents with the same name and uniqueId values. $out names each file using the documents' name and uniqueId values.

You want to write 154 MiB of data as compressed JSON files to an S3 bucket named my-s3-bucket.

Consider the following $out syntax:

1
{
2
  "$out": {
3
    "s3": {
4
      "bucket": "my-s3-bucket",
5
      "region": "us-east-1",
6
      "filename": {
7
        "$concat": [
8
          "big-box-store/",
9
          {
10
            "$toString": "$storeNumber"
11
          }, "/",
12
          {
13
            "$toString": "$saleDate"
14
          }, "/",
15
          "$partId", "/"
16
        ]
17
      },
18
      "format": {
19
        "name": "json.gz",
20
        "maxFileSize": "200MiB"
21
      }
22
    }
23
  }
24
}

$out writes 154 MiB of data to compressed JSON files, where each file contains all documents with the same storeNumber, saleDate, and partId values. To name each file, $out concatenates:

• A constant string value of big-box-store/,

• A string value of a unique store number in the storeNumber field,

• A forward slash (/),

• A string value of the date from the saleDate field,

• A forward slash (/),

• A string value of part ID from the partId field, and

• A forward slash (/).

You want to write 1 GiB of data as compressed BSON files to an Azure storage account mystorageaccount and container named my-container.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "azure": {
4
      "serviceURL": "http://mystorageaccount.blob.core.windows.net/",
5
      "container": "my-container",
6
      "filename": "big_box_store/",
7
      "format": {
8
        "name": "bson.gz"
9
      }
10
    }
11
  }
12
}

The azure.region is omitted and so Atlas Data Federation determines the region where the container named my-container is hosted from the storage configuration. $out writes five compressed BSON files:

1. The first 200 MiB of data to a file that $out names big_box_store/1.bson.gz.

   • The value of azure.filename serves as a constant in each filename. This value doesn't depend upon any document field or value.

   • Your azure.filename ends with a delimiter, so Atlas Data Federation appends the counter after the constant.

   • If it didn't end with a delimiter, Atlas Data Federation would have added a . between the constant and the counter, like big_box_store.1.bson.gz

   • Because you didn't change the maximum file size using azure.format.maxFileSize, Atlas Data Federation uses the default value of 200 MiB.

2. The second 200 MiB of data to a new file that $out names big_box_store/2.bson.gz.

3. Three more files that $out names big_box_store/3.bson.gz through big_box_store/5.bson.gz.

You want to write 90 MiB of data to JSON files to an Azure Blob Storage container named my-container.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "azure": {
4
      "serviceURL": "http://mystorageaccount.blob.core.windows.net/",
5
      "container": "my-container",
6
      "region": "eastus2",
7
      "filename": {"$toString": "$saleDate"},
8
      "format": {
9
        "name": "json",
10
        "maxFileSize": "100MiB"
11
      }
12
    }
13
  }
14
}

$out writes 90 MiB of data to JSON files in the root of the container. Each JSON file contains all of the documents with the same saleDate value. $out names each file using the documents' saleDate value converted to a string.

You want to write 176 MiB of data as BSON files to an Azure Blob Storage container named my-container.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "azure": {
4
      "serviceURL": "http://mystorageaccount.blob.core.windows.net/",
5
      "container": "my-container",
6
      "region": "eastus2",
7
      "filename": {
8
        "$concat": [
9
          "persons/",
10
          "$name", "/",
11
          "$uniqueId", "/"
12
        ]
13
      },
14
      "format": {
15
        "name": "bson",
16
        "maxFileSize": "200MiB"
17
      }
18
    }
19
  }
20
}

$out writes 176 MiB of data to BSON files. To name each file, $out concatenates:

• A constant string persons/ and, from the documents:

  • The string value of the name field,

  • A forward slash (/),

  • The string value of the uniqueId field, and

  • A forward slash (/).

Each BSON file contains all of the documents with the same name and uniqueId values. $out names each file using the documents' name and uniqueId values.

You want to write 154 MiB of data as compressed JSON files to an Azure Blob Storage container named my-container.

Consider the following $out syntax:

1
{
2
  "$out": {
3
    "azure": {
4
      "serviceURL": "http://mystorageaccount.blob.core.windows.net/",
5
      "container": "my-container",
6
      "region": "eastus2",
7
      "filename": {
8
        "$concat": [
9
          "big-box-store/",
10
          {
11
            "$toString": "$storeNumber"
12
          }, "/",
13
          {
14
            "$toString": "$saleDate"
15
          }, "/",
16
          "$partId", "/"
17
        ]
18
      },
19
      "format": {
20
        "name": "json.gz",
21
        "maxFileSize": "200MiB"
22
      }
23
    }
24
  }
25
}

$out writes 154 MiB of data to compressed JSON files, where each file contains all documents with the same storeNumber, saleDate, and partId values. To name each file, $out concatenates:

• A constant string value of big-box-store/,

• A string value of a unique store number in the storeNumber field,

• A forward slash (/),

• A string value of the date from the saleDate field,

• A forward slash (/),

• A string value of part ID from the partId field, and

• A forward slash (/).

You want to write 1 GiB of data as compressed BSON files to an Google Cloud Storage bucket named my-gcs-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "gcs": {
4
      "bucket": "my-gcs-bucket",
5
      "filename": "big_box_store/",
6
      "format": {
7
        "name": "bson.gz"
8
      }
9
    }
10
  }
11
}

The gcs.region is omitted and so, Atlas Data Federation determines the region where the bucket named my-gcs-bucket is hosted from the storage configuration. $out writes five compressed BSON files:

1. The first 200 MiB of data to a file that $out names big_box_store/1.bson.gz.

   • The value of gcs.filename serves as a constant in each filename. This value doesn't depend upon any document field or value.

   • Your gcs.filename ends with a delimiter, so Atlas Data Federation appends the counter after the constant.

   • If it didn't end with a delimiter, Atlas Data Federation would have added a . between the constant and the counter, like big_box_store.1.bson.gz

   • Because you didn't change the maximum file size using gcs.format.maxFileSize, Atlas Data Federation uses the default value of 200 MiB.

2. The second 200 MiB of data to a new file that $out names big_box_store/2.bson.gz.

3. Three more files that $out names big_box_store/3.bson.gz through big_box_store/5.bson.gz.

You want to write 90 MiB of data to JSON files to a Google Cloud Storage bucket named my-gcs-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "gcs": {
4
      "bucket": "my-gcs-bucket",
5
      "region": "us-central1",
6
      "filename": {"$toString": "$saleDate"},
7
      "format": {
8
        "name": "json",
9
        "maxFileSize": "100MiB"
10
      }
11
    }
12
  }
13
}

$out writes 90 MiB of data to JSON files in the root of the bucket. Each JSON file contains all of the documents with the same saleDate value. $out names each file using the documents' saleDate value converted to a string.

You want to write 176 MiB of data as BSON files to a Google Cloud Storage bucket named my-gcs-bucket.

Using the following $out syntax:

1
{
2
  "$out": {
3
    "gcs": {
4
      "bucket": "my-gcs-bucket",
5
      "region": "us-central1",
6
      "filename": {
7
        "$concat": [
8
          "persons/",
9
          "$name", "/",
10
          "$uniqueId", "/"
11
        ]
12
      },
13
      "format": {
14
        "name": "bson",
15
        "maxFileSize": "200MiB"
16
      }
17
    }
18
  }
19
}

$out writes 176 MiB of data to BSON files. To name each file, $out concatenates:

• A constant string persons/ and, from the documents:

  • The string value of the name field,

  • A forward slash (/),

  • The string value of the uniqueId field, and

  • A forward slash (/).

Each BSON file contains all of the documents with the same name and uniqueId values. $out names each file using the documents' name and uniqueId values.

You want to write 154 MiB of data as compressed JSON files to a Google Cloud Storage bucket named my-gcs-bucket.

Consider the following $out syntax:

1
{
2
  "$out": {
3
    "gcs": {
4
      "bucket": "my-gcs-bucket",
5
      "region": "us-central1",
6
      "filename": {
7
        "$concat": [
8
          "big-box-store/",
9
          {
10
            "$toString": "$storeNumber"
11
          }, "/",
12
          {
13
            "$toString": "$saleDate"
14
          }, "/",
15
          "$partId", "/"
16
        ]
17
      },
18
      "format": {
19
        "name": "json.gz",
20
        "maxFileSize": "200MiB"
21
      }
22
    }
23
  }
24
}

$out writes 154 MiB of data to compressed JSON files, where each file contains all documents with the same storeNumber, saleDate, and partId values. To name each file, $out concatenates:

• A constant string value of big-box-store/,

• A string value of a unique store number in the storeNumber field,

• A forward slash (/),

• A string value of the date from the saleDate field,

• A forward slash (/),

• A string value of part ID from the partId field, and

• A forward slash (/).

This example shows how to handle null values in the year field when creating a filename from the field value.

1
{
2
  "$out": {
3
    "s3": {
4
      "bucket": "my-s3-bucket",
5
      "region": "us-east-1",
6
      "filename": {
7
        "$concat": [
8
          "big-box-store/",
9
          {
10
            "$convert": {
11
              "input": "$year",
12
              "to": "string",
13
              "onNull": ""
14
            }
15
          }, "/"
16
        ]
17
      },
18
      "format": {
19
        "name": "json.gz",
20
        "maxFileSize": "200MiB"
21
      }
22
    }
23
  }
24
}

Consider the following BSON documents:

{
  "clientId": 102,
  "phoneNumbers": ["123-4567", "234-5678"],
  "clientInfo": {
    "name": "Taylor",
    "occupation": "teacher"
  }
}
{
  "clientId": "237",
  "phoneNumbers" ["345-6789"]
  "clientInfo": {
    "name": "Jordan"
  }
}

If you write the preceding BSON documents to Parquet format using $out to S3, the Parquet file schema for your BSON documents would look similar to the following:

message root {
  optional group clientId {
    optional int32 int;
    optional binary string; (STRING)
  }
  optional group phoneNumbers (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
  optional group clientInfo {
    optional binary name (STRING);
    optional binary occupation (STRING);
  }
}

Your Parquet data on S3 would look similar to the following:

1
clientId:
2
.int = 102
3
phoneNumbers:
4
.list:
5
..element = "123-4567"
6
.list:
7
..element = "234-5678"
8
clientInfo:
9
.name = "Taylor"
10
.occupation = "teacher"
11
12
clientId:
13
.string = "237"
14
phoneNumbers:
15
.list:
16
..element = "345-6789"
17
clientInfo:
18
.name = "Jordan"

The preceding example demonstrates how Atlas Data Federation handles complex data types:

• Atlas Data Federation maps documents at all levels to a Parquet group.

• Atlas Data Federation encodes arrays using the LIST logical type and the mandatory three-level list or element structure. To learn more, see Lists.

• Atlas Data Federation maps polymorphic BSON fields to a group of multiple single-type columns because Parquet doesn't support polymorphic columns. Atlas Data Federation names the group after the BSON field. In the preceding example, Atlas Data Federation creates a Parquet group named clientId for the polymorphic field named clientId with two children named after its BSON types, int and string.

This example shows how to handle null values in the year field when creating a filename from the field value.

1
{
2
  "$out": {
3
    "azure": {
4
      "serviceURL": "http://mystorageaccount.blob.core.windows.net/",
5
      "container": "my-container",
6
      "region": "eastus2",
7
      "filename": {
8
        "$concat": [
9
          "big-box-store/",
10
          {
11
            "$convert": {
12
              "input": "$year",
13
              "to": "string",
14
              "onNull": ""
15
            }
16
          }, "/"
17
        ]
18
      },
19
      "format": {
20
        "name": "json.gz",
21
        "maxFileSize": "200MiB"
22
      }
23
    }
24
  }
25
}

Consider the following BSON documents:

{
  "clientId": 102,
  "phoneNumbers": ["123-4567", "234-5678"],
  "clientInfo": {
    "name": "Taylor",
    "occupation": "teacher"
  }
}
{
  "clientId": "237",
  "phoneNumbers" ["345-6789"]
  "clientInfo": {
    "name": "Jordan"
  }
}

If you write the preceding BSON documents to Parquet format using $out to Azure, the Parquet file schema for your BSON documents would look similar to the following:

message root {
  optional group clientId {
    optional int32 int;
    optional binary string (STRING);
  }
  optional group phoneNumbers (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
  optional group clientInfo {
    optional binary name (STRING);
    optional binary occupation (STRING);
  }
}

Your Parquet data in Azure Blob Storage would look similar to the following:

1
clientId:
2
.int = 102
3
phoneNumbers:
4
.list:
5
..element = "123-4567"
6
.list:
7
..element = "234-5678"
8
clientInfo:
9
.name = "Taylor"
10
.occupation = "teacher"
11
12
clientId:
13
.string = "237"
14
phoneNumbers:
15
.list:
16
..element = "345-6789"
17
clientInfo:
18
.name = "Jordan"

The preceding example demonstrates how Atlas Data Federation handles complex data types:

• Atlas Data Federation maps documents at all levels to a Parquet group.

• Atlas Data Federation encodes arrays using the LIST logical type and the mandatory three-level list or element structure. To learn more, see Lists.

• Atlas Data Federation maps polymorphic BSON fields to a group of multiple single-type columns because Parquet doesn't support polymorphic columns. Atlas Data Federation names the group after the BSON field. In the preceding example, Atlas Data Federation creates a Parquet group named clientId for the polymorphic field named clientId with two children named after its BSON types, int and string.

This example shows how to handle null values in the year field when creating a filename from the field value.

1
{
2
  "$out": {
3
    "gcs": {
4
      "bucket": "my-gcs-bucket",
5
      "region": "us-central1",
6
      "filename": {
7
        "$concat": [
8
          "big-box-store/",
9
          {
10
            "$convert": {
11
              "input": "$year",
12
              "to": "string",
13
              "onNull": ""
14
            }
15
          }, "/"
16
        ]
17
      },
18
      "format": {
19
        "name": "json.gz",
20
        "maxFileSize": "200MiB"
21
      }
22
    }
23
  }
24
}

Consider the following BSON documents:

{
  "clientId": 102,
  "phoneNumbers": ["123-4567", "234-5678"],
  "clientInfo": {
    "name": "Taylor",
    "occupation": "teacher"
  }
}
{
  "clientId": "237",
  "phoneNumbers" ["345-6789"]
  "clientInfo": {
    "name": "Jordan"
  }
}

If you write the preceding BSON documents to Parquet format using $out to GCP, the Parquet file schema for your BSON documents would look similar to the following:

message root {
  optional group clientId {
    optional int32 int;
    optional binary string; (STRING)
  }
  optional group phoneNumbers (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
  optional group clientInfo {
    optional binary name (STRING);
    optional binary occupation (STRING);
  }
}

Your Parquet data on Google Cloud Storage would look similar to the following:

1
clientId:
2
.int = 102
3
phoneNumbers:
4
.list:
5
..element = "123-4567"
6
.list:
7
..element = "234-5678"
8
clientInfo:
9
.name = "Taylor"
10
.occupation = "teacher"
11
12
clientId:
13
.string = "237"
14
phoneNumbers:
15
.list:
16
..element = "345-6789"
17
clientInfo:
18
.name = "Jordan"

The preceding example demonstrates how Atlas Data Federation handles complex data types:

• Atlas Data Federation maps documents at all levels to a Parquet group.

• Atlas Data Federation encodes arrays using the LIST logical type and the mandatory three-level list or element structure. To learn more, see Lists.

• Atlas Data Federation maps polymorphic BSON fields to a group of multiple single-type columns because Parquet doesn't support polymorphic columns. Atlas Data Federation names the group after the BSON field. In the preceding example, Atlas Data Federation creates a Parquet group named clientId for the polymorphic field named clientId with two children named after its BSON types, int and string.