Deploy a Federated Database Instance

1. Select the dataset for your federated database instance from the Data Sources section.

   Click Add Data Sources to select your data store.

2. Specify your data store.

   Choose Google Cloud Storage to configure a federated database instance for data in Google Cloud Storage buckets.

   Corresponds to stores.[n].provider JSON configuration setting.

3. Select a Google Cloud Service Account for Atlas.

   You can select an existing Google Cloud Service Account that Atlas is authorized for from the role selection dropdown list or choose Create a Google Service Account.

   If you selected an existing account that Atlas is authorized for, click Next and proceed to the next step to list your Google Cloud Storage buckets.

   If you are creating a new service account, select Create a Google Service Account and click Next.

   Tip

   See also:

   • Set Up and Manage Google Cloud Service Account Access </security/set-up-unified-aws-access/>`

   • Create a Cloud Provider Access Role

4. In the Configure Google Cloud Storage modal, follow the provided instructions to configure the Google Cloud CLI, then click Next.

5. Configure Google Cloud Storage.

   1. Enter the name of your Google Cloud Storage bucket.

      Corresponds to the stores.[n].bucket JSON configuration setting.

   2. Specify whether the bucket is Read-only or both Read and write.

      Atlas can only query Read-only buckets; if you wish to query and save query results to your Google Cloud Storage bucket, choose Read and write.

   3. Select the region of the Google Cloud Storage bucket.

      Corresponds to the stores.[n].region JSON configuration setting.

      Note

      You can't create a federated database instance if Atlas Data Federation is unable to retrieve the region of the specified Google Cloud Storage bucket.

   4. Grant access to your Google Cloud project.

      1. In the Google Cloud console for the project that hosts your Google Cloud Storage bucket, navigate to IAM and Admin, then navigate to IAM.

      2. Click Grant Access. In the modal that appears, in the New principals field, enter the Google Cloud Service Account associated with your federated database instance.

      3. To grant read-only access to the bucket, apply the storage.viewer role. To grant read-write access to the bucket, additionally apply the storage.editor role.

   5. Optional. Specify a prefix that Data Federation should use when searching the files in the Google Cloud Storage bucket. If omitted, Data Federation does a recursive search for all files from the root of the Google Cloud Storage bucket.

      Corresponds to the stores.[n].prefix JSON configuration setting.

   6. Click Validate and finish.

6. Define the path structure for your files in the Google Cloud Storage bucket and click Next.

   For example:

   https://storage.googleapis.com/<path>/<to>/<files>/<filename>.<file-extension>

   To add additional paths to data on your Google Cloud Storage bucket, click Add Data Source and enter the path. To learn more about paths, see Define Path for S3 Data.

   Corresponds to the databases.[n].collections.[n].dataSources.[n].path JSON configuration setting.

7. Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.

   1. (Optional) Click the for the:

      • Database to edit the database name. Defaults to VirtualDatabase[n].

        Corresponds to databases.[n].name JSON configuration setting.

      • Collection to edit the collection name. Defaults to VirtualCollection[n].

        Corresponds to databases.[n].collections.[n].name JSON configuration setting.

      • View to edit the view name.

      You can click:

      • Add Database to add databases and collections.

      • associated with the database to add collections to the database.

      • associated with the collection to add views on the collection. To create a view, you must specify:

        • The name of the view.

        • The pipeline to apply to the view.

          The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.

        To learn more about views, see:

        • Views

        • db.createView

      • associated with the database, collection, or view to remove it.

   2. Select Google Cloud Storage from the dropdown in the Data Sources section.

   3. Drag and drop the data store to map with the collection.

      Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for Google Cloud Storage data store should look similar to the following:

{
  "stores" : [
    {
      "name" : "<string>",
      "provider" : "<string>",
      "region" : "<string>",
      "bucket" : "<string>",
      "prefix": "<string>",
      "delimiter": "<string>"
    }
  ],
  "databases" : [
    {
      "name" : "<string>",
      "collections" : [
	{
	  "name" : "<string>",
	  "dataSources" : [
	    {
	      "storeName" : "<string>",
	      "path" : "<string>",
	      "defaultFormat" : "<string>",
	      "provenanceFieldName": "<string>",
	      "omitAttributes": <boolean>
	    }
	  ]
	}
      ],
      "maxWildcardCollections" : <integer>,
      "views" : [ 
	{
	  "name" : "<string>", 
	  "source" : "<string>", 
	  "pipeline" : "<string>" 
	}
      ] 
    }
  ]
}
    

For more information on the configuration settings, see Define Data Stores for a Federated Database Instance.

1. Define your AWS S3 data store.

   Edit the JSON configuration settings shown in the UI for stores. Your stores cofiguration setting should resemble the following:

      "stores" : [
        {
          "name" : "<string>",
          "provider" : "<string>",
          "region" : "<string>",
          "bucket" : "<string>",
          "additionalStorageClasses" : ["<string>"],
          "prefix" : "<string>",
          "delimiter" : "<string>",
          "includeTags": <boolean>,
          "public": <boolean>
        }
      ]

   To learn more about these configuration settings, see stores.

2. Define your federated database instance virtual databases, collections, and views.

   Edit the JSON configuration settings shown in the UI for databases. Your databases cofiguration setting should resemble the following:

   "databases" : [
     {
       "name" : "<string>",
       "collections" : [
         {
           "name" : "<string>",
           "dataSources" : [
             {
               "storeName" : "<string>",
               "defaultFormat" : "<string>",
               "path" : "<string>",
               "provenanceFieldName": "<string>",
               "omitAttributes": <boolean>
             }
           ]
         }
       ], 
       "maxWildcardCollections" : <integer>,
       "views" : [
         {
           "name" : "<string>",
           "source" : "<string>",
           "pipeline" : "<string>"
         }
       ]
     }
   ]

   To learn more about these configuration settings, see databases.