Anomaly Detection
Import Historical Data
How to analyze cloud spend that pre-dates when you started using the CMP

Overview

This feature enables you to upload all your historical Google Cloud billing data into DoiT Cloud Management Platform (CMP) to immediately analyze your cloud spend and promptly identify optimization opportunities through Cloud Analytics, BigQuery Lens, and other CMP features.
To get started, first, you need to ensure that you have a Service Account that connects your Google Cloud Organization to CMP. Then, you'll have to grant two additional required permissions.
We import your historical Cloud Billing data from Google BigQuery. To start collecting your Google Cloud Billing data, you must enable Cloud Billing data export to BigQuery.
You can only import as much historical Cloud Billing data as you already have available in BigQuery.
Suppose you enable Cloud Billing data export to BigQuery for the first time while following the steps in this document. In that case, you will only have around one month of historical Cloud Billing data to import into the CMP.

Identify the service account

Within the CMP, select the gear icon (Settings) from the top navigation bar, then select Google Cloud under Cloud settings. The CMP will take you to the Google Cloud settings screen:
A screenshot showing the Google Cloud settings screen
If you do not have a service account configured yet, you will need to create a service account before you can continue.
Select your desired service account, copy the service account email, and proceed to grant the required permissions (see next section).

Grant the required Permissions

You need to have the appropriate IAM role/permission in GCP to grant the required permissions to the service account
BigQuery Data Viewer role
  1. 1.
    From GCP, navigate to the BigQuery Console and open the dataset where the billing data is stored, then click "Share dataset"
  2. 2.
    In the Dataset Permissions tab, click "Add Member" and paste the service account email that was linked to CMP as identified in Step 1.
  3. 3.
    Click "Select a Role" highlight BigQuery then choose BigQuery Data Viewer
  4. 4.
    Click "Add"
  5. 5.
    Click "Done"
A screenshot showing the location of the Share Dataset option
A screenshot showing the process of granting the BigQuery Data Viewer role
Grant the bigquery.jobs.create permission
Depending on the features initially enabled from the process Connect Google Cloud Service Account, you may already have this permission granted to the role. Follow the steps below to verify / add the permission to the project.
  1. 1.
    From the GCP console, select your organization from the dropdown options
  2. 2.
    On the navigation menu, go to "IAM & Admin" then select "Roles"
  3. 3.
    Click on the role "DoiT CMP Service Account"
  4. 4.
    Check if bigquery.jobs.create is included in the assigned permissions, if yes, no further action is needed since these permissions will be inherited by the projects
  5. 5.
    If bigquery.jobs.create permission is not yet added, click on "Edit Role"
  6. 6.
    Click "Add Permissions"
  7. 7.
    Filter by "Permission: bigquery.jobs.create" and check the box for this permission
  8. 8.
    Click "Add"
A screenshot showing how to select an organization, access the IAM & Admin menu item, and then select Roles
A screenshot showing the location of the service account listing
A screenshot showing the location of the Edit Role option and the bigquery.jobs.create permission listing
A screenshot showing the Add Permissions option, a list of permissions, and the Add button

Link your Google Cloud Direct Billing Account

Required Permission: Assets Manager
Within the CMP, select Billing from the top navigation bar and then select Assets. The CMP will take you to the Assets page.
From the Assets page, select Google Cloud from the left-hand navigation to access the Google Cloud assets page:
A screenshot showing the Google Cloud assets page
From the Google Cloud assets page, select arrow next to the New Billing Account button, then select Link Direct Account from the drop-down menu that appears. Finally, select the button a second time (which should now read Link Direct Account).
The CMP will open a modal dialog:
A screenshot showing the Link Direct Account modal dialog
To import your historical billing data into the CMP, enter the following:
  • Billing Account ID — For example 01CF51-9FDB87-7894FD
  • Source Project — Enter the Project ID
  • Source Dataset — Enter the Dataset ID
When you are done, select Import to continue.
The CMP will add a new card to your Google Cloud assets page with import status information:
A screenshot showing the new Google Cloud card with a "Processing" status
A screenshot showing the new Google Cloud card with a "Success" status