Importing historical data
This feature enables you to upload all your historical Google Cloud billing data into DoiT Cloud Management Platform (CMP) to immediately analyze your cloud spend and promptly identify optimization opportunities through Cloud Analytics, BigQuery Lens, and other CMP features.
To get started, first, you need to ensure that you have a Service Account that connects your Google Cloud Organization to CMP. Then, you'll have to grant two additional required permissions.
We import your historical Cloud Billing data from Google BigQuery. To start collecting your Google Cloud Billing data, you must enable Cloud Billing data export to BigQuery.
You can only import as much historical Cloud Billing data as you already have available in BigQuery.
Suppose you enable Cloud Billing data export to BigQuery for the first time while following the steps in this document. In that case, you will only have around one month of historical Cloud Billing data to import into the CMP.
Identify the service account
Within the CMP, select the gear icon (Settings) from the top navigation bar, then select Google Cloud under Cloud settings. The CMP will take you to the Google Cloud settings screen:
If you do not have a service account configured yet, you will need to create a service account before you can continue.
Select your desired service account, copy the service account email, and proceed to grant the required permissions (see next section).
Grant the required Permissions
You need to have the appropriate IAM role/permission in GCP to grant the required permissions to the service account.
BigQuery Data Viewer role
- From GCP, navigate to the BigQuery Console and open the dataset where the billing data is stored, then click "Share dataset"
- In the Dataset Permissions tab, click "Add Member" and paste the service account email that was linked to CMP as identified in Step 1.
- Click "Select a Role" highlight BigQuery then choose BigQuery Data Viewer
- Click "Add"
- Click "Done"
Grant the bigquery.jobs.create permission
Depending on the features initially enabled from the process Connect Google Cloud Service Account, you may already have this permission granted to the role. Follow the steps below to verify / add the permission to the project.
- From the GCP console, select your organization from the dropdown options
- On the navigation menu, go to "IAM & Admin" then select "Roles"
- Click on the role "DoiT CMP Service Account"
- Check if bigquery.jobs.create is included in the assigned permissions, if yes, no further action is needed since these permissions will be inherited by the projects
- If bigquery.jobs.create permission is not yet added, click on "Edit Role"
- Click "Add Permissions"
- Filter by "Permission: bigquery.jobs.create" and check the box for this permission
- Click "Add"
Link your Google Cloud Direct Billing Account
Required Permission: Assets Manager
Within the CMP, select Billing from the top navigation bar and then select Assets. The CMP will take you to the Assets page.
From the Assets page, select Google Cloud from the left-hand navigation to access the Google Cloud assets page:
From the Google Cloud assets page, select arrow next to the New Billing Account button, then select Link Direct Account from the drop-down menu that appears. Finally, select the button a second time (which should now read Link Direct Account).
The CMP will open a modal dialog:
To import your historical billing data into the CMP, enter the following:
- Billing Account ID — For example
- Source Project — Enter the Project ID
- Source Dataset — Enter the Dataset ID
When you are done, select Import to continue.
The CMP will add a new card to your Google Cloud assets page with import status information: