How-To Guides
Integration Guides

Google Cloud Pub/Sub Integration Guide

21min

Review the following guide for setting up an integration between Manufacturing Connect Edge and Google Pub/Sub service.

Once the integration is set up, you can use it for the following:

  • Publishing data from a topic in your edge device to a Subscription topic in the Google Cloud Platform
  • Subscribing to data published by a Publication topic in the Google Cloud Platform

Note: You can use the following authentication methods to configure the Google Cloud Pub/Sub connector:

  • Using GCP Auth type Service Account Key
  • Using GCP Auth type Workload Identity Federation

Before You Begin

Service Account Key Credentials

You have two options for configuring the service account key parameter in the Google Pub/Sub connector: GCP SA Key Authentication and GCP Workload identity federation (OIDC).

Service Account Keys

Each Google service account is associated with a public/private RSA key pair. The Service Account Credentials API uses this internal key pair to create short-lived service account credentials, and to sign blobs and JSON Web Tokens (JWTs). This key pair is known as the Google-managed key pair.

In addition, you can create multiple public/private RSA key pairs, known as user-managed key pairs, and use the private key to authenticate with Google APIs. This private key is known as a service account key.

See Service account keys to learn more.

Workload Identity Federation

Workload Identity Federation allows you to can grant on-premises or multi-cloud workloads access to Google Cloud resources without using a service account key. You may select to use this option because service account keys are powerful credentials, so they can present a security risk if they are not managed correctly.

With identity federation, you can use Identity and Access Management (IAM) to grant external identities IAM roles, including the ability to impersonate service accounts. This approach eliminates the maintenance and security burden associated with service account keys.

Workload Identity Pools

A workload identity pool is an entity that lets you manage external identities. You will review and have the option to customize this parameter when setting up these credentials.

Workload Identity Pool Providers

A workload identity pool provider is the entity that describes the relationship between Google Cloud and your identity provider (IdP).

Workload identity federation follows the OAuth 2.0 token exchange specification. You provide a credential from your IdP to the Security Token Service, which verifies the identity on the credential, and then returns a federated token in exchange.

See the following to learn more:

Set up the Outbound Connection (Publish to Google Pub/Sub)

Follow the steps below to set up the outbound connection.

Step 1: Create Publication Topic in Google Cloud Platform

In the Google Cloud Platform, create a publication topic. A matching subscription topic is created automatically, with the -sub suffix appended to the topic name.

Step 2: Add Device



Step 3: Add Tags



Step 4: Add Connector

Follow the steps to Add a Connector and select the Google Cloud Pub/Sub Connector provider.

For more information about message publication settings, see the SDK on Publish Settings.

Configure the following parameters.

  • Name: Enter a name for the connector.
  • Service Account Key (.json): Create a service account key in your Google Cloud Platform in JSON format. Copy or save all the content from the JSON file and paste or upload it here.
  • The project ID of the cloud project: Copy the ID from your Google Cloud Platform and paste it here.
  • The private key ID of the cloud project: Copy the key ID from your Google Cloud Platform and paste it here.
  • The client email of the cloud project: Enter the email from your Google Cloud Platform.
  • Integration Topic: Copy the name of the Publication topic (without the "-sub" suffix) from your Google Cloud Platform and paste it here.
  • Custom Attributes: You can add custom attributes in key/value pairs for further data processing. Refer to the following to learn more:
  • Parallel Publish Count: The number of messages being published simultaneously. The default value is 100.
  • Parallel byte threshold: The minimum size of a batch (in bytes) for the batch to be published. The default value is zero, which means that there is no threshold (limit).
  • Publish count threshold: The minimum number of messages in a batch for the batch to be published. The default value is zero, which means that there is no threshold (limit).
  • Publish delay threshold (Milliseconds): The maximum time that the client will attempt to publish a batch of messages. The default value is zero, which means that there is no threshold (limit).
  • Throttling limit: The maximum number of messages per second to be processed. The default value is zero, which means that there is no limit.
  • Persistent storage: When enabled, this will cause messages to undergo a store-and-forward procedure. Messages will be stored within Manufacturing Connect Edge when cloud providers are online.
  • Queue Mode: Select the queue mode as lifo (last in first out) or fifo (first in first out). Selecting lifo means that the last data entry is processed first, and selecting fifo means the first data entry is processed first.

Step 5: Enable the Connector

After adding the connector, click the toggle in the connector tile to enable it.

Toggle to enable connector
Toggle to enable connector


If you see a Failed status, you can review the Connector Logs and relevant error messages.

Step 6: Create Outbound Topics for Connector

You will now need to import the tags you added in Step 2 to the connector as topics.



After importing the tag(s), do the following:

  • Edit the tag and configure the Remote Data Topic. Copy and paste the name of the Subscription topic  (with the -sub suffix) from your Google Cloud Platform.
  • Make sure the connector has a CONNECTED status.

Step 7: Enable Topics

Because you imported DeviceHub tags for a CONNECTED connector, all topics will be disabled.

To enable the topics, return to the Topics tab and click the Enable all topics icon.

Document image


Step 8: Verify Connection in Google Cloud Platform

To verify the connection in Google Cloud Platform:

  1. Pull the subscription topic to see messages it receives from the Manufacturing Connect Edge outbound topics created previously.

    List of Manufacturing Connect Edge outbound topics
    List of Manufacturing Connect Edge outbound topics
    
  2. View the subscription statistics.

    Graph of subscription statistic
    Graph of subscription statistic
    

Set up the Inbound Connection (Subscribe to Google Pub/Sub)

Follow the steps below to set up the inbound connection.

See Publish messages to topics to learn more about publishing messages in Google Pub/Sub.

Step 1: Create Publication Topic in Google Cloud Platform

In the Google Cloud Platform, create a publication topic.

Step 2: Add Connector

Follow the steps to Add a Connector and select the Google Cloud Pub/Sub Connector provider.

Configure the following parameters.

  • Name: Enter a name for the connector.
  • Service Account Key (.json): Create a service account key in your Google Cloud Platform in JSON format. Copy or save all the content from the JSON file and paste or upload it here.
  • The project ID of the cloud project: Copy the ID from your Google Cloud Platform and paste it here.
  • The private key ID of the cloud project: Copy the key ID from your Google Cloud Platform and paste it here.
  • The client email of the cloud project: Enter the email from your Google Cloud Platform.
  • Integration Topic: Copy the name of the Publication topic from your Google Cloud Platform and paste it here.
  • Custom Attributes: You can add custom attributes in key/value pairs for further data processing. Refer to the following to learn more:
  • Parallel Publish Count: The number of messages being published simultaneously. The default value is 100.
  • Parallel byte threshold: The minimum size of a batch (in bytes) for the batch to be published. The default value is zero, which means that there is no threshold (limit).
  • Publish count threshold: The minimum number of messages in a batch for the batch to be published. The default value is zero, which means that there is no threshold (limit).
  • Publish delay threshold (Milliseconds): The maximum time that the client will attempt to publish a batch of messages. The default value is zero, which means that there is no threshold (limit).
  • Throttling limit: The maximum number of messages per second to be processed. The default value is zero, which means that there is no limit.
  • Persistent storage: When enabled, this will cause messages to undergo a store-and-forward procedure. Messages will be stored within Manufacturing Connect Edge when cloud providers are online.
  • Queue Mode: Select the queue mode as lifo (last in first out) or fifo (first in first out). Selecting lifo means that the last data entry is processed first, and selecting fifo means the first data entry is processed first.

Step 3: Enable the Connector

After adding the connector, click the toggle in the connector tile to enable it.

Toggle to enable connector
Toggle to enable connector


If you see a Failed status, you can review the Connector Logs and relevant error messages.

Step 4: Create Inbound Topics for Connector

You will now need to create a topic in Manufacturing Connect Edge from the Google Cloud Platform publication topic created in Step 1.

To create inbound topics:

  1. Navigate to Integration.
  2. Click the connector tile.
  3. Click the Topics tab.
  4. Click the Add a new subscription icon. The Data Integration dialog box appears.

    Document image
    
  5. Configure the following parameters.
    • Data Direction: Select Remote to Local - Inbound.
    • Local Data Topic: Enter a name for the topic name in Manufacturing Connect Edge.
    • Remote Data Topic: Copy and paste the Publication topic from your Google Cloud Platform.
    • Enable: Select the toggle to enable the topic.
  6. Click Yes to add the topic.
  7. From the connector tile, ensure the connector is not disabled and still shows a CONNECTED status. Also verify the topic shows an Enabled status.
Document image


Step 5: Send Messages in Google Cloud Platform

Start sending messages through the publication topic (created in Step 1) from the Google Cloud Platform. See Publish messages to topics to learn more.

Step 9: Verify Connection in Manufacturing Connect Edge

You can do one of the following to verify the connection in Manufacturing Connect Edge. When configuring the Flow or application, use the the Local Data Topic name configured in Step 4.

  • Create a Flow to view the messages coming from Google Pub/Sub through the connector you created.
  • Visualize the incoming data using one of the dedicated Applications.