Google Cloud Pub/Sub for Long-Running Tasks

What is Cloud Pub/Sub?

Data ingestion is the foundation for any analytics and machine learning based application. If you are building a stream, batch, or unified pipelines, then Cloud Pub/Sub provides a simple and reliable location for your event data on its journey towards processing, storage, and analysis. By providing many-to-many, asynchronous messaging that decouples senders and receivers, cloud pub/sub allows for secure and highly available communication among independently written applications. Cloud Pub/Sub delivers low-latency, durable messaging which helps developers quickly integrate applications hosted on the Google Cloud Platform and those present externally.

The key features of Cloud Pub/Sub :

Delivery System: Synchronization, cross-zone message replication and per-message receipt tracking, which ensures at-least-once delivery at any scale.

Global by default: Allows to publish from anywhere in the world and is accessible from anywhere, with low latency without any replication.

Security & Compliance: Cloud Pub/Sub is a HIPAA (Health Insurance Portability and Accountability Act ) compliant service, which is offering fine-grained access controls and end-to-end encryption system for the best security 

Integration: Cloud Pub/Sub can be integrated with multiple services, such as Cloud Storage and Gmail update events and Cloud Functions for serverless event-driven computing

Publisher Application: 

The work of a publisher application is to create and send messages to a topic and Subscriber applications create a subscription to a topic in order to receive messages from it. 

Types of communication :

.(1) One-to-many (fan-out)

 (2)many-to-one (fan-in), 

 (3) many-to-many.

Cloud Pub/Sub message flow:

This shows an overview of the components in the Cloud Pub/Sub System and how the messages flow between them. A publisher application creates a topic in Cloud Pub/Sub service and sends messages to the topic.  A message contains a payload and attributes (optional) that describe the payload content.

1. Messages are preserved in a message store until they are delivered and acknowledged by subscribers.

2. Cloud Pub/Sub forwards messages from a topic to all of its subscriptions one by one. Each subscription receives messages by Cloud Pub/Sub, pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service.

3. The subscriber receives the pending messages from its subscription and also acknowledges each one to the Cloud Pub/Sub service.

4. When a message is acknowledged by a subscriber, it is removed from the subscription's message queue.