README.md
1## Overview
2This file describes an API for a Pub/Sub (Publish/Subscribe) system. This system
3provides a reliable many-to-many communication mechanism between independently
4written publishers and subscribers where the publisher publishes messages to
5*topics* and each subscriber creates a *subscription* and consumes *messages*
6from it.
7
81. The Pub/Sub system maintains bindings between topics and subscriptions.
92. A publisher publishes messages into a topic.
103. The Pub/Sub system delivers messages from topics into attached
11 subscriptions.
124. A subscriber receives pending messages from its subscription and
13 acknowledges each one to the Pub/Sub system.
145. The Pub/Sub system removes acknowledged messages from that subscription.
15
16## Data Model
17The data model consists of the following:
18
19* **Topic**: A topic is a resource to which messages are published by
20 publishers. Topics are named, and the name of the topic is unique within the
21 Pub/Sub system.
22
23* **Subscription**: A subscription records the subscriber's interest in a
24 topic. The Pub/Sub system maintains those messages which still need
25 to be delivered and acknowledged so that they can retried as needed.
26 The set of messages that have not been acknowledged is called the
27 subscription backlog.
28
29* **Message**: A message is a unit of data that flows in the system. It
30 contains opaque data from the publisher along with its *attributes*.
31
32* **Message Attributes** (optional): A set of opaque key-value pairs assigned
33 by the publisher to a message. Attributes are delivered unmodified to
34 subscribers together with the message data, if there's any.
35
36## Publisher Flow
37A publisher publishes messages to the topic using the `Publish` call:
38
39```data
40PubsubMessage message;
41message.set_data("....");
42message.attributes.put("key1", "value1");
43PublishRequest request;
44request.set_topic("topicName");
45request.add_message(message);
46Publisher.Publish(request);
47```
48
49## Subscriber Flow
50The subscriber part of the API is richer than the publisher part and has a
51number of concepts for subscription creation and use:
52
531. A subscriber (user or process) creates a subscription using the
54 `CreateSubscription` call.
55
562. A subscriber receives messages in one of two ways: via pull or push.
57
58 * To receive messages via pull, a subscriber calls the `Pull` method on the
59 `Subscriber` to get messages from the subscription. For each individual
60 message, the subscriber may use the `ack_id` received in the
61 `PullResponse` to `Acknowledge` the message, or modify the *ack deadline*
62 with `ModifyAckDeadline`. See the `Subscription.ack_deadline_seconds`
63 field documentation for details on the ack deadline behavior. Messages
64 must be acknowledged or they will be redelivered in a future `Pull` call.
65
66 **Note:** Messages may be consumed in parallel by multiple processes
67 making `Pull` calls to the same subscription; this will result in the set
68 of messages from the subscription being split among the processes, each
69 process receiving a subset of the messages.
70
71 * To receive messages via push, the `PushConfig` field must be specified in
72 the `Subscription` parameter when creating a subscription, or set with
73 `ModifyPushConfig`. The PushConfig specifies an endpoint at which the
74 subscriber exposes the `PushEndpointService` or some other handler,
75 depending on the endpoint. Messages are received via the
76 `ProcessPushMessage` method. The push subscriber responds to the method
77 with a result code that indicates one of three things: `Acknowledge` (the
78 message has been successfully processed and the Pub/Sub system may delete
79 it), `Nack` (the message has been rejected and the Pub/Sub system should
80 resend it at a later time).
81
82 **Note:** The endpoint may be a load balancer for better scalability, so
83 that multiple processes may handle the message processing load.
84
85Subscription creation:
86
87```data
88Subscription subscription;
89subscription.set_topic("topicName");
90subscription.set_name("subscriptionName");
91subscription.push_config().set_push_endpoint("machinename:8888");
92Subscriber.CreateSubscription(subscription);
93```
94
95Consuming messages via pull:
96
97```data
98// The subscription must be created without setting the push_config field.
99
100PullRequest pull_request;
101pull_request.set_subscription("subscriptionName");
102pull_request.set_return_immediately(false);
103pull_request.set_max_messages(10);
104while (true) {
105 PullResponse pull_response;
106 AcknowledgeRequest ack_request;
107 ackRequest.set_subscription("subscriptionName");
108 if (Subscriber.Pull(pull_request, pull_response) == OK) {
109 for (ReceivedMessage received in pull_response.received_messages()) {
110 Process(received.message().data());
111 ackRequest.add_ack_id(received.ack_id());
112 }
113 }
114 if (ackRequest.ack_ids().size() > 0) {
115 Subscriber.Acknowledge(ack_request);
116 }
117}
118```
119
120## Reliability Semantics
121When a subscriber successfully creates a subscription using
122`Subscriber.CreateSubscription`, it establishes a "subscription point" for
123that subscription, no later than the time that `Subscriber.CreateSubscription`
124returns. The subscriber is guaranteed to receive any message published after
125this subscription point. Note that messages published before the subscription
126point may or may not be delivered.
127
128Messages are not delivered in any particular order by the Pub/Sub system.
129Furthermore, the system guarantees *at-least-once* delivery of each message
130until acknowledged.
131
132## Deletion
133Both topics and subscriptions may be deleted.
134
135When a subscription is deleted, all messages are immediately dropped. If it
136is a pull subscriber, future pull requests will return NOT_FOUND.
137