1# Block and Transaction Broadcasting with ZeroMQ
2
3[ZeroMQ](https://zeromq.org/) is a lightweight wrapper around TCP
4connections, inter-process communication, and shared-memory,
5providing various message-oriented semantics such as publish/subscribe,
6request/reply, and push/pull.
7
8The Bitcoin Core daemon can be configured to act as a trusted "border
9router", implementing the bitcoin wire protocol and relay, making
10consensus decisions, maintaining the local blockchain database,
11broadcasting locally generated transactions into the network, and
12providing a queryable RPC interface to interact on a polled basis for
13requesting blockchain related data. However, there exists only a
14limited service to notify external software of events like the arrival
15of new blocks or transactions.
16
17The ZeroMQ facility implements a notification interface through a set
18of specific notifiers. Currently there are notifiers that publish
19blocks and transactions. This read-only facility requires only the
20connection of a corresponding ZeroMQ subscriber port in receiving
21software; it is not authenticated nor is there any two-way protocol
22involvement. Therefore, subscribers should validate the received data
23since it may be out of date, incomplete or even invalid.
24
25ZeroMQ sockets are self-connecting and self-healing; that is,
26connections made between two endpoints will be automatically restored
27after an outage, and either end may be freely started or stopped in
28any order.
29
30Because ZeroMQ is message oriented, subscribers receive transactions
31and blocks all-at-once and do not need to implement any sort of
32buffering or reassembly.
33
34## Prerequisites
35
36The ZeroMQ feature in Bitcoin Core requires the ZeroMQ API >= 4.0.0
37[libzmq](https://github.com/zeromq/libzmq/releases).
38For version information, see [dependencies.md](dependencies.md).
39Typically, it is packaged by distributions as something like
40*libzmq3-dev*. The C++ wrapper for ZeroMQ is *not* needed.
41
42In order to run the example Python client scripts in the `contrib/zmq/`
43directory, one must also install [PyZMQ](https://github.com/zeromq/pyzmq)
44(generally with `pip install pyzmq`), though this is not necessary for daemon
45operation.
46
47## Enabling
48
49By default, the ZeroMQ feature is not automatically compiled.
50To enable, use `-DWITH_ZMQ=ON` when configuring the build system:
51
52    $ cmake -B build -DWITH_ZMQ=ON
53
54To actually enable operation, one must set the appropriate options on
55the command line or in the configuration file.
56
57## Usage
58
59Currently, the following notifications are supported:
60
61    -zmqpubhashtx=address
62    -zmqpubhashblock=address
63    -zmqpubrawblock=address
64    -zmqpubrawtx=address
65    -zmqpubsequence=address
66
67The socket type is PUB and the address must be a valid ZeroMQ socket
68address. The same address can be used in more than one notification.
69The same notification can be specified more than once.
70
71The option to set the PUB socket's outbound message high water mark
72(SNDHWM) may be set individually for each notification:
73
74    -zmqpubhashtxhwm=n
75    -zmqpubhashblockhwm=n
76    -zmqpubrawblockhwm=n
77    -zmqpubrawtxhwm=n
78    -zmqpubsequencehwm=n
79
80The high water mark value must be an integer greater than or equal to 0.
81
82For instance:
83
84    $ bitcoind -zmqpubhashtx=tcp://127.0.0.1:28332 \
85               -zmqpubhashtx=tcp://192.168.1.2:28332 \
86               -zmqpubhashblock="tcp://[::1]:28333" \
87               -zmqpubrawtx=unix:/tmp/bitcoind.tx.raw \
88               -zmqpubhashtxhwm=10000
89
90`bitcoin node` or `bitcoin gui` can also be substituted for `bitcoind`.
91
92Notification types correspond to message topics (details in next section). For instance,
93for the notification `-zmqpubhashtx` the topic is `hashtx`. These options can also be
94provided in bitcoin.conf.
95
96### Message format
97
98All ZMQ messages share the same structure with three parts: _topic_ string,
99message _body_, and _message sequence number_:
100
101    | topic     | body                                                 | message sequence number  |
102    |-----------+------------------------------------------------------+--------------------------|
103    | rawtx     | <serialized transaction>                             | <4-byte LE uint>         |
104    | hashtx    | <reversed 32-byte transaction hash>                  | <4-byte LE uint>         |
105    | rawblock  | <serialized block>                                   | <4-byte LE uint>         |
106    | hashblock | <reversed 32-byte block hash>                        | <4-byte LE uint>         |
107    | sequence  | <reversed 32-byte block hash>C                       | <4-byte LE uint>         |
108    | sequence  | <reversed 32-byte block hash>D                       | <4-byte LE uint>         |
109    | sequence  | <reversed 32-byte transaction hash>R<8-byte LE uint> | <4-byte LE uint>         |
110    | sequence  | <reversed 32-byte transaction hash>A<8-byte LE uint> | <4-byte LE uint>         |
111
112where:
113
114 - message sequence number represents message count to detect lost messages, distinct for each topic
115 - all transaction and block hashes are in _reversed byte order_ (i. e. with bytes
116   produced by hashing function reversed), the same format as the RPC interface and block
117   explorers use to display transaction and block hashes
118
119#### rawtx
120
121Notifies about all transactions, both when they are added to mempool or when a new block
122arrives. This means a transaction could be published multiple times: first when it enters
123mempool and then again in each block that includes it. The body part of the message is the
124serialized transaction.
125
126#### hashtx
127
128Notifies about all transactions, both when they are added to mempool or when a new block
129arrives. This means a transaction could be published multiple times: first when it enters
130mempool and then again in each block that includes it. The body part of the message is the
13132-byte transaction hash in reversed byte order.
132
133#### rawblock
134
135Notifies when the chain tip is updated. When assumeutxo is in use, this notification will
136not be issued for historical blocks connected to the background validation chainstate. The
137body part of the message is the serialized block.
138
139#### hashblock
140
141Notifies when the chain tip is updated. When assumeutxo is in use, this notification will
142not be issued for historical blocks connected to the background validation chainstate. The
143body part of the message is the 32-byte block hash in reversed byte order.
144
145#### sequence
146
147The 8-byte LE uints correspond to _mempool sequence number_ and the types of bodies are:
148
149   - `C` : block with this hash connected
150   - `D` : block with this hash disconnected
151   - `R` : transaction with this hash removed from mempool for non-block inclusion reason
152   - `A` : transaction with this hash added to mempool
153
154### Implementing ZMQ client
155
156ZeroMQ endpoint specifiers for TCP (and others) are documented in the
157[ZeroMQ API](https://libzmq.readthedocs.io/en/zeromq4-x/).
158
159Client side, then, the ZeroMQ subscriber socket must have the
160ZMQ_SUBSCRIBE option set to one or either of these prefixes (for
161instance, just `hash`); without doing so will result in no messages
162arriving. Please see [`contrib/zmq/zmq_sub.py`](/contrib/zmq/zmq_sub.py) for a working example.
163
164The ZMQ_PUB socket's ZMQ_TCP_KEEPALIVE option is enabled. This means that
165the underlying SO_KEEPALIVE option is enabled when using a TCP transport.
166The effective TCP keepalive values are managed through the underlying
167operating system configuration and must be configured prior to connection establishment.
168
169For example, when running on GNU/Linux, one might use the following
170to lower the keepalive setting to 10 minutes:
171
172    sudo sysctl -w net.ipv4.tcp_keepalive_time=600
173
174Setting the keepalive values appropriately for your operating environment may
175improve connectivity in situations where long-lived connections are silently
176dropped by network middle boxes.
177
178Also, the socket's ZMQ_IPV6 option is enabled to accept connections from IPv6
179hosts as well. If needed, this option has to be set on the client side too.
180
181## Remarks
182
183From the perspective of bitcoind, the ZeroMQ socket is write-only; PUB
184sockets don't even have a read function. Thus, there is no state
185introduced into bitcoind directly. Furthermore, no information is
186broadcast that wasn't already received from the public P2P network.
187
188No authentication or authorization is done on connecting clients; it
189is assumed that the ZeroMQ port is exposed only to trusted entities,
190using other means such as firewalling.
191
192Note that for `*block` topics, when the block chain tip changes,
193a reorganisation may occur and just the tip will be notified.
194It is up to the subscriber to retrieve the chain from the last known
195block to the new tip. Also note that no notification will occur if the tip
196was in the active chain, as would be the case after calling the `invalidateblock` RPC.
197In contrast, the `sequence` topic publishes all block connections and
198disconnections.
199
200There are several possibilities that ZMQ notification can get lost
201during transmission depending on the communication type you are
202using. Bitcoind appends an up-counting sequence number to each
203notification which allows listeners to detect lost notifications.
204
205The `sequence` topic refers specifically to the mempool sequence
206number, which is also published along with all mempool events. This
207is a different sequence value than in ZMQ itself in order to allow a total
208ordering of mempool events to be constructed.
209