1# User-space, Statically Defined Tracing (USDT) for Bitcoin Core
2
3Bitcoin Core includes statically defined tracepoints to allow for more
4observability during development, debugging, code review, and production usage.
5These tracepoints make it possible to keep track of custom statistics and
6enable detailed monitoring of otherwise hidden internals. They have
7little to no performance impact when unused.
8
9```
10eBPF and USDT Overview
11======================
12
13                ┌──────────────────┐            ┌──────────────┐
14                │ tracing script   │            │ bitcoind     │
15                │==================│      2.    │==============│
16                │  eBPF  │ tracing │      hooks │              │
17                │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
18                └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
19            1.       │      │ 4.              │ │ ...          │   │ to eBPF
20    User    compiles │      │ pass data to    │ └──────────────┘   │ program
21    Space    & loads │      │ tracing script  │                    │
22    ─────────────────┼──────┼─────────────────┼────────────────────┼───
23    Kernel           │      │                 │                    │
24    Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
25                │  │  eBPF program                         │◄──────┘
26                │  └───────────────────────────────────────┤
27                │ eBPF kernel Virtual Machine (sandboxed)  │
28                └──────────────────────────────────────────┘
29
301. The tracing script compiles the eBPF code and loads the eBPF program into a kernel VM
312. The eBPF program hooks into one or more tracepoints
323. When the tracepoint is called, the arguments are passed to the eBPF program
334. The eBPF program processes the arguments and returns data to the tracing script
34```
35
36The Linux kernel can hook into the tracepoints during runtime and pass data to
37sandboxed [eBPF] programs running in the kernel. These eBPF programs can, for
38example, collect statistics or pass data back to user-space scripts for further
39processing.
40
41[eBPF]: https://ebpf.io/
42
43The two main eBPF front-ends with support for USDT are [bpftrace] and
44[BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and
45`bpftrace` is preferred for one-liners and shorter scripts. Examples for both can
46be found in [contrib/tracing].
47
48[bpftrace]: https://github.com/bpftrace/bpftrace
49[BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc
50[contrib/tracing]: ../contrib/tracing/
51
52## Tracepoint documentation
53
54The currently available tracepoints are listed here.
55
56### Context `net`
57
58[^address-length]: An Onion v3 address with a `:` and a five digit port has 68
59  chars. However, addresses of peers added with host names might be longer.
60
61#### Tracepoint `net:inbound_message`
62
63Is called when a message is received from a peer over the P2P network. Passes
64information about our peer, the connection and the message as arguments.
65
66Arguments passed:
671. Peer ID as `int64`
682. Peer Address and Port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
693. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
704. Message Type (inv, ping, getdata, addrv2, ...) as `pointer to C-style String` (max. length 20 characters)
715. Message Size in bytes as `uint64`
726. Message Bytes as `pointer to unsigned chars` (i.e. bytes)
73
74Note: The message is passed to the tracepoint in full, however, due to space
75limitations in the eBPF kernel VM it might not be possible to pass the message
76to user-space in full. Messages longer than 32kb might be cut off. This can
77be detected in tracing scripts by comparing the message size to the length of
78the passed message.
79
80#### Tracepoint `net:outbound_message`
81
82Is called when a message is sent to a peer over the P2P network. Passes
83information about our peer, the connection and the message as arguments.
84
85Arguments passed:
861. Peer ID as `int64`
872. Peer Address and Port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
883. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
894. Message Type (inv, ping, getdata, addrv2, ...) as `pointer to C-style String` (max. length 20 characters)
905. Message Size in bytes as `uint64`
916. Message Bytes as `pointer to unsigned chars` (i.e. bytes)
92
93Note: The message is passed to the tracepoint in full, however, due to space
94limitations in the eBPF kernel VM it might not be possible to pass the message
95to user-space in full. Messages longer than 32kb might be cut off. This can
96be detected in tracing scripts by comparing the message size to the length of
97the passed message.
98
99#### Tracepoint `net:inbound_connection`
100
101Is called when a new inbound connection is opened to us. Passes information about
102the peer and the number of inbound connections including the newly opened connection.
103
104Arguments passed:
1051. Peer ID as `int64`
1062. Peer address and port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
1073. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
1084. Network the peer connects from as `uint32` (1 = IPv4, 2 = IPv6, 3 = Onion, 4 = I2P, 5 = CJDNS). See `Network` enum in `netaddress.h`.
1095. Number of existing inbound connections as `uint64` including the newly opened inbound connection.
110
111#### Tracepoint `net:outbound_connection`
112
113Is called when a new outbound connection is opened by us. Passes information about
114the peer and the number of outbound connections including the newly opened connection.
115
116Arguments passed:
1171. Peer ID as `int64`
1182. Peer address and port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
1193. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
1204. Network of the peer as `uint32` (1 = IPv4, 2 = IPv6, 3 = Onion, 4 = I2P, 5 = CJDNS). See `Network` enum in `netaddress.h`.
1215. Number of existing outbound connections as `uint64` including the newly opened outbound connection.
122
123#### Tracepoint `net:evicted_inbound_connection`
124
125Is called when an inbound connection is evicted by us. Passes information about the evicted peer and the time at connection establishment.
126
127Arguments passed:
1281. Peer ID as `int64`
1292. Peer address and port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
1303. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
1314. Network the peer connects from as `uint32` (1 = IPv4, 2 = IPv6, 3 = Onion, 4 = I2P, 5 = CJDNS). See `Network` enum in `netaddress.h`.
1325. Connection established UNIX epoch timestamp in seconds as `uint64`.
133
134#### Tracepoint `net:misbehaving_connection`
135
136Is called when a connection is misbehaving. Passes the peer id and a
137reason for the peers misbehavior.
138
139Arguments passed:
1401. Peer ID as `int64`.
1412. Reason why the peer is misbehaving as `pointer to C-style String` (max. length 128 characters).
142
143#### Tracepoint `net:closed_connection`
144
145Is called when a connection is closed. Passes information about the closed peer
146and the time at connection establishment.
147
148Arguments passed:
1491. Peer ID as `int64`
1502. Peer address and port (IPv4, IPv6, Tor v3, I2P, ...) as `pointer to C-style String` (normally up to 68 characters[^address-length])
1513. Connection Type (inbound, feeler, outbound-full-relay, ...) as `pointer to C-style String` (max. length 20 characters)
1524. Network the peer connects from as `uint32` (1 = IPv4, 2 = IPv6, 3 = Onion, 4 = I2P, 5 = CJDNS). See `Network` enum in `netaddress.h`.
1535. Connection established UNIX epoch timestamp in seconds as `uint64`.
154
155### Context `validation`
156
157#### Tracepoint `validation:block_connected`
158
159Is called *after* a block is connected to the chain. Can, for example, be used
160to benchmark block connections together with `-reindex`.
161
162Arguments passed:
1631. Block Header Hash as `pointer to unsigned chars` (i.e. 32 bytes in little-endian)
1642. Block Height as `int32`
1653. Transactions in the Block as `uint64`
1664. Inputs spend in the Block as `int32`
1675. SigOps in the Block (excluding coinbase SigOps) `uint64`
1686. Time it took to connect the Block in nanoseconds (ns) as `uint64`
169
170### Context `utxocache`
171
172The following tracepoints cover the in-memory UTXO cache. UTXOs are, for example,
173added to and removed (spent) from the cache when we connect a new block.
174**Note**: Bitcoin Core uses temporary clones of the _main_ UTXO cache
175(`chainstate.CoinsTip()`). For example, the RPCs `generateblock` and
176`getblocktemplate` call `TestBlockValidity()`, which applies the UTXO set
177changes to a temporary cache. Similarly, mempool consistency checks, which are
178frequent on regtest, also apply the UTXO set changes to a temporary cache.
179Changes to the _main_ UTXO cache and to temporary caches trigger the tracepoints.
180We can't tell if a temporary cache or the _main_ cache was changed.
181
182#### Tracepoint `utxocache:flush`
183
184Is called *after* the in-memory UTXO cache is flushed.
185
186Arguments passed:
1871. Time it took to flush the cache microseconds as `int64`
1882. Flush state mode as `uint32`. It's an enumerator class with values
189   `0` (`NONE`), `1` (`IF_NEEDED`), `2` (`PERIODIC`), `3` (`FORCE_FLUSH`), `4` (`FORCE_SYNC`)
1903. Cache size (number of coins) before the flush as `uint64`
1914. Cache memory usage in bytes as `uint64`
1925. If pruning caused the flush as `bool`
193
194#### Tracepoint `utxocache:add`
195
196Is called when a coin is added to a UTXO cache. This can be a temporary UTXO cache too.
197
198Arguments passed:
1991. Transaction ID (hash) as `pointer to unsigned chars` (i.e. 32 bytes in little-endian)
2002. Output index as `uint32`
2013. Block height the coin was added to the UTXO-set as  `uint32`
2024. Value of the coin as `int64`
2035. If the coin is a coinbase as `bool`
204
205#### Tracepoint `utxocache:spent`
206
207Is called when a coin is spent from a UTXO cache. This can be a temporary UTXO cache too.
208
209Arguments passed:
2101. Transaction ID (hash) as `pointer to unsigned chars` (i.e. 32 bytes in little-endian)
2112. Output index as `uint32`
2123. Block height the coin was spent, as `uint32`
2134. Value of the coin as `int64`
2145. If the coin is a coinbase as `bool`
215
216#### Tracepoint `utxocache:uncache`
217
218Is called when a coin is purposefully unloaded from a UTXO cache. This
219happens, for example, when we load an UTXO into a cache when trying to accept
220a transaction that turns out to be invalid. The loaded UTXO is uncached to avoid
221filling our UTXO cache up with irrelevant UTXOs.
222
223Arguments passed:
2241. Transaction ID (hash) as `pointer to unsigned chars` (i.e. 32 bytes in little-endian)
2252. Output index as `uint32`
2263. Block height the coin was uncached, as `uint32`
2274. Value of the coin as `int64`
2285. If the coin is a coinbase as `bool`
229
230### Context `coin_selection`
231
232#### Tracepoint `coin_selection:selected_coins`
233
234Is called when `SelectCoins` completes.
235
236Arguments passed:
2371. Wallet name as `pointer to C-style string`
2382. Coin selection algorithm name as `pointer to C-style string`
2393. Selection target value as `int64`
2404. Calculated waste metric of the solution as `int64`
2415. Total value of the selected inputs as `int64`
242
243#### Tracepoint `coin_selection:normal_create_tx_internal`
244
245Is called when the first `CreateTransactionInternal` completes.
246
247Arguments passed:
2481. Wallet name as `pointer to C-style string`
2492. Whether `CreateTransactionInternal` succeeded as `bool`
2503. The expected transaction fee as an `int64`