Inside Bitcoin

1# libmultiprocess Design
2
3Given an interface description of an object with one or more methods, libmultiprocess generates:
4
5* A C++ `ProxyClient` class template specialization with an implementation of each interface method that sends a request over a socket, waits for a response, and returns the result.
6* A C++ `ProxyServer` class template specialization that listens for requests over a socket and calls a wrapped C++ object implementing the same interface to actually execute the requests.
7
8The function call ⇆ request translation supports input and output arguments, standard types like `unique_ptr`, `vector`, `map`, and `optional`, and bidirectional calls between processes through interface pointer and `std::function` arguments.
9
10If the wrapped C++ object inherits from an abstract base class declaring virtual methods, the generated `ProxyClient` objects can inherit from the same class, allowing interprocess calls to replace local calls without changes to existing code.
11
12There is also optional support for thread mapping, so each thread making interprocess calls can have a dedicated thread processing requests from it, and callbacks from processing threads are executed on corresponding request threads (so recursive mutexes and thread names function as expected in callbacks).
13
14Libmultiprocess acts as a pure wrapper or layer over the underlying protocol. Clients and servers written in other languages, but using a shared capnproto schema can communicate with interprocess counterparties using libmultiprocess without having to use libmultiprocess themselves or having to know about the implementation details of libmultiprocess.
15
16## Core Architecture
17
18The `ProxyServer` generated class is not directly exposed to the user. The `ProxyClient` generated class inherits from the C++ interface class, so user code interacts with it through the abstract interface type, and the `ProxyClient` type itself is generally not visible or accessible without a cast. `ConnectStream` returns a `unique_ptr<ProxyClient<InitInterface>>` as an exception for the root Init interface, but even there users typically treat it as a pointer to the abstract `InitInterface`. For all interfaces returned by Init methods (e.g., `Printer`, `Calculator`), the return type is the abstract class pointer, hiding the underlying `ProxyClient` entirely. They are first instantiated when calling `ConnectStream` and `ServeStream` respectively for creating the `InitInterface`. These methods establish connections through sockets, internally creating `Connection` objects wrapping a `capnp::RpcSystem` configured for client and server mode respectively.
19
20The `InitInterface` interface will typically have methods which return other interfaces, giving the connecting process the ability to call other functions in the serving process. Interfaces can also have methods accepting other interfaces as parameters, giving serving processes the ability to call back and invoke functions in connecting processes. Creating new interfaces does not create new connections, and typically many interface objects will share the same connection.
21
22Both `ConnectStream` and `ServeStream` also require an instantiation of the `EventLoop`. The `EventLoop` owns pending requests, notifies on request dispatch, allows clients from multiple threads to make synchronous calls, and handles some cleanup routines on exit. It must be run in a separate thread so it is always active and can process incoming requests from local clients and remote connections.
23
24When a generated method on the `ProxyClient` is called, it calls `clientInvoke` with the capnp-translated types. `clientInvoke` creates a self-executing promise (`kj::TaskSet`) that drives the execution of the request and gives ownership of it to the `EventLoop`. `clientInvoke` blocks until a response is received, or until there is a call from the server that needs to run on the same client thread, using a `Waiter` object.
25
26On the server side, the `capnp::RpcSystem` receives the capnp request and invokes the corresponding C++ method through the corresponding `ProxyServer` and the heavily templated `serverInvoke` triggering a `ServerCall`. The return values from the actual C++ methods are copied into capnp responses by `ServerRet` and exceptions are caught and copied by `ServerExcept`. The two are connected through `ServerField`. The main method driving execution of a request is `PassField`, which is invoked through `ServerField`. Instantiated interfaces, or capabilities in capnp speak, are tracked and owned by the server's `capnp::RpcSystem`.
27
28## Request and Response Flow
29
30Method parameters and return values are serialized using Cap'n Proto's Builder objects (for sending) and Reader objects (for receiving). Input parameters flow from the client to the server, while output parameters (return values) flow back from the server to the client.
31
32```mermaid
33sequenceDiagram
34    participant clientInvoke
35    participant BuildField as BuildField<br/>(Client)
36    participant ReadField_C as ReadField<br/>(Client)
37    participant Request as Request<br/>message
38    participant serverInvoke
39    participant ReadField as ReadField<br/>(Server)
40    participant BuildField_S as BuildField<br/>(Server)
41    participant Response as Response<br/>message
42
43    Note over clientInvoke,ReadField: Input Parameter Flow
44    clientInvoke->>BuildField: BuildField(input_arg)
45    BuildField->>Request: Serialize input
46    Request->>serverInvoke: Cap'n Proto message
47    serverInvoke->>ReadField: Deserialize input
48
49    Note over clientInvoke,Response: Output Parameter Flow
50    serverInvoke-->>BuildField_S: BuildField(output)
51    BuildField_S-->Response: Serialize output
52    Response-->>ReadField_C: Cap'n Proto message
53    ReadField_C-->>clientInvoke: Deserialize output
54```
55
56### Detailed Serialization Mechanism
57
58Parameters are represented as Fields that must be set on Cap'n Proto Builder objects (for sending) and read from Reader objects (for receiving).
59
60#### Building Fields
61
62`BuildField` uses a generated parameter `Accessor` to set the appropriate field in the Cap'n Proto Builder object.
63
64```mermaid
65sequenceDiagram
66    participant clientInvoke as clientInvoke or<br/>serverInvoke
67    participant BuildField
68    participant Accessor
69    participant Builder as Params::Builder
70
71    Note over clientInvoke,Builder: Serializing Parameters
72    clientInvoke->>BuildField: BuildField(param1)
73    BuildField->>Accessor: Use generated field accessor
74    Accessor->>Builder: builder.setField1(param1)
75
76    clientInvoke->>BuildField: BuildField(param2)
77    BuildField->>Accessor: Use generated field Accessor
78    Accessor->>Builder: builder.setField2(param2)
79```
80
81#### Reading Fields
82
83`ReadField` uses a generated parameter `Accessor` to read the appropriate field from the Cap'n Proto Reader object and reconstruct C++ parameters.
84
85```mermaid
86sequenceDiagram
87    participant serverInvoke as clientInvoke or<br/>serverInvoke
88    participant ReadField
89    participant Accessor
90    participant Reader as Params::Reader
91    participant ServerCall
92
93    Note over serverInvoke,ServerCall: Deserializing Parameters
94    serverInvoke->>ReadField: Read param1
95    ReadField->>Accessor: Use generated field accessor
96    Accessor->>Reader: reader.getField1()
97    Reader-->>ServerCall: call function with param1
98```
99
100## Server-Side Request Processing
101
102The generated server code uses a Russian nesting doll structure to process method fields. Each `ServerField` wraps another `ServerField` (for the next parameter), or wraps `ServerRet` (for the return value), which finally wraps `ServerCall` (which invokes the actual C++ method).
103
104Each `ServerField` invokes `PassField`, which:
1051. Calls `ReadField` to deserialize the parameter from the `Params::Reader`
1062. Calls the next nested layer's `invoke()` with the accumulated parameters
1073. Calls `BuildField` to serialize the parameter back if it's an output parameter
108
109`ServerRet` invokes the next layer (typically `ServerCall`), stores the result, and calls `BuildField` to serialize it into the `Results::Builder`.
110
111`ServerCall` uses the generated `ProxyMethod<MethodParams>::impl` pointer-to-member to invoke the actual C++ method on the wrapped implementation object.
112
113```mermaid
114sequenceDiagram
115    participant serverInvoke
116    participant SF1 as ServerField<br/>(param 1)
117    participant SF2 as ServerField<br/>(param 2)
118    participant SR as ServerRet<br/>(return value)
119    participant SC as ServerCall
120    participant PMT as ProxyMethodTraits
121    participant Impl as Actual C++ Method
122
123    serverInvoke->>SF1: SF1::invoke 
124    SF1->>SF2: SF2::invoke
125    SF2->>SR: SR::invoke
126    SR->>SC: SC::invoke
127    SC->>PMT: PMT::invoke
128    PMT->>Impl: Call impl method
129    Impl->>PMT: return
130    PMT->>SC: return
131    SC->>SR: return
132    SR->>SF2: return
133    SF2->>SF1: return
134    SF1->>serverInvoke: return
135```
136
137Destroy methods are a special case: a capnp interface can declare a `destroy` method that is handled by `ServerDestroy` instead of `ServerCall`. Rather than dispatching through the `ServerField`/`ServerRet`/`ServerCall` chain to a C++ interface method, `ServerDestroy` calls `invokeDestroy()` on the `ProxyServer`, which resets `m_impl` and runs any registered cleanup functions, giving the client a way to synchronously destroy the wrapped object on the server side.
138
139## Advanced Features
140
141### Callbacks
142
143Callbacks (passed as `std::function` arguments) are intercepted by `CustomBuildField` and converted into Cap'n Proto capabilities that can be invoked across process boundaries. On the receiving end, `CustomReadField` intercepts the capability and constructs a `ProxyCallFn` object with an `operator()` that sends function calls back over the socket to invoke the original callback.
144
145```mermaid
146sequenceDiagram
147    participant CT as Client Thread
148    participant C as clientInvoke
149    participant CBF1 as CustomBuildField (Client)
150    participant S as Socket
151    participant CRF1 as CustomReadField (Server)
152    participant Srv as Server Code
153    participant PCF as ProxyCallFn
154
155    C->>CBF1: send function parameter
156    CBF1->>S: creates a Server for the function and sends a capability
157    S->>CRF1: receives a capability and creates ProxyCallFn
158    CRF1->>Srv:
159    Srv->>PCF: call the callback
160    PCF-->>CT: sends request to Client
161```
162
163### Thread Mapping
164
165Thread mapping enables each client thread to have a dedicated server thread processing its requests, preserving thread-local state and allowing recursive mutex usage across process boundaries.
166
167Thread mapping is initialized by defining an interface method with a `ThreadMap` parameter and/or response. The example below adds `ThreadMap` to the `construct` method because libmultiprocess calls the `construct` method automatically.
168
169```capnp
170interface InitInterface $Proxy.wrap("Init") { 
171    construct @0 (threadMap: Proxy.ThreadMap) -> (threadMap :Proxy.ThreadMap);
172}
173```
174
175- **ThreadMap in parameter**: The client's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the server, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
176- **ThreadMap in response**: The server's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the client, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
177
178You can specify ThreadMap in the parameter only, response only, or both:
179- **Parameter only**: Server can create threads on the client
180- **Response only**: Client can create threads on the server
181- **Both (as shown)**: Bidirectional thread creation
182
183When both parameter and response include ThreadMap, both processes end up with `ThreadMap::Client` capabilities pointing to each other's `ThreadMap::Server`, allowing both sides to create threads on the other process.
184
185### Async Processing with Context
186
187By adding a `Context` parameter to a method in the capnp interface file, you enable async processing where the client tells the server to execute the request in a separate worker thread. For example:
188
189```capnp
190processData @5 (context :Proxy.Context, data :Data) -> (result :Result);
191```
192
193If a method does not have a `Context` parameter, then libmultiprocess will execute IPC requests invoking that method on the I/O event loop thread. This is fine for fast and non-blocking methods, but should be avoided for any methods that are slow or blocking or make any IPC calls(including callbacks to the client), since as long as the method is executing, the Cap'n Proto event loop will not be able to perform any I/O.
194
195When a method has a `Context` parameter:
196
197**Client side** (`CustomBuildField`):
198If this is the first asynchronous request made from the current client thread, `CustomBuildField` will:
1991. Call `connection.m_thread_map.makeThreadRequest()` to request a dedicated worker thread on the server (stored in `request_threads` map)
2002. Set the remote thread capability in `Context.thread`
2013. Create a local `Thread::Server` object for the current thread (stored in `callback_threads` map)
2024. Set the local thread capability in `Context.callbackThread`
203
204Subsequent requests will reuse the existing thread capabilities held in `callback_threads` and `request_threads`.
205
206**Server side** (`PassField`):
2071. Looks up the local `Thread::Server` object specified by `context.thread`
2082. The worker thread:
209   - Stores `context.callbackThread` in its `request_threads` map (so callbacks go to the right client thread)
210   - Posts the work lambda to that thread's queue via `waiter->post(invoke)`
211   - Cleans up the `request_threads` entry
212
213## Interface Definitions
214
215As explained in the [usage](usage.md) document, interface descriptions need to be consumed both by the _libmultiprocess_ code generator, and by C++ code that calls and implements the interfaces. The C++ code only needs to know about C++ arguments and return types, while the code generator only needs to know about capnp arguments and return types, but both need to know class and method names, so the corresponding `.h` and `.capnp` source files contain some of the same information, and have to be kept in sync manually when methods or parameters change. Despite the redundancy, reconciling the interface definitions is designed to be _straightforward_ and _safe_. _Straightforward_ because there is no need to write manual serialization code or use awkward intermediate types like [`UniValue`](https://github.com/bitcoin/bitcoin/blob/master/src/univalue/include/univalue.h) instead of native types. _Safe_ because if there are any inconsistencies between API and data definitions (even minor ones like using a narrow int data type for a wider int API input), there are errors at build time instead of errors or bugs at runtime.
216
217In the future, it would be possible to combine API and data definitions together using [C++ attributes](https://en.cppreference.com/w/cpp/language/attributes). To do this we would add attributes to the API definition files, and then generate the data definitions from the API definitions and attributes. I didn't take this approach mostly because it would be extra work, but also because until C++ standardizes reflection, this would require either hooking into compiler APIs like https://github.com/RosettaCommons/binder, or parsing C++ code manually like http://www.swig.org/.
218
219## What is `kj`?
220
221KJ is a concurrency framework [bundled with
222capnproto](https://capnproto.org/cxxrpc.html#kj-concurrency-framework); it is used as a
223basis in this library to construct the event-loop necessary to service IPC requests.
224
225## Future directions
226
227_libmultiprocess_ uses the [Cap'n Proto](https://capnproto.org) interface description language and protocol, but it could be extended or changed to use a different IDL/protocol like [gRPC](https://grpc.io). The nice thing about _Cap'n Proto_ compared to _gRPC_ and most other lower level protocols is that it allows interface pointers (_Services_ in gRPC parlance) to be passed as method arguments and return values, so object references and bidirectional requests work out of the box. Supporting a lower-level protocol would require adding maps and tracking code to proxy objects.
228
229_libmultiprocess_ is currently compatible with sandboxing but could add platform-specific sandboxing support or integration with a sandboxing library like [SAPI](https://github.com/google/sandboxed-api).
230