Inside Bitcoin

1# Bootstrappable Bitcoin Core Builds
2
3This directory contains the files necessary to perform bootstrappable Bitcoin
4Core builds.
5
6[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
7to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
8downloads.
9
10We achieve bootstrappability by using Guix as a functional package manager.
11
12# Requirements
13
14Conservatively, you will need:
15
16- 16GB of free disk space on the partition that /gnu/store will reside in
17- 8GB of free disk space **per platform triple** you're planning on building
18  (see the `HOSTS` [environment variable description][env-vars-list])
19
20# Installation and Setup
21
22If you don't have Guix installed and set up, please follow the instructions in
23[INSTALL.md](./INSTALL.md)
24
25# Usage
26
27If you haven't considered your security model yet, please read [the relevant
28section](#choosing-your-security-model) before proceeding to perform a build.
29
30## Making the Xcode SDK available for macOS cross-compilation
31
32In order to perform a build for macOS (which is included in the default set of
33platform triples to build), you'll need to extract the macOS SDK tarball using
34tools found in the [`macdeploy` directory](../macdeploy/README.md#sdk-extraction).
35
36You can then either point to the SDK using the `SDK_PATH` environment variable:
37
38```sh
39# Extract the SDK tarball to /path/to/parent/dir/of/extracted/SDK/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers
40tar -C /path/to/parent/dir/of/extracted/SDK -xaf /path/to/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers.tar
41
42# Indicate where to locate the SDK tarball
43export SDK_PATH=/path/to/parent/dir/of/extracted/SDK
44```
45
46or extract it into `depends/SDKs`:
47
48```sh
49mkdir -p depends/SDKs
50tar -C depends/SDKs -xaf /path/to/SDK/tarball
51```
52
53## Building
54
55*The author highly recommends at least reading over the [common usage patterns
56and examples](#common-guix-build-invocation-patterns-and-examples) section below
57before starting a build. For a full list of customization options, see the
58[recognized environment variables][env-vars-list] section.*
59
60To build Bitcoin Core reproducibly with all default options, invoke the
61following from the top of a clean repository:
62
63```sh
64./contrib/guix/guix-build
65```
66
67## Codesigning build outputs
68
69The `guix-codesign` command attaches codesignatures (produced by codesigners) to
70existing non-codesigned outputs. Please see the [release process
71documentation](/doc/release-process.md#codesigning) for more context.
72
73It respects many of the same environment variable flags as `guix-build`, with 2
74crucial differences:
75
761. Since only Windows and macOS build outputs require codesigning, the `HOSTS`
77   environment variable will have a sane default value of `x86_64-w64-mingw32
78   x86_64-apple-darwin arm64-apple-darwin` instead of all the platforms.
792. The `guix-codesign` command ***requires*** a `DETACHED_SIGS_REPO` flag.
80    * _**DETACHED_SIGS_REPO**_
81
82      Set the directory where detached codesignatures can be found for the current
83      Bitcoin Core version being built.
84
85      _REQUIRED environment variable_
86
87An invocation with all default options would look like:
88
89```
90env DETACHED_SIGS_REPO=<path/to/bitcoin-detached-sigs> ./contrib/guix/guix-codesign
91```
92
93## Cleaning intermediate work directories
94
95By default, `guix-build` leaves all intermediate files or "work directories"
96(e.g. `depends/work`, `guix-build-*/distsrc-*`) intact at the end of a build so
97that they are available to the user (to aid in debugging, etc.). However, these
98directories usually take up a large amount of disk space. Therefore, a
99`guix-clean` convenience script is provided which cleans the current `git`
100worktree to save disk space:
101
102```
103./contrib/guix/guix-clean
104```
105
106## Gathering shasums of build outputs
107
108After a successful build, the shasums of the build outputs are gathered
109into files named `SHA256SUMS`. These files are located in each of the
110architecture-specific output directories.
111
112To gather all shasums and output them together to console, for e.g.
113inclusion in a Guix pull request comment, run:
114
115``` sh
116source contrib/shell/git-utils.bash && uname -m && find guix-build-$(git_head_version)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
117```
118
119## Attesting to build outputs
120
121Much like how Gitian build outputs are attested to in a `gitian.sigs`
122repository, Guix build outputs are attested to in the [`guix.sigs`
123repository](https://github.com/bitcoin-core/guix.sigs).
124
125After you've cloned the `guix.sigs` repository, to attest to the current
126worktree's commit/tag:
127
128```
129env GUIX_SIGS_REPO=<path/to/guix.sigs> SIGNER=<gpg-key-name> ./contrib/guix/guix-attest
130```
131
132See `./contrib/guix/guix-attest --help` for more information on the various ways
133`guix-attest` can be invoked.
134
135## Verifying build output attestations
136
137After at least one other signer has uploaded their signatures to the `guix.sigs`
138repository:
139
140```
141git -C <path/to/guix.sigs> pull
142env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
143```
144
145
146## Common `guix-build` invocation patterns and examples
147
148### Keeping caches and SDKs outside of the worktree
149
150If you perform a lot of builds and have a bunch of worktrees, you may find it
151more efficient to keep the depends tree's download cache, build cache, and SDKs
152outside of the worktrees to avoid duplicate downloads and unnecessary builds. To
153help with this situation, the `guix-build` script honours the `SOURCES_PATH`,
154`BASE_CACHE`, and `SDK_PATH` environment variables so that you can do something
155like:
156
157```sh
158env SOURCES_PATH="$HOME/depends-SOURCES_PATH" BASE_CACHE="$HOME/depends-BASE_CACHE" SDK_PATH="$HOME/macOS-SDKs" ./contrib/guix/guix-build
159```
160
161Note that the paths that these environment variables point to **must be
162directories**, and **NOT symlinks to directories**.
163
164See the [recognized environment variables][env-vars-list] section for more
165details.
166
167### Building a subset of platform triples
168
169Sometimes you only want to build a subset of the supported platform triples, in
170which case you can override the default list by setting the space-separated
171`HOSTS` environment variable:
172
173```sh
174env HOSTS='x86_64-w64-mingw32 x86_64-apple-darwin' ./contrib/guix/guix-build
175```
176
177See the [recognized environment variables][env-vars-list] section for more
178details.
179
180### Controlling the number of threads used by `guix` build commands
181
182Depending on your system's RAM capacity, you may want to decrease the number of
183threads used to decrease RAM usage or vice versa.
184
185By default, the scripts under `./contrib/guix` will invoke all `guix` build
186commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
187specified. However, astute manual readers will also notice that `guix` build
188commands also accept a `--max-jobs=` flag (which defaults to 1 if unspecified).
189
190Here is the difference between `--cores=` and `--max-jobs=`:
191
192> Note: When I say "derivation," think "package"
193
194`--cores=`
195
196  - controls the number of CPU cores to build each derivation. This is the value
197    passed to `make`'s `--jobs=` flag.
198
199`--max-jobs=`
200
201  - controls how many derivations can be built in parallel
202  - defaults to 1
203
204Therefore, the default is for `guix` build commands to build one derivation at a
205time, utilizing `$JOBS` threads.
206
207Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
208can also modify the value for `--max-jobs=` by specifying
209`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
210may want to set:
211
212```sh
213export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
214```
215
216Which allows for a maximum of 8 derivations to be built at the same time, each
217utilizing `$JOBS` threads.
218
219Or, if you'd like to avoid spurious build failures caused by issues with
220parallelism within a single package, but would still like to build multiple
221packages when the dependency graph allows for it, you may want to try:
222
223```sh
224export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
225```
226
227See the [recognized environment variables][env-vars-list] section for more
228details.
229
230## Recognized environment variables
231
232* _**HOSTS**_
233
234  Override the space-separated list of platform triples for which to perform a
235  bootstrappable build.
236
237  _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu
238  riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
239  x86\_64-w64-mingw32 x86\_64-apple-darwin arm64-apple-darwin")_
240
241* _**SOURCES_PATH**_
242
243  Set the depends tree download cache for sources. This is passed through to the
244  depends tree. Setting this to the same directory across multiple builds of the
245  depends tree can eliminate unnecessary redownloading of package sources.
246
247  The path that this environment variable points to **must be a directory**, and
248  **NOT a symlink to a directory**.
249
250* _**BASE_CACHE**_