Inside Bitcoin

1<!DOCTYPE html>
2<html>
3<head>
4<title>LevelDB Benchmarks</title>
5<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
6<style>
7body {
8  font-family:Helvetica,sans-serif;
9  padding:20px;
10}
11
12h2 {
13  padding-top:30px;
14}
15
16table.bn {
17  width:800px;
18  border-collapse:collapse;
19  border:0;
20  padding:0;
21}
22
23table.bnbase {
24  width:650px;
25}
26
27table.bn td {
28  padding:2px 0;
29}
30
31table.bn td.c1 {
32  font-weight:bold;
33  width:150px;
34}
35
36table.bn td.c1 div.e {
37  float:right;
38  font-weight:normal;
39}
40
41table.bn td.c2 {
42  width:150px;
43  text-align:right;
44  padding:2px;
45}
46
47table.bn td.c3 {
48  width:350px;
49}
50
51table.bn td.c4 {
52  width:150px;
53  font-size:small;
54  padding-left:4px;
55}
56
57/* chart bars */
58div.bldb {
59  background-color:#0255df;
60}
61
62div.bkct {
63  background-color:#df5555;
64}
65
66div.bsql {
67  background-color:#aadf55;
68}
69
70.code {
71  font-family:monospace;
72  font-size:large;
73}
74
75.todo {
76  color: red;
77}
78
79</style>
80</head>
81<body>
82<h1>LevelDB Benchmarks</h1>
83<p>Google, July 2011</p>
84<hr>
85
86<p>In order to test LevelDB's performance, we benchmark it against other well-established database implementations. We compare LevelDB (revision 39) against <a href="http://www.sqlite.org/">SQLite3</a> (version 3.7.6.3) and <a href="https://dbmx.net/kyotocabinet/spex.html">Kyoto Cabinet's</a> (version 1.2.67) TreeDB (a B+Tree based key-value store). We would like to acknowledge Scott Hess and Mikio Hirabayashi for their suggestions and contributions to the SQLite3 and Kyoto Cabinet benchmarks, respectively.</p>
87
88<p>Benchmarks were all performed on a six-core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, with 12288 KB of total L3 cache and 12 GB of DDR3 RAM at 1333 MHz. (Note that LevelDB uses at most two CPUs since the benchmarks are single threaded: one to run the benchmark, and one for background compactions.) We ran the benchmarks on two machines (with identical processors), one with an Ext3 file system and one with an Ext4 file system. The machine with the Ext3 file system has a SATA Hitachi HDS721050CLA362 hard drive. The machine with the Ext4 file system has a SATA Samsung HD502HJ hard drive. Both hard drives spin at 7200 RPM and have hard drive write-caching enabled (using `hdparm -W 1 [device]`). The numbers reported below are the median of three measurements.</p>
89
90<h4>Benchmark Source Code</h4>
91<p>We wrote benchmark tools for SQLite and Kyoto TreeDB based on LevelDB's <span class="code">db_bench</span>. The code for each of the benchmarks resides here:</p>
92<ul>
93	<li> <b>LevelDB:</b> <a href="https://github.com/google/leveldb/blob/master/benchmarks/db_bench.cc">benchmarks/db_bench.cc</a>.</li>
94	<li> <b>SQLite:</b> <a href="https://github.com/google/leveldb/blob/master/benchmarks/db_bench_sqlite3.cc">benchmarks/db_bench_sqlite3.cc</a>.</li>
95	<li> <b>Kyoto TreeDB:</b> <a href="https://github.com/google/leveldb/blob/master/benchmarks/db_bench_tree_db.cc">benchmarks/db_bench_tree_db.cc</a>.</li>
96</ul>
97
98<h4>Custom Build Specifications</h4>
99<ul>
100<li>LevelDB: LevelDB was compiled with the <a href="http://code.google.com/p/google-perftools">tcmalloc</a> library and the <a href="http://code.google.com/p/snappy/">Snappy</a> compression library (revision 33).  Assertions were disabled.</li>
101<li>TreeDB: TreeDB was compiled using the <a href="http://www.oberhumer.com/opensource/lzo/">LZO</a> compression library (version 2.03). Furthermore, we enabled the TSMALL and TLINEAR options when opening the database in order to reduce the footprint of each record.</li>
102<li>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive.  We also enabled SQLite's <a href="http://www.sqlite.org/draft/wal.html">write-ahead logging</a>.</li>
103</ul>
104
105<h2>1. Baseline Performance</h2>
106<p>This section gives the baseline performance of all the
107databases.  Following sections show how performance changes as various
108parameters are varied.  For the baseline:</p>
109<ul>
110	<li> Each database is allowed 4 MB of cache memory.</li>
111        <li> Databases are opened in <em>asynchronous</em> write mode.
112             (LevelDB's sync option, TreeDB's OAUTOSYNC option, and
113             SQLite3's synchronous options are all turned off).  I.e.,
114             every write is pushed to the operating system, but the
115             benchmark does not wait for the write to reach the disk.</li>
116	<li> Keys are 16 bytes each.</li>
117        <li> Value are 100 bytes each (with enough redundancy so that
118             a simple compressor shrinks them to 50% of their original
119             size).</li>
120	<li> Sequential reads/writes traverse the key space in increasing order.</li>
121	<li> Random reads/writes traverse the key space in random order.</li>
122</ul>
123
124<h3>A. Sequential Reads</h3>
125<table class="bn bnbase">
126<tr><td class="c1">LevelDB</td>
127    <td class="c2">4,030,000 ops/sec</td>
128    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
129<tr><td class="c1">Kyoto TreeDB</td>
130    <td class="c2">1,010,000 ops/sec</td>
131    <td class="c3"><div class="bkct" style="width:95px">&nbsp;</div></td>
132<tr><td class="c1">SQLite3</td>
133    <td class="c2">383,000 ops/sec</td>
134    <td class="c3"><div class="bsql" style="width:33px">&nbsp;</div></td>
135</table>
136<h3>B. Random Reads</h3>
137<table class="bn bnbase">
138<tr><td class="c1">LevelDB</td>
139    <td class="c2">129,000 ops/sec</td>
140    <td class="c3"><div class="bldb" style="width:298px">&nbsp;</div></td>
141<tr><td class="c1">Kyoto TreeDB</td>
142    <td class="c2">151,000 ops/sec</td>
143    <td class="c3"><div class="bkct" style="width:350px">&nbsp;</div></td>
144<tr><td class="c1">SQLite3</td>
145    <td class="c2">134,000 ops/sec</td>
146    <td class="c3"><div class="bsql" style="width:310px">&nbsp;</div></td>
147</table>
148<h3>C. Sequential Writes</h3>
149<table class="bn bnbase">
150<tr><td class="c1">LevelDB</td>
151    <td class="c2">779,000 ops/sec</td>
152    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
153<tr><td class="c1">Kyoto TreeDB</td>
154    <td class="c2">342,000 ops/sec</td>
155    <td class="c3"><div class="bkct" style="width:154px">&nbsp;</div></td>
156<tr><td class="c1">SQLite3</td>
157    <td class="c2">48,600 ops/sec</td>
158    <td class="c3"><div class="bsql" style="width:22px">&nbsp;</div></td>
159</table>
160<h3>D. Random Writes</h3>
161<table class="bn bnbase">
162<tr><td class="c1">LevelDB</td>
163    <td class="c2">164,000 ops/sec</td>
164    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
165<tr><td class="c1">Kyoto TreeDB</td>
166    <td class="c2">88,500 ops/sec</td>
167    <td class="c3"><div class="bkct" style="width:188px">&nbsp;</div></td>
168<tr><td class="c1">SQLite3</td>
169    <td class="c2">9,860 ops/sec</td>
170    <td class="c3"><div class="bsql" style="width:21px">&nbsp;</div></td>
171</table>
172
173<p>LevelDB outperforms both SQLite3 and TreeDB in sequential and random write operations and sequential read operations. Kyoto Cabinet has the fastest random read operations.</p>
174
175<h2>2. Write Performance under Different Configurations</h2>
176<h3>A. Large Values </h3>
177<p>For this benchmark, we start with an empty database, and write 100,000 byte values (~50% compressible). To keep the benchmark running time reasonable, we stop after writing 1000 values.</p>
178<h4>Sequential Writes</h4>
179<table class="bn bnbase">
180<tr><td class="c1">LevelDB</td>
181    <td class="c2">1,100 ops/sec</td>
182    <td class="c3"><div class="bldb" style="width:234px">&nbsp;</div></td></tr>
183<tr><td class="c1">Kyoto TreeDB</td>
184    <td class="c2">1,000 ops/sec</td>
185    <td class="c3"><div class="bkct" style="width:224px">&nbsp;</div></td></tr>
186<tr><td class="c1">SQLite3</td>
187    <td class="c2">1,600 ops/sec</td>
188    <td class="c3"><div class="bsql" style="width:350px">&nbsp;</div></td></tr>
189</table>
190<h4>Random Writes</h4>
191<table class="bn bnbase">
192<tr><td class="c1">LevelDB</td>
193    <td class="c2">480 ops/sec</td>
194    <td class="c3"><div class="bldb" style="width:105px">&nbsp;</div></td></tr>
195<tr><td class="c1">Kyoto TreeDB</td>
196    <td class="c2">1,100 ops/sec</td>
197    <td class="c3"><div class="bkct" style="width:240px">&nbsp;</div></td></tr>
198<tr><td class="c1">SQLite3</td>
199    <td class="c2">1,600 ops/sec</td>
200    <td class="c3"><div class="bsql" style="width:350px">&nbsp;</div></td></tr>
201</table>
202<p>LevelDB doesn't perform as well with large values of 100,000 bytes each. This is because LevelDB writes keys and values at least twice: first time to the transaction log, and second time (during a compaction) to a sorted file.
203With larger values, LevelDB's per-operation efficiency is swamped by the
204cost of extra copies of large values.</p>
205<h3>B. Batch Writes</h3>
206<p>A batch write is a set of writes that are applied atomically to the underlying database. A single batch of N writes may be significantly faster than N individual writes. The following benchmark writes one thousand batches where each batch contains one thousand 100-byte values. TreeDB does not support batch writes and is omitted from this benchmark.</p>
207<h4>Sequential Writes</h4>
208<table class="bn">
209<tr><td class="c1">LevelDB</td>
210    <td class="c2">840,000 entries/sec</td>
211    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
212    <td class="c4">(1.08x baseline)</td></tr>
213<tr><td class="c1">SQLite3</td>
214    <td class="c2">124,000 entries/sec</td>
215    <td class="c3"><div class="bsql" style="width:52px">&nbsp;</div></td>
216    <td class="c4">(2.55x baseline)</td></tr>
217</table>
218<h4>Random Writes</h4>
219<table class="bn">
220<tr><td class="c1">LevelDB</td>
221    <td class="c2">221,000 entries/sec</td>
222    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
223    <td class="c4">(1.35x baseline)</td></tr>
224<tr><td class="c1">SQLite3</td>
225    <td class="c2">22,000 entries/sec</td>
226    <td class="c3"><div class="bsql" style="width:34px">&nbsp;</div></td>
227    <td class="c4">(2.23x baseline)</td></tr>
228</table>
229
230<p>Because of the way LevelDB persistent storage is organized, batches of
231random writes are not much slower (only a factor of 4x) than batches
232of sequential writes.</p>
233
234<h3>C. Synchronous Writes</h3>
235<p>In the following benchmark, we enable the synchronous writing modes
236of all of the databases.  Since this change significantly slows down the
237benchmark, we stop after 10,000 writes. For synchronous write tests, we've
238disabled hard drive write-caching (using `hdparm -W 0 [device]`).</p>
239<ul>
240    <li>For LevelDB, we set WriteOptions.sync = true.</li>
241    <li>In TreeDB, we enabled TreeDB's OAUTOSYNC option.</li>
242    <li>For SQLite3, we set "PRAGMA synchronous = FULL".</li>
243</ul>
244<h4>Sequential Writes</h4>
245<table class="bn">
246<tr><td class="c1">LevelDB</td>
247    <td class="c2">100 ops/sec</td>
248    <td class="c3"><div class="bldb" style="width:350px">&nbsp;</div></td>
249    <td class="c4">(0.003x baseline)</td></tr>
250<tr><td class="c1">Kyoto TreeDB</td>