Skip to main content

RPC Performance: Forest vs Lotus

The following measurements were taken while a Forest node and a Lotus node handled a similar stream of real RPC provider traffic during the same time window. Because both nodes served an almost identical request stream, the differences below are attributable to the nodes themselves rather than to uneven load. The "Comparable load" section documents the checks used to support that assumption.

Environment

The measurements cover the following node builds:

NodeVersionCommit
Forest0.34.1git.0ba362e
Lotus1.36.1-rc1git.45eade8

Hardware: Intel / 8 vCPUs / 32 GB RAM / 640 GB disk

RPC latency

Average wall time per HTTP request was approximately 10-15 ms for Forest and 30-50 ms for Lotus over most of the measurement window. Both traces rose toward the end as the load increased, with Forest reaching about 30 ms and Lotus about 48 ms.

Average wall time per HTTP request

Per-flow latency

At the median (P50), the two nodes were close: approximately 7 ms for Forest and 8 ms for Lotus. The difference was larger in the tail. At P95, Forest stayed near 50 ms and rose to roughly 115 ms under load, while Lotus stayed around 70-85 ms and reached roughly 185 ms. At P99, Forest held around 200 ms and rose to about 350 ms, while Lotus ranged from approximately 240 ms to a peak near 600 ms.

P50 per-flow latency

P95 per-flow latency

P99 per-flow latency

Amortized latency

The amortized view accounts for batched requests. It shows the same pattern as the per-flow view: comparable values at P50 and a larger difference at P95, P99, and on average.

P50 amortized latency

P95 amortized latency

P99 amortized latency

Average amortized latency

Comparable load

The request rates reaching each node were similar, supporting a direct comparison between the two nodes. HTTP request counts averaged 45.2 req/s for Forest and 45.0 req/s for Lotus, with batch-aware counts at 20.4 and 20.3 req/s, respectively. Both nodes, therefore, received a comparable volume of traffic.

Request count (HTTP only)

Request count (batch-aware)

Concurrency followed from the latency difference: Forest kept fewer requests in flight at any instant (mean 0.43, peak 4) than Lotus (mean 1.45, peak 8).

In-flight requests

Resource usage

While serving this workload, Forest used approximately 10-15% CPU, and Lotus used approximately 45-55%, briefly spiking to about 80% during the higher-load period. Forest's CPU trace was also less variable.

CPU usage

Memory usage was approximately 8-10% for Forest and approximately 25-30% for Lotus, with Lotus briefly reaching about 40% during the higher-load period.

Memory usage

note

The absolute memory figures are higher than they appear here because of OS page caches, but the relative proportions between the two nodes still hold.