How we built Pingora, the proxy that connects Cloudflare to the Internet

Cloudflare has built Pingora, a new HTTP proxy written in Rust, to replace their aging NGINX infrastructure. Pingora now serves over 1 trillion requests a day, boosting performance and enabling new features while requiring only a third of the CPU and memory resources of the previous system. As Cloudflare scaled, they outgrew NGINX due to its process-based architecture, which limited connection reuse and made it difficult to implementing complex features safely.

Pingora was designed to solve these limitations by using a multithreaded architecture with work stealing, rather than NGINX's multiprocess model. This allows for better sharing of resources, particularly connection pools, across all threads. The move to Rust provided memory safety without compromising performance, a significant advantage over the C and Lua codebase of NGINX, which was prone to memory safety issues and difficult to extend.

The new proxy has delivered substantial benefits in production. It consumes about 70% less CPU and 67% less memory compared to the old service with the same traffic load. Connection reuse ratios improved dramatically (e.g., from 87.1% to 99.92% for one customer), reducing the overhead of TCP and TLS handshakes. Furthermore, Pingora provides a developer-friendly, programmable interface that allows engineers to build features quickly and safely, without the fear of causing crashes due to memory errors.

Key Concepts

• Architecture Limitations of NGINX: The process-based model of NGINX limited connection reuse (pools were per-worker) and suffered from performance issues when blocking operations occurred, as requests were pinned to workers.
• Pingora's Design: Built from scratch in Rust, Pingora uses a multithreaded architecture with work stealing to share resources like connection pools efficiently across all requests.
• Performance & Efficiency: Pingora achieves 70% less CPU and 67% less memory usage than the previous NGINX setup, with significantly higher connection reuse rates reducing handshake overhead.
• Safety & Velocity: Rust guarantees memory safety, eliminating entire classes of bugs (like segfaults) and allowing Cloudflare engineers to ship features faster and with more confidence.
• Customizability: Pingora offers a "life of a request" event-based programmable interface, allowing for easy extension and separation of business logic from generic proxy logic.