Smarter Storage for Azure HorizonDB

A key benefit of Azure HorizonDB  comes from its smart storage layer, which works seamlessly with Postgres and takes over durability and high availability work typically done by Postgres itself.  This frees Postgres to operate as a stateless compute engine focused on running queries and transactions leaving more CPU, disk, and network bandwidth available for your application’s business logic.  For example, HorizonDB’s storage layer handles work such as replicating WAL to read replicas and archiving WAL to blob storage for point-in-time restore. PostgreSQL also no longer needs full-page writes or background dirty-page flushing, that work is handled by the storage layer. The rest of this post explains the storage architecture that enables this division of responsibility.

HorizonDB Storage Overview

HorizonDB’s storage design is similar to that of Socrates.  The write‑ahead log (WAL) is the sole mechanism used to persist writes to storage (Orange in Figure 1).   PostgreSQL compute nodes never write data pages to storage directly in HorizonDB (blue in Figure 1). Instead, pages and other on‑disk structures are treated as derived state and are reconstructed by the storage layer from WAL records.  Each Postgres instance in a cluster shares the same copy of the data pages in storage (blue “Page Reads” arrow in Figure 1).   A Postgres read replica doesn’t need to maintain its own copy of WAL and data pages as in a typical Postgres setup with a hot standby or readable secondary.

Figure 1 - HorizonDB WAL and Page Storage

HorizonDB uses separate storage services for WAL and data pages. This separation allows each to be specialized for the very different patterns of reads and writes for WAL versus pages.  The WAL server is optimized for low latency writes to the tail of the log stream and reads often at or near the tail.  The page server is optimized for elastic storage of potentially many terabytes of pages and it must support random reads and writes of pages. This specialization improves query performance as we have demonstrated in the past.

Moving Work into Storage

Figure 2) shows the work Postgres does in a typical configuration with high availability, backups and WAL archiving configured.  The Postgres primary is doing the work to send WAL to the secondary, and archive WAL to blob storage.   The primary and secondary nodes both maintain their own complete copies of the data.  

Figure 2 - Postgres without Smart Storage

Figure 3) shows where the equivalent work is done by HorizonDB storage.   The primary and secondaries share one copy of the data in storage in this design. 

Figure 3 – HorizonDB storage offloading work from Postgres

Some example work HorizonDB storage is now doing in Figure 3.

• The work usually done by the Postgres walsender and archiver processes to send WAL to standbys\secondaries or to WAL archive (blob storage) is now done by HorizonDB’s WAL servers in Figure 3) instead of by Postgres as in Figure 2).   In HorizonDB, Postgres secondaries don’t ever communicate directly the with primary.
• When Postgres starts up, it no longer needs to redo WAL to bring stale page content up to date speeding up startup times. HorizonDB’s page servers in Figure 3) take care of page recovery for Postgres and can send it any pages it needs on demand.
• The Postgres checkpointer and background writer no longer need to write dirty pages to disk. The page server handles this work.  Both Postgres primary and any secondaries can retrieve the page versions they need from the page servers.
• HorizonDB page servers protect against torn pages making Postgres Full page writes unnecessary reducing the amount of WAL generated and improving write throughput.   
• On read replicas, HorizonDB only requires redo of WAL onto pages that the read replica has stored in memory already saving the work to persist the WAL and redo it. HorizonDB read replicas don’t maintain their own private copy of the data, so redo is only used as a mechanism to keep their in-memory copy in sync with the primary. 

Tables 1) and 2) below summarize the Postgres processes whose work moved into HorizonDBs storage layer.