The need for a maintainer mindset in the age of LLMs
For a few years now, people have been predicting the end of software engineering as a profession. Matt Welsh argued in Communications of the ACM in 2023 that the conventional idea of writing a program was headed for extinction. In June, CACM ran a piece called "The End of the Coder?", which notes that the head of Claude Code at Anthropic says all his own code is now AI-generated and the profession isn’t about writing code anymore. Regardless of whether you think humans can write better code than LLMs, there is still work involved in committing a patch to PostgreSQL that people can do better than AI. As PostgreSQL committers, we have been reflecting on the need for a maintainer mindset in the age of LLMs. Asking the right questions LLMs can be a huge help in software development, but they need to be coupled with software maintainers who have a mental model of the system and a vision for the product. LLMs are specifically adept at finding bugs, but are best at this when guided by an experienced engineer. Index-only scans in Postgres are a powerful performance boost, but they rely on visibility information for tuples being up to date. In Postgres 19, there is a feature that enables SELECT queries to set the visibility map to allow index-only scans to be performed, in many cases without need for vacuum. It went through extensive testing and verification. But after commit, several oversights were discovered. For example, it neglected to update the free space map when setting the page all-visible, leading to bloat. So why weren’t these found during development? Many LLM models were used to review the code and perform adversarial testing, yet none found issues. It was found by maintainers analyzing the code armed with their own understanding of the system. Even as the LLM models improve, it’s easiest to guide them to identify bugs and discrepancies when you have a mental model of the system and can ask probing questions, and you know the likely failure modes. To develop that mental model, over the years, maintainers have spent countless hours studying the code, documenting function call stacks, and stepping through code paths. This foundation is what makes both developing and debugging committable code with LLMs work. Understanding users and designing interfaces These days, the prep work for starting a new Postgres project includes using an LLM to do analysis for how to implement a feature consistently. It might involve summarizing relevant mailing list discussions or surveying other similar cases in our code bases and other systems. While helpful, it isn’t a substitute for understanding users and their production needs. At a high level, how does this fit into Postgres as a whole? Is there user-visible surface area? Is it exposed in EXPLAIN? Do properties of the underlying tables matter to the implementation? How does it interact with replication? These questions need to be considered from both a technical as well as product angle. In 2024, there was a community discussion where a contributor reported that only the parallel leader's stats were being reported in parallel bitmap heap scan EXPLAIN. The contributor’s team had repeatedly drawn wrong conclusions in their query analysis because they assumed the reported block counts included the workers. A patch was proposed to fix it, and then there was extensive debate about what the right fix was. The debate was mostly not about the code. It was about what the output should mean. This kind of audit used to be done by hand. Today an LLM can do the audit, but the processing and analysis still takes time. The need for developing the mental model hasn't been replaced, only the tools at hand to do it. The fix was eventually to make the leader report its own count and a per-worker line beneath it. Committing the patch because it fixed the specific reported issue wasn't good enough: maintaining product quality requires looking at the wider picture and doing so in collaboration with users. Defining, essentially, how the feature should work. Widening scope Another area where LLMs can help software maintainers is surveying large codebases like Postgres and helping developers see a bigger picture rather than fixing issues in isolation. But even though it sounds cliché, LLMs are no substitute for judgment. Maintainers are responsible for knowing when a redesign or paradigm change is needed. There was recently another community discussion thread about partition indexes losing certain properties after a particular ALTER command. The participants used LLMs to survey other ALTER commands and other index properties and found that many were incorrectly dropped. Knowing all of these, the shape of the solution changed. Interrogating this, thinking about what kind of a system could prevent these omissions, what is the correct model, that takes time. You can instruct an LLM to look for other precedents, but ultimately a good portion of the work is the human judgment to think about if and how to change an existing precedent or introduce a new one that will better encompass all cases. You may need a new precedent: a bigger change that will prevent problems like this from happening again. Perhaps a new system could automate detection of omissions or fail when required components are omitted. LLMs learn from existing code, but humans are still responsible for determining when we should do something new. Postgres 19, LLMs, and the rise of good reproducers Many of the new features in Postgres 19 started their life a long time ago, as an idea in the head of a developer or through a discussion in the hallway at a conference. Some saw development long before AI tools were even available, but all of them have seen a tremendous amount of review and scrutiny thanks to LLMs. Many bugs have been identified, some features have been reverted, and the remaining features have been polished and stabilized late in the release. Online checksums, the ability to enable checksums on data pages in a running cluster, has been worked on for years, culminating in it being committed and then reverted from Postgres 19. Changing state synchronized across a distributed system is not easy, and bugs are often dependent on very small race condition windows. Being able to consistently reproduce errors is as helpful as it can be hard to achieve. This is an area where LLMs have filled a very important role. There was one bug in particular which broke the tests intermittently. Sometimes it took 5 runs, sometimes 500. The only common characteristic between test runs was inconsistency, and that it needed to be executed in a resource-constrained environment. Running the tests on a typical developer machine did not reproduce anything, making it even harder to observe. The window in question was, however, frustratingly hard to define. Eventually the bug was found through countless hours of reading logs and code. Since then, LLMs have become magnitudes better at creating tests, and bugs found since have often had a reproducer from the start. If this bug was found today, chances are we could have been able to focus on the fix and not the identification. Understanding every line Tempting as it may seem, the code generated by the LLM to reproduce the bug cannot be committed verbatim along with the fix. Every line of code must be read, understood, and checked for correctness. A lot is then also rewritten to improve readability, maintainability and match project style. LLMs tend to be verbose and encode all assumptions. This often has the effect of communicating nothing. Maintainers are responsible for understanding what is non-obvious and including only the important details. This time spent by maintainers saves the time of countless future developers. It is the developers who suffer the consequences of over-wrought tests and verbose comments. LLMs don’t care if tests take hours to run or if comments take minutes to read. Being able to describe the bug in plain prose, with references to code, is a skillset a maintainer grows as their knowledge of the system and code proves. Internalizing every line also builds a foundation for devising systemic change to push the project forward Trust still requires humans In his 1984 Turing Award lecture, Ken Thompson taught us that we cannot trust code we didn’t create ourselves. This is as true today as it was in 1984. The trust given by users of our software is a trust in the maintainers, not the lines of code. There is no free lunch, but when LLM tools are used to further our knowledge and help us see things we overlooked, perhaps we can at least have a subsidized meal. About PostgreSQL Perspectives PostgreSQL Perspectives is a monthly blog series featuring insights from Microsoft engineers, contributors, and builders working with PostgreSQL. Explore more articles on the Microsoft for PostgreSQL blog.
