Home
%3CLINGO-SUB%20id%3D%22lingo-sub-885243%22%20slang%3D%22en-US%22%3EAuditing%3A%20Know%20what's%20going%20on%20in%20your%20Postgres%20database%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-885243%22%20slang%3D%22en-US%22%3E%3CP%3EToday%20it's%20almost%20unsurprising%20to%20see%20another%20headline%20related%20to%20data%20breaches%20or%20data%20privacy.%20This%20affects%20you%20personally%20as%20a%20customer%20of%20those%20companies%20-%20you%20want%20someone%20to%20take%20good%20care%20of%20your%20data.%20It%20also%20affects%20you%20professionally%3B%20as%20a%20person%20who%20works%20with%20data%2C%20you%20are%20that%20'someone'%20for%20your%20customers.%20Good%20audit%20logging%20is%20one%20important%20tool%20in%20the%20belt%20of%20a%20security-aware%20data%20professional.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20are%20different%20kinds%20of%20security%20tools.%20Some%20are%20preventative%2C%20like%20role-based%20access%20control%20and%20authentication%20policies.%20Audit%20logging%2C%20on%20the%20other%20hand%2C%20is%20in%20the%20investigative%20section%20of%20the%20toolbox%20because%20a%20log%20is%20a%20record%20of%20a%20past%20action.%20(Although%20knowing%20that%20their%20actions%20will%20be%20tracked%20may%20inspire%20employees%20to%20be%20more%20careful).%20Audit%20logs%20help%20you%20identify%20the%20who%2C%20what%2C%20and%20when%20of%20data%20access%2C%20and%20to%20an%20extent%20the%20where%20and%20how.%20Having%20audit%20logs%20available%20for%20review%20is%20a%20key%20requirement%20for%20compliance%20in%20certain%20regulated%20industries%2C%20but%20also%20increasingly%20in%20other%20industries%20as%20data%20privacy%20laws%20like%20GDPR%20take%20effect.%20If%20worst%20comes%20to%20worst%20and%20there%20is%20a%20breach%2C%20these%20logs%20help%20you%20trace%20the%20extent%20and%20ideally%20the%20culprits.%20Knowing%20these%20exact%20details%20can%20help%20you%20prevent%20such%20events%20in%20the%20future%2C%20by%20clearly%20identifying%20where%20vulnerabilities%20were%20exploited.%20On%20the%20bright%20side%2C%20with%20alerting%20built%20on%20audit%20logs%20you%20could%20identify%20unusual%20activity%20and%20stop%20a%20breach%20early.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20a%20Postgres%20user%20you%20may%20have%20heard%20of%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.pgaudit.org%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EPostgreSQL%20Auditing%20Extension%3C%2FA%3E.%20It%20is%20more%20commonly%20known%20by%20its%20shorter%20name%3A%20pgAudit.%20How%20does%20pgAudit%20help%20you%20know%20the%20who%2C%20what%2C%20when%20of%20data%20access%3F%20Let's%20look%20at%20the%20format%20of%20pgAudit%20logs%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F134892iAD23F4ED506FDC3B%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22techcommunity-azure-blog-pgaudit-format.png%22%20title%3D%22techcommunity-azure-blog-pgaudit-format.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20see%20that%20pgAudit%20is%20very%20good%20at%20the%20'what'.%20The%20CLASS%20categorizes%20the%20kind%20of%20statement%20(READ%2C%20WRITE%2C%20DDL%20etc)%20and%20COMMAND%20tells%20you%20what%20sub-class%20it%20is.%20The%20OBJECT_TYPE%20and%20OBJECT_NAME%20identify%20whether%20a%20table%2C%20view%20etc%20was%20accessed%20and%20which%20one%20it%20was.%20The%20STATEMENT%20field%20is%20a%20key%20factor%20that%20sets%20pgAudit%20apart%20from%20using%20Postgres%20statement%20logging%20only.%20You%20may%20have%20been%20asking%2C%20why%20can't%20I%20just%20use%20log_statement%3Dall%20as%20my%20audit%20logging%3F%20It's%20because%20pgAudit%20shows%20you%20not%20only%20the%20query%20that%20was%20sent%20by%20the%20client%2C%20but%20the%20actual%20statement%20executed%20on%20the%20back%20end.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E(For%20a%20full%20breakdown%20of%20the%20log%20format%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fpgaudit%2Fpgaudit%2Fblob%2Fmaster%2FREADME.md%23format%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Evisit%20the%20pgAudit%20documentation%3C%2FA%3E).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou've%20probably%20noticed%20that%20pgAudit%20alone%20does%20not%20give%20us%20the%20who%20or%20the%20when.%20pgAudit%20relies%20on%20Postgres's%20%3CA%20href%3D%22https%3A%2F%2Fwww.postgresql.org%2Fdocs%2Fcurrent%2Fruntime-config-logging.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Elog_line_prefix%20parameter%3C%2FA%3E%20for%20this%20information.%20Log_line_prefix%20sets%20the%20initial%20text%20of%20each%20log.%20Postgres's%20default%20prefix%20is%20'%25m%20%5B%25p%5D%20'%E2%80%AFwhich%20logs%20time%20stamp%20and%20process%20ID.%20For%20auditing%2C%20it%20is%20helpful%20to%20enhance%20this%20prefix.%20For%20example%2C%20the%20setting%20%60t%3D%25m%20u%3D%25u%20db%3D%25d%20pid%3D%5B%25p%5D%3A%20%60%20gives%20you%20a%20log%20like%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Et%3D2019-10-01%2001%3A21%3A41.377%20UTC%20u%3Dalice%20db%3Dpostgres%20pid%3D%5B96%5D%3A%20LOG%3A%20AUDIT%3A%20SESSION%2C1%2C1%2CREAD%2CSELECT%2C%2C%2CSELECT%201%2C%3CNOT%20logged%3D%22%22%3E%3C%2FNOT%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELabels%20like%20'u'%20and%20'd'%20indicate%20which%20field%20is%20which%3B%20you%20can%20remove%20or%20customize%20to%20suit%20your%20taste.%20Now%20you%20can%20see%20who%20executed%20the%20statement%20(user)%2C%20when%20they%20did%20so%20(timestamp)%2C%20and%20in%20what%20database.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20process%20ID%20can%20be%20handy%20for%20tracking%20all%20events%20for%20a%20particular%20login.%20If%20you%20see%20suspicious%20activity%20you%20can%20use%20the%20process%20ID%20to%20trace%20other%20statements%20executed%20during%20that%20process.%20In%20fact%2C%20the%20process%20ID%20helps%20us%20with%20another%20important%20'w'%20-%20where%20-%20which%20is%20not%20available%20with%20pgAudit%20alone.%20By%20turning%20on%20Postgres's%20log_connections%20parameter%20(default%20is%20off)%20and%20logging%20process%20ID%2C%20you%20can%20trace%20an%20unusual%20statement%20back%20to%20the%20IP%20address%20that%20initiated%20the%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Et%3D2019-10-01%2006%3A25%3A35.766%20UTC%20u%3D%5Bunknown%5D%20db%3D%5Bunknown%5D%20pid%3D%5B108%5D%3A%20LOG%3A%20connection%20received%3A%20host%3D111.107.109.5%20port%3D5432%20pid%3D108%0At%3D2019-10-01%2006%3A25%3A36.110%20UTC%20u%3Dbob%20db%3Dpostgres%20pid%3D%5B108%5D%3A%20LOG%3A%20connection%20authorized%3A%20user%3Dbob%20database%3Dpostgres%0At%3D2019-10-01%2008%3A25%3A36.110%20UTC%20u%3Dbob%20db%3Dpostgres%20pid%3D%5B108%5D%3A%20LOG%3A%20AUDIT%3A%20SESSION%2C5%2C1%2CREAD%2CSELECT%2C%2C%2CSELECT%20*%20FROM%20accounts%3B%2C%3CNOT%20logged%3D%22%22%3E%3C%2FNOT%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20you%20would%20imagine%2C%20turning%20on%20auditing%20generates%20a%20large%20volume%20of%20logs.%20This%20is%20a%20good%20thing%2C%20but%20also%20a%20tough%20thing.%20Tough%20because%20it%20can%20be%20hard%20to%20act%20on%20that%20volume%20of%20information.%20pgAudit%20offers%20some%20settings%20that%20can%20help%20control%20the%20volume%20of%20logs%20that%20are%20generated.%20There%20are%20two%20types%20of%20logging%20and%20you%20can%20enable%20one%20or%20both%3A%20session-level%20and%20object-level%20auditing.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESession%20auditing%20logs%20all%20statements%20that%20are%20executed%20by%20users.%20You%20can%20use%20the%20pgaudit.log%20parameter%20to%20scope%20this%20down%20to%20certain%20classes%20of%20statements%3A%20READ%2C%20WRITE%2C%20FUNCTION%2C%20ROLE%2C%20DDL%2C%20MISC.%20Select%20all%20the%20classes%20or%20the%20subset%20that%20is%20most%20pertinent%20for%20you.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20the%20other%20hand%2C%20object%20auditing%20records%20statements%20for%20specific%20relations.%20For%20example%2C%20you%20can%20have%20an%20audit%20trail%20for%20a%20table%20that%20stores%20credit%20card%20information%2C%20but%20not%20have%20auditing%20for%20a%20table%20that%20stores%20non-personally%20identifiable%20data.%20(Of%20course%2C%20account%20for%20your%20compliance%20requirements%20when%20deciding).%20Using%20this%20finer%20grained%20audit%20option%20requires%20some%20additional%20setup.%20You%20need%20to%20register%20a%20Postgres%20role%20that%20has%20permissions%20on%20the%20relation%20in%20question.%20As%20new%20relations%20are%20created%2C%20you%20need%20to%20be%20sure%20you%20are%20including%20the%20relevant%20ones%20in%20the%20auditor%20role's%20permissions.%20Also%2C%20object%20auditing%20logs%20only%20selects%2C%20inserts%2C%20updates%2C%20and%20deletes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EDepending%20on%20your%20industry%2C%20it%20is%20safer%20to%20err%20on%20the%20side%20of%20logging%20more.%20Storage%20is%20not%20expensive%20these%20days.%20On%20the%20other%20hand%2C%20not%20logging%20something%20you%20later%20realize%20is%20pertinent%20can%20be%20an%20expensive%20decision.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20put%20the%20logs%20in%20long%20term%20storage%20and%20only%20come%20back%20to%20them%20after%20an%20incident%20or%20when%20external%20auditors%20visit.%20But%20there%20is%20the%20potential%20to%20do%20more%20with%20your%20audit%20logs.%20Instead%20of%20reducing%20the%20volume%20of%20logs%20generated%2C%20pipe%20them%20to%20a%20service%20that%20offers%20good%20search%20tooling%20(if%20you%E2%80%99re%20an%20Azure%20user%2C%20check%20out%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-platform-logs%23what-can-you-do-with-azure-monitor-logs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Monitor%20Logs%3C%2FA%3E).%20Search%20will%20help%20you%20quickly%20narrow%20down%20the%20pertinent%20lines%20from%20the%20thousands%20of%20files%20that%20are%20generated.%20Even%20better%2C%20if%20the%20service%20offers%20alerting%2C%20you%20can%20set%20up%20notifications%20for%20when%20certain%20classes%20of%20operations%20or%20actions%20on%20restricted%20objects%20take%20place.%20That%20can%20be%20the%20difference%20between%20you%20stopping%20a%20breach%20early%20and%20finding%20out%20about%20it%20months%20later.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2Fpgaudit%2Fpgaudit%2Fblob%2Fmaster%2FREADME.md%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EpgAudit%20documentation%3C%2FA%3E%20provides%20step%20by%20step%20guidance%20on%20how%20to%20set%20up%20the%20extension.%20If%20you're%20interested%20in%20pgAudit%20on%20Azure%20Database%20for%20PostgreSQL%2C%20visit%20our%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fpostgresql%2Fconcepts-audit%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eauditing%20article%3C%2FA%3E.%20%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2FAzureDBPostgres%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3EKeep%20in%20touch%3C%2FA%3E.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-885243%22%20slang%3D%22en-US%22%3E%3CP%3EDo%20you%20know%20who%20is%20doing%20what%20on%20your%20Postgres%20database%3F%20pgAudit%20can%20help.%3CBR%20%2F%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F134892iAD23F4ED506FDC3B%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22techcommunity-azure-blog-pgaudit-format.png%22%20title%3D%22techcommunity-azure-blog-pgaudit-format.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Microsoft

Today it's almost unsurprising to see another headline related to data breaches or data privacy. This affects you personally as a customer of those companies - you want someone to take good care of your data. It also affects you professionally; as a person who works with data, you are that 'someone' for your customers. Good audit logging is one important tool in the belt of a security-aware data professional.

 

There are different kinds of security tools. Some are preventative, like role-based access control and authentication policies. Audit logging, on the other hand, is in the investigative section of the toolbox because a log is a record of a past action. (Although knowing that their actions will be tracked may inspire employees to be more careful). Audit logs help you identify the who, what, and when of data access, and to an extent the where and how. Having audit logs available for review is a key requirement for compliance in certain regulated industries, but also increasingly in other industries as data privacy laws like GDPR take effect. If worst comes to worst and there is a breach, these logs help you trace the extent and ideally the culprits. Knowing these exact details can help you prevent such events in the future, by clearly identifying where vulnerabilities were exploited. On the bright side, with alerting built on audit logs you could identify unusual activity and stop a breach early.

 

As a Postgres user you may have heard of the PostgreSQL Auditing Extension. It is more commonly known by its shorter name: pgAudit. How does pgAudit help you know the who, what, when of data access? Let's look at the format of pgAudit logs:

 

techcommunity-azure-blog-pgaudit-format.png

 

You can see that pgAudit is very good at the 'what'. The CLASS categorizes the kind of statement (READ, WRITE, DDL etc) and COMMAND tells you what sub-class it is. The OBJECT_TYPE and OBJECT_NAME identify whether a table, view etc was accessed and which one it was. The STATEMENT field is a key factor that sets pgAudit apart from using Postgres statement logging only. You may have been asking, why can't I just use log_statement=all as my audit logging? It's because pgAudit shows you not only the query that was sent by the client, but the actual statement executed on the back end.

  

(For a full breakdown of the log format visit the pgAudit documentation).

 

You've probably noticed that pgAudit alone does not give us the who or the when. pgAudit relies on Postgres's log_line_prefix parameter for this information. Log_line_prefix sets the initial text of each log. Postgres's default prefix is '%m [%p] ' which logs time stamp and process ID. For auditing, it is helpful to enhance this prefix. For example, the setting `t=%m u=%u db=%d pid=[%p]: ` gives you a log like:

 

 

 

t=2019-10-01 01:21:41.377 UTC u=alice db=postgres pid=[96]: LOG: AUDIT: SESSION,1,1,READ,SELECT,,,SELECT 1,<not logged>

 

 

 

Labels like 'u' and 'd' indicate which field is which; you can remove or customize to suit your taste. Now you can see who executed the statement (user), when they did so (timestamp), and in what database.

 

The process ID can be handy for tracking all events for a particular login. If you see suspicious activity you can use the process ID to trace other statements executed during that process. In fact, the process ID helps us with another important 'w' - where - which is not available with pgAudit alone. By turning on Postgres's log_connections parameter (default is off) and logging process ID, you can trace an unusual statement back to the IP address that initiated the process.

 

 

 

t=2019-10-01 06:25:35.766 UTC u=[unknown] db=[unknown] pid=[108]: LOG: connection received: host=111.107.109.5 port=5432 pid=108
t=2019-10-01 06:25:36.110 UTC u=bob db=postgres pid=[108]: LOG: connection authorized: user=bob database=postgres
t=2019-10-01 08:25:36.110 UTC u=bob db=postgres pid=[108]: LOG: AUDIT: SESSION,5,1,READ,SELECT,,,SELECT * FROM accounts;,<not logged>

 

 

 

As you would imagine, turning on auditing generates a large volume of logs. This is a good thing, but also a tough thing. Tough because it can be hard to act on that volume of information. pgAudit offers some settings that can help control the volume of logs that are generated. There are two types of logging and you can enable one or both: session-level and object-level auditing.

 

Session auditing logs all statements that are executed by users. You can use the pgaudit.log parameter to scope this down to certain classes of statements: READ, WRITE, FUNCTION, ROLE, DDL, MISC. Select all the classes or the subset that is most pertinent for you.

 

On the other hand, object auditing records statements for specific relations. For example, you can have an audit trail for a table that stores credit card information, but not have auditing for a table that stores non-personally identifiable data. (Of course, account for your compliance requirements when deciding). Using this finer grained audit option requires some additional setup. You need to register a Postgres role that has permissions on the relation in question. As new relations are created, you need to be sure you are including the relevant ones in the auditor role's permissions. Also, object auditing logs only selects, inserts, updates, and deletes.

 

Depending on your industry, it is safer to err on the side of logging more. Storage is not expensive these days. On the other hand, not logging something you later realize is pertinent can be an expensive decision.

 

You can put the logs in long term storage and only come back to them after an incident or when external auditors visit. But there is the potential to do more with your audit logs. Instead of reducing the volume of logs generated, pipe them to a service that offers good search tooling (if you’re an Azure user, check out Azure Monitor Logs). Search will help you quickly narrow down the pertinent lines from the thousands of files that are generated. Even better, if the service offers alerting, you can set up notifications for when certain classes of operations or actions on restricted objects take place. That can be the difference between you stopping a breach early and finding out about it months later.

 

The pgAudit documentation provides step by step guidance on how to set up the extension. If you're interested in pgAudit on Azure Database for PostgreSQL, visit our auditing article. Keep in touch