Introduction to Network Trace Analysis 06: Kerberos it’s AUTH-some!
New to the series? Be sure to check out the previous posts! Introduction to Network Trace Analysis Part 0: Laying the Groundwork Introduction to Network Trace Analysis Part 1: Asking Questions and Collecting Data Introduction to Network Trace Analysis 2: Jumping into TCP Connectivity Introduction to Network Trace Analysis 3: TCP Performance Introduction to Network Trace Analysis 4: DNS (it's always DNS) Introduction to Network Trace Analysis 5: SMB? Sounds good to me! Howdy everyone! I'm going to cut to the chase, today we will be talking about Kerberos. Kerberos is crucial in network trace analysis as it is commonly leveraged within protocols to secure authentication across assorted services. With that said, 3, 2, 1 let's go! What is the Kerberos Protocol? Kerberos is a network protocol used to authenticate users to resources. This can be things like: Login SMB shares RDP sessions Web resources Or really anything else that supports having a Kerberos Ticket attached to it. Speaking of tickets, they are a key component of the protocol. A client will reach out to a Key Distribution Center (KDC) and (assuming all goes well) receive a Kerberos Ticket back. This initial ticket (a ticket granting ticket) can then be used to request additional Kerberos Tickets which can be used with the service you are trying to leverage. How does it work? This blog post won’t go into every detail, but this is a case where I highly recommend reading the protocol specifications for more details. The Microsoft specific implementation of Kerberos MS-KILE The Kerberos v5 RFC Got it? Good. Let's break down the legs of Kerberos authentication. Proving I am who I say I am. AS-REQ: Authentication request AS-REP: Authentication response providing the ticket granting ticket (TGT) Can I get some tickets? TGS-REQ: Ticket Granting Service (TGS) request TGS-REP: TGS Response Hey application, can I authenticate to you? AP-REQ: Application request AP-REP: Application response For folks who enjoy visual representation, here is a diagram from the MS-KILE. Walking through the legs This work will just about always take place over TCP port 88 although UDP port 88 can be used as well. Now let's get into some packet captures. 62 9.464448 172.16.1.17 172.16.1.10 TCP 66 51826 → 88 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM 63 9.467630 172.16.1.10 172.16.1.17 TCP 66 88 → 51826 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 64 9.467729 172.16.1.17 172.16.1.10 TCP 54 51826 → 88 [ACK] Seq=1 Ack=1 Win=262656 Len=0 65 9.467768 172.16.1.17 172.16.1.10 KRB5 281 AS-REQ 66 9.471645 172.16.1.10 172.16.1.17 KRB5 239 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 67 9.471724 172.16.1.17 172.16.1.10 TCP 54 51826 → 88 [FIN, ACK] Seq=228 Ack=186 Win=262400 Len=0 68 9.471938 172.16.1.17 172.16.1.10 TCP 66 51827 → 88 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM 69 9.475638 172.16.1.10 172.16.1.17 TCP 60 88 → 51826 [ACK] Seq=186 Ack=229 Win=2097664 Len=0 70 9.475638 172.16.1.10 172.16.1.17 TCP 60 88 → 51826 [RST, ACK] Seq=186 Ack=229 Win=0 Len=0 71 9.475638 172.16.1.10 172.16.1.17 TCP 66 88 → 51827 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 72 9.475750 172.16.1.17 172.16.1.10 KRB5 361 AS-REQ 73 9.481665 172.16.1.10 172.16.1.17 TCP 1514 88 → 51827 [ACK] Seq=1 Ack=308 Win=2097664 Len=1460 [TCP segment of a reassembled PDU] 74 9.481665 172.16.1.10 172.16.1.17 KRB5 178 AS-REP 75 9.481716 172.16.1.17 172.16.1.10 TCP 54 51827 → 88 [ACK] Seq=308 Ack=1585 Win=262656 Len=0 76 9.481750 172.16.1.17 172.16.1.10 TCP 54 51827 → 88 [FIN, ACK] Seq=308 Ack=1585 Win=262656 Len=0 You may have noticed that the first AS-REQ gets a response of KRB5KDC_ERR_PREAUTH_REQUIRED. This is the KDC saying "I need you to prove who you are" and in the subsequent AS-REQ the client provides that information in the form of the pA-ENC-TIMESTAMP: // Without the pre-auth data Frame 65: 281 bytes on wire (2248 bits), 281 bytes captured (2248 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:07 (00:15:5d:01:2b:07), Dst: Microsoft_01:2b:01 (00:15:5d:01:2b:01) Internet Protocol Version 4, Src: 172.16.1.17, Dst: 172.16.1.10 Transmission Control Protocol, Src Port: 51826, Dst Port: 88, Seq: 1, Ack: 1, Len: 227 Kerberos Record Mark: 223 bytes as-req pvno: 5 msg-type: krb-as-req (10) padata: 1 item PA-DATA pA-PAC-REQUEST padata-type: pA-PAC-REQUEST (128) padata-value: 3005a0030101ff include-pac: True req-body // With the pre-auth data Frame 72: 361 bytes on wire (2888 bits), 361 bytes captured (2888 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:07 (00:15:5d:01:2b:07), Dst: Microsoft_01:2b:01 (00:15:5d:01:2b:01) Internet Protocol Version 4, Src: 172.16.1.17, Dst: 172.16.1.10 Transmission Control Protocol, Src Port: 51827, Dst Port: 88, Seq: 1, Ack: 1, Len: 307 Kerberos Record Mark: 303 bytes as-req pvno: 5 msg-type: krb-as-req (10) padata: 2 items PA-DATA pA-ENC-TIMESTAMP padata-type: pA-ENC-TIMESTAMP (2) padata-value: 3041a003020112a23a0438bfc3a7dcf5d5db568884edef5dcbcfbd67c1f17b4d9feea596b68d4dd04fe3e2cbac4399aabd3263c4adde1d498d09959a82c84721227265 etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) cipher: bfc3a7dcf5d5db568884edef5dcbcfbd67c1f17b4d9feea596b68d4dd04fe3e2cbac4399aabd3263c4adde1d498d09959a82c84721227265 PA-DATA pA-PAC-REQUEST req-body The pa-ENC-TIMESTAMP is the client’s time, encrypted with a hash of the client’s password. The KDC can verify the authenticity of the requestor by decrypting the timestamp. In the case of the capture above, the AS-REQ was successful, and we received an AS-REP back. Here is the ticket that was returned: Frame 74: 178 bytes on wire (1424 bits), 178 bytes captured (1424 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:01 (00:15:5d:01:2b:01), Dst: Microsoft_01:2b:07 (00:15:5d:01:2b:07) Internet Protocol Version 4, Src: 172.16.1.10, Dst: 172.16.1.17 Transmission Control Protocol, Src Port: 88, Dst Port: 51827, Seq: 1461, Ack: 308, Len: 124 [2 Reassembled TCP Segments (1584 bytes): #73(1460), #74(124)] Kerberos Record Mark: 1580 bytes as-rep pvno: 5 msg-type: krb-as-rep (11) padata: 1 item crealm: CONTOSO.COM cname name-type: kRB5-NT-PRINCIPAL (1) cname-string: 1 item CNameString: will ticket tkt-vno: 5 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: krbtgt SNameString: CONTOSO.COM enc-part etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) kvno: 2 cipher [truncated]: 2bae323b2bcf3b2235902be41e2aefe74d6137420e35043c3b53f7de24f1bb3362f357cff5744fcd144085c49486ae07930df6da4938292939209a6ff56110fcc3d7c4de03edbf4284268985e6d2c19bd4e917d5b7308370240f1fba0d844bb7ac03ba2561c753affb13da9d9b3 enc-part With this response, we can see that the user Will authenticated and received a TGT referencing the krbtgt account. With TGT in hand, the client can subsequently request tickets to other services. In our example, the client is trying to authenticate to an SMB share and will need a service ticket: 55 9.452560 172.16.1.17 172.16.1.18 TCP 66 51825 → 445 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM 56 9.455440 172.16.1.18 172.16.1.17 TCP 66 445 → 51825 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 57 9.455552 172.16.1.17 172.16.1.18 TCP 54 51825 → 445 [ACK] Seq=1 Ack=1 Win=262656 Len=0 58 9.455596 172.16.1.17 172.16.1.18 SMB 127 Negotiate Protocol Request 59 9.459426 172.16.1.18 172.16.1.17 SMB2 306 Negotiate Protocol Response 60 9.459503 172.16.1.17 172.16.1.18 SMB2 342 Negotiate Protocol Request // This response is what prompts us to proceed onto the Kerberos authentication 61 9.463426 172.16.1.18 172.16.1.17 SMB2 430 Negotiate Protocol Response 65 9.467768 172.16.1.17 172.16.1.10 KRB5 281 AS-REQ 66 9.471645 172.16.1.10 172.16.1.17 KRB5 239 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 72 9.475750 172.16.1.17 172.16.1.10 KRB5 361 AS-REQ 74 9.481665 172.16.1.10 172.16.1.17 KRB5 178 AS-REP // Here is our TGS-REQ 82 9.485807 172.16.1.17 172.16.1.10 KRB5 245 TGS-REQ 85 9.495715 172.16.1.10 172.16.1.17 KRB5 207 TGS-REP Breaking down the TGS-REQ: Frame 82: 245 bytes on wire (1960 bits), 245 bytes captured (1960 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:07 (00:15:5d:01:2b:07), Dst: Microsoft_01:2b:01 (00:15:5d:01:2b:01) Internet Protocol Version 4, Src: 172.16.1.17, Dst: 172.16.1.10 Transmission Control Protocol, Src Port: 51828, Dst Port: 88, Seq: 1461, Ack: 1, Len: 191 [2 Reassembled TCP Segments (1651 bytes): #81(1460), #82(191)] Kerberos Record Mark: 1647 bytes tgs-req pvno: 5 msg-type: krb-tgs-req (12) padata: 2 items PA-DATA pA-TGS-REQ padata-type: pA-TGS-REQ (1) padata-value [truncated]: 6e82051c30820518a003020105a10302010ea20703050000000000a382046c6182046830820464a003020105a10d1b0b434f4e544f534f2e434f4da220301ea003020102a11730151b066b72627467741b0b434f4e544f534f2e434f4da382042a30820426a003020112a ap-req pvno: 5 msg-type: krb-ap-req (14) Padding: 0 ap-options: 00000000 // Here is the ticket that we are attaching to the `TGS-REQ` to prove we are allowed to request this ticket ticket tkt-vno: 5 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: krbtgt SNameString: CONTOSO.COM enc-part authenticator PA-DATA pA-PAC-OPTIONS req-body Padding: 0 kdc-options: 40810000 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) // Here is the service that we are trying to get a ticket for cifs/MB01.contoso.com sname-string: 2 items SNameString: cifs SNameString: MB01.contoso.com till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time nonce: 1281991308 etype: 5 items enc-authorization-data We can see how the TGT was glued into the TGS-REQ proving who we are and that we are allowed to request a ticket. Frame 85: 207 bytes on wire (1656 bits), 207 bytes captured (1656 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:01 (00:15:5d:01:2b:01), Dst: Microsoft_01:2b:07 (00:15:5d:01:2b:07) Internet Protocol Version 4, Src: 172.16.1.10, Dst: 172.16.1.17 Transmission Control Protocol, Src Port: 88, Dst Port: 51828, Seq: 1461, Ack: 1652, Len: 153 [2 Reassembled TCP Segments (1613 bytes): #84(1460), #85(153)] Kerberos Record Mark: 1609 bytes tgs-rep pvno: 5 msg-type: krb-tgs-rep (13) crealm: CONTOSO.COM cname name-type: kRB5-NT-PRINCIPAL (1) cname-string: 1 item CNameString: will ticket tkt-vno: 5 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: cifs SNameString: MB01.contoso.com enc-part enc-part And in the response above, nice and simple, confirms that Will is in fact getting a service ticket for cifs/MB01.contoso.com. With the service ticket in hand, we can proceed onto the AP-REQ. This gets a bit trickier since it is typically embedded in the protocol of the service we are trying to leverage. In this case, SMB. Frame 89: 467 bytes on wire (3736 bits), 467 bytes captured (3736 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:07 (00:15:5d:01:2b:07), Dst: Microsoft_01:2b:08 (00:15:5d:01:2b:08) Internet Protocol Version 4, Src: 172.16.1.17, Dst: 172.16.1.18 Transmission Control Protocol, Src Port: 51825, Dst Port: 445, Seq: 1822, Ack: 629, Len: 413 [2 Reassembled TCP Segments (1873 bytes): #88(1460), #89(413)] NetBIOS Session Service SMB2 (Server Message Block Protocol version 2) SMB2 Header Session Setup Request (0x01) [Preauth Hash: 4dae9574f1c7f3e9acc9c4425ca4bd33d95368c79095a19d999eed29527656dc3b3fc4298f21ff115d45977dc234366678297134e7a1c1cc46fb259f4f430e4c] StructureSize: 0x0019 Flags: 0 Security mode: 0x01, Signing enabled Capabilities: 0x00000001, DFS Channel: None (0x00000000) Previous Session Id: 0x0000000000000000 Blob Offset: 0x00000058 Blob Length: 1781 Security Blob [truncated]: 608206f106062b0601050502a08206e5308206e1a030302e06092a864882f71201020206092a864886f712010202060a2b06010401823702021e060a2b06010401823702020aa28206ab048206a7608206a306092a864886f71201020201006e8206923082068ea00302 GSS-API Generic Security Service Application Program Interface OID: 1.3.6.1.5.5.2 (SPNEGO - Simple Protected Negotiation) Simple Protected Negotiation negTokenInit mechTypes: 4 items mechToken [truncated]: 608206a306092a864886f71201020201006e8206923082068ea003020105a10302010ea20703050020000000a38204d1618204cd308204c9a003020105a10d1b0b434f4e544f534f2e434f4da2233021a003020102a11a30181b04636966731b104d4230312e636f6e746f73 krb5_blob [truncated]: 608206a306092a864886f71201020201006e8206923082068ea003020105a10302010ea20703050020000000a38204d1618204cd308204c9a003020105a10d1b0b434f4e544f534f2e434f4da2233021a003020102a11a30181b04636966731b104d4230312e636f6e746f73 KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_tok_id: KRB5_AP_REQ (0x0001) Kerberos ap-req pvno: 5 msg-type: krb-ap-req (14) Padding: 0 ap-options: 20000000 ticket tkt-vno: 5 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: cifs SNameString: MB01.contoso.com enc-part authenticator Within the SMB session setup request, the Kerberos AP-REQ is embedded including our service ticket for cifs/MB01.contoso.com. And as a part of the session setup response: Frame 93: 314 bytes on wire (2512 bits), 314 bytes captured (2512 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:08 (00:15:5d:01:2b:08), Dst: Microsoft_01:2b:07 (00:15:5d:01:2b:07) Internet Protocol Version 4, Src: 172.16.1.18, Dst: 172.16.1.17 Transmission Control Protocol, Src Port: 445, Dst Port: 51825, Seq: 629, Ack: 2235, Len: 260 NetBIOS Session Service SMB2 (Server Message Block Protocol version 2) SMB2 Header Session Setup Response (0x01) [Preauth Hash: 4dae9574f1c7f3e9acc9c4425ca4bd33d95368c79095a19d999eed29527656dc3b3fc4298f21ff115d45977dc234366678297134e7a1c1cc46fb259f4f430e4c] StructureSize: 0x0009 Session Flags: 0x0000 Blob Offset: 0x00000048 Blob Length: 184 Security Blob [truncated]: a181b53081b2a0030a0100a10b06092a864882f712010202a2819d04819a60819706092a864886f71201020202006f8187308184a003020105a10302010fa2783076a003020112a26f046d4fb5f35670d74463f524d38a6355511a5ab0a3bd4808a4a20ce1514bcf2e2d GSS-API Generic Security Service Application Program Interface Simple Protected Negotiation negTokenTarg negResult: accept-completed (0) supportedMech: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5) responseToken [truncated]: 60819706092a864886f71201020202006f8187308184a003020105a10302010fa2783076a003020112a26f046d4fb5f35670d74463f524d38a6355511a5ab0a3bd4808a4a20ce1514bcf2e2d56f237b1ae1938390c6ca34ce689edeabdfd240b5aa2691a52cbcd235f0b krb5_blob [truncated]: 60819706092a864886f71201020202006f8187308184a003020105a10302010fa2783076a003020112a26f046d4fb5f35670d74463f524d38a6355511a5ab0a3bd4808a4a20ce1514bcf2e2d56f237b1ae1938390c6ca34ce689edeabdfd240b5aa2691a52cbcd235f0ba2c6 KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_tok_id: KRB5_AP_REP (0x0002) Kerberos ap-rep pvno: 5 msg-type: krb-ap-rep (15) enc-part No errors, we are good to go! Pretty cool right? The Kerberos Ticket Cache The tickets we are receiving have many important fields, but I want to focus on the timing fields. Frame 72: 361 bytes on wire (2888 bits), 361 bytes captured (2888 bits) on interface \Device\NPF_{F26B04EB-93FE-45B6-8E1F-7DED5BBC122C}, id 0 Ethernet II, Src: Microsoft_01:2b:07 (00:15:5d:01:2b:07), Dst: Microsoft_01:2b:01 (00:15:5d:01:2b:01) Internet Protocol Version 4, Src: 172.16.1.17, Dst: 172.16.1.10 Transmission Control Protocol, Src Port: 51827, Dst Port: 88, Seq: 1, Ack: 1, Len: 307 Kerberos Record Mark: 303 bytes as-req pvno: 5 msg-type: krb-as-req (10) padata: 2 items req-body <snip> till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time rtime: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time <snip> The till and rtime fields indicate how long the ticket is valid for. In my case, the ticket is valid until Sept 12, 2100 (which is shorthand for this time or your local policy. Whichever comes first. Since the ticket has a valid lifetime, it will be added to my Kerberos Ticket cache. We can interrogate the state of this cache by using the klist executable. PS C:\> klist Current LogonId is 0:0xb72981 Cached Tickets: (2) #0> Client: will @ CONTOSO.COM Server: krbtgt/CONTOSO.COM @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize Start Time: 5/28/2024 8:26:11 (local) End Time: 5/28/2024 18:26:11 (local) Renew Time: 6/4/2024 8:26:11 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0x1 -> PRIMARY Kdc Called: DC01.contoso.com #1> Client: will @ CONTOSO.COM Server: cifs/MB01.contoso.com @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize Start Time: 5/28/2024 8:26:11 (local) End Time: 5/28/2024 18:26:11 (local) Renew Time: 6/4/2024 8:26:11 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0 Kdc Called: DC01.contoso.com Yep! Here are the tickets I requested, sitting in my cache. Now let's jump into some scenarios. Bad Password Ring ring, you are sitting at your desk on a lovely Tuesday afternoon when one of your Sysadmins walks up to your desk and is saying "Open a bug with Microsoft. The Kerberos cache is broken." They go on to explain: I make a WMI connection to srv.contoso.com I see I have a ticket in the Kerberos cache for host/srv.contoso.com I am getting a new ticket. What the heck is going on?!? Let's start with some good questions. Q: When did you first notice this? A: Earlier today but looking at some historic data it looks like the Domain Controllers (DCs) have seen an influx in traffic since the beginning of last month. Q: Can you reproduce this behavior at will? A: Yep! It is extremely easy Q: How do we reproduce this? A: Here is a repro script klist purge $s = New-PSSession -ComputerName srv.contoso.com -Authentication Kerberos klist Remove-PSSession $s $s = New-PSSession -ComputerName srv.contoso.com -Authentication Kerberos Q: Why is this important? A: Nothing is failing, but it looks like all our Windows clients are doing this leading to an uptick in CPU utilization on our DCs. Good enough for me, let's look at the traffic from the client and see what we can see. Here is the Kerberos traffic in the capture on TCP port 88. 124 5.342076 10.160.207.41 10.160.207.26 KRB5 270 AS-REQ 125 5.342832 10.160.207.26 10.160.207.41 KRB5 331 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 132 5.343449 10.160.207.41 10.160.207.26 KRB5 350 AS-REQ 133 5.344254 10.160.207.26 10.160.207.41 KRB5 1662 AS-REP 141 5.345269 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ 143 5.346211 10.160.207.26 10.160.207.41 KRB5 1729 TGS-REP 538 20.365347 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ 540 20.366688 10.160.207.26 10.160.207.41 KRB5 1729 TGS-REP We have two sets of TGS requests and responses. What are they for? 141 5.345269 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ Frame 141: 1729 bytes on wire (13832 bits), 1729 bytes captured (13832 bits) on interface \Device\NPF_{C5D00D90-485A-4651-A1E8-48BC9ECC7CD6}, id 0 Ethernet II, Src: Microsoft_88:41:34 (00:15:5d:88:41:34), Dst: Microsoft_88:41:33 (00:15:5d:88:41:33) Internet Protocol Version 4, Src: 10.160.207.41, Dst: 10.160.207.26 Transmission Control Protocol, Src Port: 55359, Dst Port: 88, Seq: 1, Ack: 1, Len: 1675 Kerberos Record Mark: 1671 bytes tgs-req pvno: 5 msg-type: krb-tgs-req (12) padata: 2 items req-body Padding: 0 kdc-options: 40810000 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: HOST SNameString: srv.contoso.com till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time nonce: 2055763826 etype: 5 items enc-authorization-data [Response in: 143] 538 20.365347 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ Frame 538: 1729 bytes on wire (13832 bits), 1729 bytes captured (13832 bits) on interface \Device\NPF_{C5D00D90-485A-4651-A1E8-48BC9ECC7CD6}, id 0 Ethernet II, Src: Microsoft_88:41:34 (00:15:5d:88:41:34), Dst: Microsoft_88:41:33 (00:15:5d:88:41:33) Internet Protocol Version 4, Src: 10.160.207.41, Dst: 10.160.207.26 Transmission Control Protocol, Src Port: 55364, Dst Port: 88, Seq: 1, Ack: 1, Len: 1675 Kerberos Record Mark: 1671 bytes tgs-req pvno: 5 msg-type: krb-tgs-req (12) padata: 2 items req-body Padding: 0 kdc-options: 40810000 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: HOST SNameString: srv.contoso.com till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time nonce: 683875312 etype: 5 items enc-authorization-data [Response in: 540] Weird. We are requesting the same ticket twice. Why is that? What are we doing with this ticket? It's time to look at the third leg of Kerberos communication. The AP request and response. The message type for an AP-REQ is 14 and AS-REP is 15. With this in mind, we use the following filter in Wireshark to look for what we want kerberos.msg_type == 14 || kerberos.msg_type == 15. This is what we can see: 141 5.345269 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ 151 5.348214 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 153 5.352337 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 163 5.416425 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 165 5.419429 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 188 5.491898 10.160.207.41 10.160.207.42 HTTP 2638 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 190 5.494668 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 532 20.345396 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 538 20.365347 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ 545 20.367420 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 547 20.370774 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 558 20.432506 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 560 20.435330 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 582 20.511828 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 584 20.514729 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 We can see a pattern of a POST followed by an HTTP 200 except for one POST which does not have an AP-REP. Why? Let's filter to that TCP conversation. PRO TIP You can right click and filter the conversation! After applying this filter, we have a better sense of what is going on: 529 20.344846 10.160.207.41 10.160.207.42 TCP 66 55363 → 5985 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 530 20.345254 10.160.207.42 10.160.207.41 TCP 66 5985 → 55363 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 531 20.345313 10.160.207.41 10.160.207.42 TCP 54 55363 → 5985 [ACK] Seq=1 Ack=1 Win=65280 Len=0 532 20.345396 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 Frame 532: 2642 bytes on wire (21136 bits), 2642 bytes captured (21136 bits) on interface \Device\NPF_{C5D00D90-485A-4651-A1E8-48BC9ECC7CD6}, id 0 Ethernet II, Src: Microsoft_88:41:34 (00:15:5d:88:41:34), Dst: Microsoft_88:41:35 (00:15:5d:88:41:35) Internet Protocol Version 4, Src: 10.160.207.41, Dst: 10.160.207.42 Transmission Control Protocol, Src Port: 55363, Dst Port: 5985, Seq: 1, Ack: 1, Len: 2588 Hypertext Transfer Protocol POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1\r\n Connection: Keep-Alive\r\n Content-Type: application/soap+xml;charset=UTF-8\r\n […]Authorization: Kerberos YIIG4QYJKoZIhvcSAQICAQBuggbQMIIGzKADAgEFoQMCAQ6iBwMFACAAAACjggUQYYIFDDCCBQigAwIBBaENGwtDT05UT1NPLkNPTaIiMCCgAwIBAqEZMBcbBEhPU1QbD3Nydi5jb250b3NvLmNvbaOCBMwwggTIoAMCARKhAwIBLqKCBLoEggS23WKZXRjsQZxpXt+Fa6ZwGkm6p GSS-API Generic Security Service Application Program Interface OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_blob […]: 01006e8206d0308206cca003020105a10302010ea20703050020000000a38205106182050c30820508a003020105a10d1b0b434f4e544f534f2e434f4da2223020a003020102a11930171b04484f53541b0f7372762e636f6e746f736f2e636f6da38204cc308204c8a003020112a1 krb5_tok_id: KRB5_AP_REQ (0x0001) Kerberos ap-req pvno: 5 msg-type: krb-ap-req (14) Padding: 0 ap-options: 20000000 ticket tkt-vno: 5 realm: CONTOSO.COM sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: HOST SNameString: srv.contoso.com enc-part etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) kvno: 46 cipher […]: dd62995d18ec419c695edf856ba6701a49baa4a659e56c413530688ab23f3de12d505d5ca8a40bf6c1a75350e5761f77ec53e27f46186598216f3a9dba122fabb7a3d75d7b53a67bbceabac23ba45f98b77b880f29c90f60759224845b7234a581b15a864154f446abfea7115bb25782e authenticator etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) cipher […]: d149108d35081895b10296ba928297dd2216b6850e41a7c198869c0345415395c96b38698312e84a0656c0654da590a039ff6a29a70453fa08308fd2a40fe59f077f5bac33b573a1315522a0fdea0e0647600512f758c62b3b5be9612aa5ac997bc9786f19131416ee5b2012fc005fb8e User-Agent: Microsoft WinRM Client\r\n Content-Length: 0\r\n Host: srv.contoso.com:5985\r\n \r\n [Response in frame: 534] [Full request URI: http://srv.contoso.com:5985/wsman?PSVersion=5.1.27714.1000] 534 20.363551 10.160.207.42 10.160.207.41 HTTP 315 KRB Error: KRB5KRB_AP_ERR_MODIFIED Frame 534: 315 bytes on wire (2520 bits), 315 bytes captured (2520 bits) on interface \Device\NPF_{C5D00D90-485A-4651-A1E8-48BC9ECC7CD6}, id 0 Ethernet II, Src: Microsoft_88:41:35 (00:15:5d:88:41:35), Dst: Microsoft_88:41:34 (00:15:5d:88:41:34) Internet Protocol Version 4, Src: 10.160.207.42, Dst: 10.160.207.41 Transmission Control Protocol, Src Port: 5985, Dst Port: 55363, Seq: 1, Ack: 2589, Len: 261 Hypertext Transfer Protocol HTTP/1.1 401 \r\n WWW-Authenticate: Kerberos YFwGCSqGSIb3EgECAgMAfk0wS6ADAgEFoQMCAR6kERgPMjAyNDEwMDMxOTAxMTVapQUCAwVj06YDAgEpqQ0bC0NPTlRPU08uQ09NqhEwD6ADAgEBoQgwBhsEc3J2JA==\r\n GSS-API Generic Security Service Application Program Interface OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_blob: 03007e4d304ba003020105a10302011ea411180f32303234313030333139303131355aa50502030563d3a603020129a90d1b0b434f4e544f534f2e434f4daa11300fa003020101a10830061b0473727624 krb5_tok_id: KRB5_ERROR (0x0003) Kerberos krb-error pvno: 5 msg-type: krb-error (30) stime: Oct 3, 2024 15:01:15.000000000 Eastern Daylight Time susec: 353235 error-code: eRR-MODIFIED (41) realm: CONTOSO.COM sname name-type: kRB5-NT-PRINCIPAL (1) sname-string: 1 item SNameString: srv$ Server: Microsoft-HTTPAPI/2.0\r\n Date: Thu, 03 Oct 2024 19:01:15 GMT\r\n Content-Length: 0\r\n \r\n [Request in frame: 532] [Time since request: 0.018155000 seconds] [Request URI: /wsman?PSVersion=5.1.27714.1000] [Full request URI: http://srv.contoso.com:5985/wsman?PSVersion=5.1.27714.1000] Interesting... We sent the Kerberos Ticket, and when we re-used the ticket, we received back a response of KRB_AP_ERR_MODIFIED. Checking the Microsoft Open Specification for Kerberos MS-KILE. 3.4.5 Message Processing Events and Sequencing Rules If the decryption of the ticket fails and the KILE server has older versions of the server key, the server retries decrypting the ticket with the older keys. If the decryption routines detect a modification of the ticket, the KRB_AP_ERR_MODIFIED error message is returned. If decryption shows that the authenticator has been modified, the KRB_AP_ERR_MODIFIED error message is returned. ... If the service ticket received for the computer's principal is encrypted with DES, the KILE server MUST return KRB_AP_ERR_MODIFIED regardless of supporting DES.<79> We can cut out that last point since we can clearly see from the packet capture that both the authenticator and the attached ticket are encrypted using AES256-SHA1. And the others, while none of these are explicit call outs for the cause of the issue, the common theme here is issues decrypting the ticket that is sent in the AP-REQ. That is interesting, especially since we know there is no functionality breakage. Meaning, we can decrypt the ticket in the following request. Let's remove our filter and get a better sense of what is going on currently. 530 20.345254 10.160.207.42 10.160.207.41 TCP 66 5985 → 55363 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 531 20.345313 10.160.207.41 10.160.207.42 TCP 54 55363 → 5985 [ACK] Seq=1 Ack=1 Win=65280 Len=0 532 20.345396 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 533 20.345500 10.160.207.42 10.160.207.41 TCP 54 5985 → 55363 [ACK] Seq=1 Ack=2589 Win=65280 Len=0 534 20.363551 10.160.207.42 10.160.207.41 HTTP 315 KRB Error: KRB5KRB_AP_ERR_MODIFIED 535 20.364980 10.160.207.41 10.160.207.26 TCP 66 55364 → 88 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 536 20.365266 10.160.207.26 10.160.207.41 TCP 66 88 → 55364 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 537 20.365325 10.160.207.41 10.160.207.26 TCP 54 55364 → 88 [ACK] Seq=1 Ack=1 Win=65280 Len=0 538 20.365347 10.160.207.41 10.160.207.26 KRB5 1729 TGS-REQ 539 20.365553 10.160.207.26 10.160.207.41 TCP 54 88 → 55364 [ACK] Seq=1 Ack=1676 Win=65280 Len=0 540 20.366688 10.160.207.26 10.160.207.41 KRB5 1729 TGS-REP 541 20.366720 10.160.207.41 10.160.207.26 TCP 54 55364 → 88 [ACK] Seq=1676 Ack=1676 Win=65280 Len=0 542 20.366764 10.160.207.41 10.160.207.26 TCP 54 55364 → 88 [FIN, ACK] Seq=1676 Ack=1676 Win=65280 Len=0 543 20.366961 10.160.207.26 10.160.207.41 TCP 54 88 → 55364 [ACK] Seq=1676 Ack=1677 Win=65280 Len=0 544 20.366961 10.160.207.26 10.160.207.41 TCP 54 88 → 55364 [RST, ACK] Seq=1676 Ack=1677 Win=0 Len=0 545 20.367420 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 546 20.367723 10.160.207.42 10.160.207.41 TCP 54 5985 → 55363 [ACK] Seq=262 Ack=5177 Win=65280 Len=0 547 20.370774 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 Ah... I see. Between the response of KRB_AP_ERR_MODIFIED and the successful authentication, a new TGS-REQ is going out which will be newly encrypted with the password of the service. If that is the case and that ticket is successful, we can surmise that the old ticket was NOT encrypted with the service account password OR was encrypted with an old version of the service account password. To get a more holistic understanding, let’s look at the server side of the connection. When filtering on Kerberos, we can see the following. 989 16.905025 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 991 16.909584 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 1001 16.982557 10.160.207.41 10.160.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 1003 16.985137 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 1033 17.070445 10.160.207.41 10.16.207.42 HTTP 2642 POST /wsman?PSVersion=5.1.27714.1000 HTTP/1.1 1035 17.072923 10.160.207.42 10.160.207.41 HTTP 395 HTTP/1.1 200 1387 23.487065 10.160.207.42 10.160.207.26 KRB5 1723 TGS-REQ 1389 23.488214 10.160.207.26 10.160.207.42 KRB5 1655 TGS-REP 1398 23.490324 10.160.207.42 10.160.207.26 KPASSWD 1748 Request 1402 23.515327 10.160.207.26 10.160.207.42 KPASSWD 155 1499 24.392837 10.160.207.42 10.160.207.26 KRB5 1723 TGS-REQ 1501 24.394083 10.160.207.26 10.160.207.42 KRB5 1655 TGS-REP 1511 24.396372 10.160.207.42 10.160.207.26 KPASSWD 1748 Request 1521 24.425829 10.160.207.26 10.160.207.42 KPASSWD 155 1640 25.670504 10.160.207.42 10.160.207.26 KRB5 1723 TGS-REQ 1642 25.671599 10.160.207.26 10.160.207.42 KRB5 1655 TGS-REP 1650 25.673386 10.160.207.42 10.160.207.26 KPASSWD 1748 Request 1659 25.702341 10.160.207.26 10.160.207.42 KPASSWD 155 Oh? What is this? KPASSWD seems interesting. After doing a mountain of research here is what we can find: KPASSWD is a protocol for changing Kerberos passwords (Duh) It is one of the mechanisms used to update Windows Passwords Given this information, along with the knowledge that a Kerberos ticket that was valid mere seconds earlier now being unable to be decrypted leads me to believe that someone or something is changing the password regularly. There is an automatic password change every 30 days by default so, this doesn't make sense to be the culprit. It is time to start asking our colleagues. "Has anyone done any security 'hardening' on srv.contoso.com recently?" you ask. The InfoSec lead mentions that an intern had a project idea to increase the security of the machine account password, and he automated it with a script. Looking at the script you can see it is the following: while($true) { Reset-ComputerMachinePassword -Server DC Reset-ComputerMachinePassword -Server DC Reset-ComputerMachinePassword -Server DC Start-Sleep -Seconds 3 } That would do it... Since we are authenticating to the machine service account, when the machine account password is changed 3 times, we can no longer leverage the old password for authentication. When asked why, the intern stated that "if the password is constantly changing, we are resistant to Kerberoasting since the compromised password will not be valid!" …After removing the script, the issue disappears. No supported encryption types Another beautiful day, another interesting problem. A support ticket comes in stating that over the weekend, a backup service is failing. Beginning with our questions: Q: "What changed over the weekend?" A: "We implemented some security hardening." Q: "What security hardening?" A: "I don't know. The Sec team did that, and they are at a conference this week." Q: "Can this wait for them to get back?" A: "No." Cool... With that information in mind, let’s start by trying to reproduce our behavior. The ticket filer mentioned that the resource name is "app.contoso.com". Testing from my own machine, I can see the following. PS C:\> dir \\app.contoso.com\backups Directory: \\app.contoso.com\backups Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 5/19/2025 1:11 PM 0 BACKUP_0.bin -a---- 5/19/2025 1:11 PM 0 BACKUP_1.bin This seems to have worked... After chatting with the ticket filer some more, we found out that the failure only occurred from machines in the production site. We log into a machine from the production site and attempt the behavior again. PS C:\> dir \\app.contoso.com\backups dir : Cannot find path '\\app.contoso.com\backups' because it does not exist. At line:1 char:1 + dir \\app.contoso.com\backups + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (\\app.contoso.com\backups:String) [Get-ChildItem], ItemNotFoundException + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand Now that is a failure! Unfortunately, not with a ton of detail. What happens if we try this from another view like net.exe? PS C:\> net use \\app.contoso.com\backups The password is invalid for \\app.contoso.com\backups. Enter the user name for 'app.contoso.com': This is more interesting. If we are prompted for credentials, then we are trying to use NTLM instead of Kerberos. Why is that? Do I have anything in my Kerberos Ticket Cache? PS C:\> klist Current LogonId is 0:0xe5289 Cached Tickets: (2) #0> Client: Administrator @ CONTOSO.COM Server: krbtgt/CONTOSO.COM @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize Start Time: 5/19/2025 14:11:03 (local) End Time: 5/20/2025 0:11:03 (local) Renew Time: 5/26/2025 14:11:03 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0x1 -> PRIMARY Kdc Called: DC1 #1> Client: Administrator @ CONTOSO.COM Server: ldap/DC1.contoso.com @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize Start Time: 5/19/2025 14:11:04 (local) End Time: 5/20/2025 0:11:03 (local) Renew Time: 5/26/2025 14:11:03 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0 Kdc Called: DC1.contoso.com I do. Which means that Kerberos isn't entirely broken; maybe the issue is just with the backup service. What happens if I try to manually get a Kerberos Ticket to that endpoint? PS C:\> klist get CIFS/app.contoso.com Current LogonId is 0:0xe5289 Error calling API LsaCallAuthenticationPackage (GetTicket substatus): 0x80090342 klist failed with 0xc00002fd/-1073741059: The encryption type requested is not supported by the KDC. We have a response that the encryption type (etype) isn't supported by the KDC? What? That doesn’t make sense; the Sec team didn’t make any changes to the KDC. Time to capture some traces and see what we can see. Starting with a capture of the failure accessing the share from the client: 85 18:17:40.614031 172.24.196.28 172.24.203.47 TCP 66 59072 → 445 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 86 18:17:40.617565 172.24.203.47 172.24.196.28 TCP 66 445 → 59072 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 87 18:17:40.617648 172.24.196.28 172.24.203.47 TCP 54 59072 → 445 [ACK] Seq=1 Ack=1 Win=65280 Len=0 88 18:17:40.617681 172.24.196.28 172.24.203.47 SMB 127 Negotiate Protocol Request 89 18:17:40.619558 172.24.203.47 172.24.196.28 SMB2 315 Negotiate Protocol Response 90 18:17:40.619628 172.24.196.28 172.24.203.47 SMB2 350 Negotiate Protocol Request 91 18:17:40.621566 172.24.203.47 172.24.196.28 SMB2 446 Negotiate Protocol Response 95 18:17:40.628425 172.24.196.28 172.24.196.47 KRB5 276 AS-REQ 96 18:17:40.630074 172.24.196.47 172.24.196.28 KRB5 326 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 99 18:17:40.631883 172.24.196.28 172.24.203.47 TCP 54 59072 → 445 [ACK] Seq=370 Ack=654 Win=64768 Len=0 103 18:17:40.634219 172.24.196.28 172.24.196.47 KRB5 356 AS-REQ 105 18:17:40.636230 172.24.196.47 172.24.196.28 KRB5 301 AS-REP 114 18:17:40.643760 172.24.196.28 172.24.196.47 KRB5 360 TGS-REQ 116 18:17:40.647681 172.24.196.47 172.24.196.28 KRB5 152 KRB Error: KRB5KDC_ERR_ETYPE_NOSUPP 124 18:17:40.653848 172.24.196.28 172.24.196.47 KRB5 360 TGS-REQ 126 18:17:40.658033 172.24.196.47 172.24.196.28 KRB5 152 KRB Error: KRB5KDC_ERR_ETYPE_NOSUPP 128 18:17:40.659640 172.24.196.28 172.24.203.47 TCP 54 59072 → 445 [RST, ACK] Seq=370 Ack=654 Win=0 Len=0 // tcp.port == 445 || kerberos Breaking down what we see above: TCP 3-way handshake to the endpoint looks good SMB negotiate starts off good We proceed onto Kerberos work and get a response of KRB5KDC_ERR_ETYPE_NOSUPP That seems bad but what etype isn't supported? Looking at the TGS-REQ what are we asking for? Kerberos Record Mark: 1762 bytes tgs-req pvno: 5 msg-type: krb-tgs-req (12) padata: 2 items req-body Padding: 0 kdc-options: 40810000 realm: CONTOSO.COM sname till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time nonce: 984156602 etype: 2 items ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17) enc-authorization-data [Response in: 116] We are requesting the etypes AES256-CTS-HMAC-SHA1-96 and AES128-CTS-HMAC-SHA1-96 (I will be referring to this group as AES-SHA1). If I am requesting AES-SHA1, and the KDC reports KRB5KDC_ERR_ETYPE_NOSUPP, wouldn't that mean that the KDC doesn't support AES-SHA1? If we think back to what we saw in the Kerberos Ticket Cache, we know the KDC supports AES-SHA1. PS C:\> klist Current LogonId is 0:0xe5289 Cached Tickets: (2) #0> Client: Administrator @ CONTOSO.COM Server: krbtgt/CONTOSO.COM @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 <---- Using AES-SHA1 Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize Start Time: 5/19/2025 14:11:03 (local) End Time: 5/20/2025 0:11:03 (local) Renew Time: 5/26/2025 14:11:03 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 <---- Using AES-SHA1 Cache Flags: 0x1 -> PRIMARY Kdc Called: DC1 #1> Client: Administrator @ CONTOSO.COM Server: ldap/DC1.contoso.com @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 <---- Using AES-SHA1 Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize Start Time: 5/19/2025 14:11:04 (local) End Time: 5/20/2025 0:11:03 (local) Renew Time: 5/26/2025 14:11:03 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 <---- Using AES-SHA1 Cache Flags: 0 Kdc Called: DC1.contoso.com If the KDC supports AES-SHA1 and we support AES-SHA1, then that only leaves one party, the target. This is a good theory, but I would like to back up this hypothesis with data. What does our behavior look like in the working scenario? Here is a capture from the working scenario: 2 17:14:15.089660 172.24.196.28 172.24.203.47 TCP 66 56098 → 445 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 3 17:14:15.091033 172.24.203.47 172.24.196.28 TCP 66 445 → 56098 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 4 17:14:15.091121 172.24.196.28 172.24.203.47 TCP 54 56098 → 445 [ACK] Seq=1 Ack=1 Win=65280 Len=0 5 17:14:15.091189 172.24.196.28 172.24.203.47 SMB 127 Negotiate Protocol Request 6 17:14:15.095065 172.24.203.47 172.24.196.28 SMB2 315 Negotiate Protocol Response 7 17:14:15.095117 172.24.196.28 172.24.203.47 SMB2 350 Negotiate Protocol Request 8 17:14:15.097025 172.24.203.47 172.24.196.28 SMB2 446 Negotiate Protocol Response 12 17:14:15.101146 172.24.196.28 172.24.196.47 KRB5 279 AS-REQ 13 17:14:15.103026 172.24.196.47 172.24.196.28 KRB5 332 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 18 17:14:15.105171 172.24.196.28 172.24.196.47 KRB5 359 AS-REQ 20 17:14:15.107207 172.24.196.47 172.24.196.28 KRB5 301 AS-REP 23 17:14:15.107409 172.24.196.28 172.24.203.47 TCP 54 56098 → 445 [ACK] Seq=370 Ack=654 Win=64768 Len=0 30 17:14:15.109382 172.24.196.28 172.24.196.47 KRB5 395 TGS-REQ 33 17:14:15.111274 172.24.196.47 172.24.196.28 KRB5 371 TGS-REP 36 17:14:15.111695 172.24.196.28 172.24.203.47 TCP 1514 56098 → 445 [ACK] Seq=370 Ack=654 Win=64768 Len=1460 [TCP PDU reassembled in 37] 37 17:14:15.111695 172.24.196.28 172.24.203.47 SMB2 695 Session Setup Request We successfully get a TGS-REP from the KDC. What did we ask for? Kerberos Record Mark: 1797 bytes tgs-req pvno: 5 msg-type: krb-tgs-req (12) padata: 2 items req-body Padding: 0 kdc-options: 40810000 realm: CONTOSO.COM sname till: Sep 12, 2100 22:48:05.000000000 Eastern Daylight Time nonce: 1560955428 etype: 3 items ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17) ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23) enc-authorization-data [Response in: 33] In the working scenario we are advertising the addition of ARCFOUR-HMAC-MD5 also called RC4. Looking at the Kerberos Ticket Cache for the working scenario, it confirms that we are using RC4 for the etype of the service ticket. PS C:\> klist Current LogonId is 0:0x2f0439 Cached Tickets: (2) #0> Client: Administrator @ CONTOSO.COM Server: krbtgt/CONTOSO.COM @ CONTOSO.COM KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96 Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize Start Time: 5/19/2025 13:14:15 (local) End Time: 5/19/2025 23:14:15 (local) Renew Time: 5/26/2025 13:14:15 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0x1 -> PRIMARY Kdc Called: DC1.contoso.com #1> Client: Administrator @ CONTOSO.COM Server: cifs/app.contoso.com @ CONTOSO.COM KerbTicket Encryption Type: RSADSI RC4-HMAC(NT) Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize Start Time: 5/19/2025 13:14:15 (local) End Time: 5/19/2025 23:14:15 (local) Renew Time: 5/26/2025 13:14:15 (local) Session Key Type: RSADSI RC4-HMAC(NT) Cache Flags: 0 Kdc Called: DC1.contoso.com Bingo. RC4! The question that is left is why? Well without getting too far into the weeds of Kerberos etype we know that it is a shared mask of the msds-SupportedEncryptionTypes (msds-SET) attribute values of the relevant parties. Since this is a service ticket the parties are: the client (source) the KDC the server (target) Let’s find those three values and see what they tell us. Running the following PowerShell cmdlet from the KDC: PS C:\> Get-ADComputer -Filter 'Name -eq "DC1" -or Name -eq "APP" -or Name -eq "M1"' -Properties "msds-SupportedEncryptionTypes" | % { Write-Host $("{0}`t0x{1:X2}" -f $_.Name,$_.'msds-SupportedEncryptionTypes') } DC1 0x1C M1 0x18 APP 0x04 Parsing these values using the bitmask at https://aka.ms/msds-SET, we can see that the etypes supported are: DC1 0x1C # (RC4 | AES128-SHA1-96 | AES256-SHA1-96) M1 0x18 # (AES128-SHA1-96 | AES256-SHA1-96) APP 0x04 # (RC4) (Alternatively, we can use the newly published etype calculator to see this: Kerberos EType Calculator) Got it. Our missing shared etype is RC4, but it does leave us with a pending question. Why does APP only support RC4. This is a part of a larger consideration. If it is a misconfiguration, then that is a simple remedy, however if it is a matter of APP doesn't support any newer ciphers, then the longer-term plan must be to replace APP. Microsoft strongly recommends moving away from RC4 usage as it is vulnerable to attacks like Kerberoasting. Security hardening A week after our previous incident, the same problem seems to have occurred. A ticket comes in stating that backups are failing. Jumping into our questions. Q: "What happened?" A: "The security team listened to your advice and now things are broken." Q: "What did they change?" A: "I don't know and now they are at another conference." I wish I got to go to that many conferences. Anyway, it sounds similar to the previous issue, so let’s use the same steps that we used before. Can I access the machine? PS C:\> dir \\app.contoso.com\backups dir : Cannot find path '\\app.contoso.com\backups' because it does not exist. At line:1 char:1 + dir \\app.contoso.com\backups + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (\\app.contoso.com\backups:String) [Get-ChildItem], ItemNotFoundException + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand No. Can I get a Kerberos Ticket to the service? #1> Client: Administrator @ CONTOSO.COM Server: CIFS/app.contoso.com @ CONTOSO.COM KerbTicket Encryption Type: RSADSI RC4-HMAC(NT) Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize Start Time: 5/20/2025 10:13:16 (local) End Time: 5/20/2025 20:13:16 (local) Renew Time: 5/27/2025 10:13:16 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0 Kdc Called: DC1.contoso.com Yep! Meaning that if we think about the three legs of the Kerberos work, it is likely an issue between my machine and endpoint. Time to take a packet capture. 48 14:25:50.370991 172.24.196.28 172.24.203.47 TCP 66 60549 → 445 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 49 14:25:50.371494 172.24.203.47 172.24.196.28 TCP 66 445 → 60549 [SYN, ACK, ECE] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM 50 14:25:50.371554 172.24.196.28 172.24.203.47 TCP 54 60549 → 445 [ACK] Seq=1 Ack=1 Win=65280 Len=0 51 14:25:50.371623 172.24.196.28 172.24.203.47 SMB 127 Negotiate Protocol Request 52 14:25:50.375395 172.24.203.47 172.24.196.28 SMB2 306 Negotiate Protocol Response 53 14:25:50.375458 172.24.196.28 172.24.203.47 SMB2 350 Negotiate Protocol Request 54 14:25:50.377509 172.24.203.47 172.24.196.28 SMB2 430 Negotiate Protocol Response 58 14:25:50.381655 172.24.196.28 172.24.196.47 KRB5 276 AS-REQ 59 14:25:50.383594 172.24.196.47 172.24.196.28 KRB5 326 KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED 65 14:25:50.385553 172.24.196.28 172.24.196.47 KRB5 356 AS-REQ 66 14:25:50.387413 172.24.196.28 172.24.203.47 TCP 54 60549 → 445 [ACK] Seq=370 Ack=629 Win=64768 Len=0 68 14:25:50.387509 172.24.196.47 172.24.196.28 KRB5 301 AS-REP 77 14:25:50.391491 172.24.196.28 172.24.196.47 KRB5 360 TGS-REQ 80 14:25:50.397401 172.24.196.47 172.24.196.28 KRB5 375 TGS-REP 85 14:25:50.399804 172.24.196.28 172.24.203.47 TCP 1514 60549 → 445 [ACK] Seq=370 Ack=629 Win=64768 Len=1460 [TCP PDU reassembled in 86] 86 14:25:50.399804 172.24.196.28 172.24.203.47 SMB2 545 Session Setup Request 87 14:25:50.401476 172.24.203.47 172.24.196.28 TCP 60 445 → 60549 [ACK] Seq=629 Ack=2321 Win=65280 Len=0 88 14:25:50.403453 172.24.203.47 172.24.196.28 SMB2 250 KRB Error: KRB5KRB_AP_ERR_MODIFIED 89 14:25:50.403875 172.24.196.28 172.24.203.47 TCP 54 60549 → 445 [RST, ACK] Seq=2321 Ack=825 Win=0 Len=0 This is interesting.... The TCP 3-way handshake looks good The SMB Negotiate looks good The AS and TGS flow look good But it looks like there is a problem in the session setup? Looking more closely at the SMB session setup: 86 14:25:50.399804 172.24.196.28 172.24.203.47 SMB2 545 Session Setup Request SMB2 (Server Message Block Protocol version 2) SMB2 Header Session Setup Request (0x01) [Preauth Hash: 273fca50772e510e53f6a53a3fb80cba0def1aea39292b1fbcf76cc641f3a2f4cfd768f006c274ab56ae26c2e53bc10255dc1d7d1e6c30ce95e8f665ee1ba541] StructureSize: 0x0019 Flags: 0 Security mode: 0x02, Signing required Capabilities: 0x00000001, DFS Channel: None (0x00000000) Previous Session Id: 0x0000000000000000 Blob Offset: 0x00000058 Blob Length: 1859 Security Blob […]: 6082073f06062b0601050502a08207333082072fa039303706092a864886f71201020206062b0601050205060a2b06010401823702021e060a2b06010401823702020a060a2b060104018237020228a28206f0048206ec608206e806092a864886f71201020201006e8206d730 GSS-API Generic Security Service Application Program Interface OID: 1.3.6.1.5.5.2 (SPNEGO - Simple Protected Negotiation) Simple Protected Negotiation negTokenInit mechTypes: 5 items mechToken […]: 608206e806092a864886f71201020201006e8206d7308206d3a003020105a10302010ea20703050020000000a38205716182056d30820569a003020105a10d1b0b434f4e544f534f2e434f4da2223020a003020102a11930171b04636966731b0f6170702e636f6e746f736f2e636f krb5_blob […]: 608206e806092a864886f71201020201006e8206d7308206d3a003020105a10302010ea20703050020000000a38205716182056d30820569a003020105a10d1b0b434f4e544f534f2e434f4da2223020a003020102a11930171b04636966731b0f6170702e636f6e746f736f2e636f KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_tok_id: KRB5_AP_REQ (0x0001) Kerberos ap-req pvno: 5 msg-type: krb-ap-req (14) Padding: 0 ap-options: 20000000 ticket tkt-vno: 5 realm: CONTOSO.COM sname enc-part authenticator 88 14:25:50.403453 172.24.203.47 172.24.196.28 SMB2 250 KRB Error: KRB5KRB_AP_ERR_MODIFIED SMB2 (Server Message Block Protocol version 2) SMB2 Header Session Setup Response (0x01) [Preauth Hash: 1afccd73093bb3c5bd44a55bc7bacde572955dec82ea016e3dab4ab90f8f6c638d7480cf0cd91d1d87996bca2fc5af6cbaef6fc443d35e28103ce8f8eae6b149] StructureSize: 0x0009 Session Flags: 0x0000 Blob Offset: 0x00000048 Blob Length: 120 Security Blob […]: a1763074a0030a0101a10b06092a864886f712010202a260045e605c06092a864886f71201020203007e4d304ba003020105a10302011ea411180f32303235303532303134323535305aa50502030dd833a603020129a90d1b0b434f4e544f534f2e434f4daa11300fa0030201 GSS-API Generic Security Service Application Program Interface Simple Protected Negotiation negTokenTarg negResult: accept-incomplete (1) supportedMech: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) responseToken: 605c06092a864886f71201020203007e4d304ba003020105a10302011ea411180f32303235303532303134323535305aa50502030dd833a603020129a90d1b0b434f4e544f534f2e434f4daa11300fa003020101a10830061b0461707024 krb5_blob: 605c06092a864886f71201020203007e4d304ba003020105a10302011ea411180f32303235303532303134323535305aa50502030dd833a603020129a90d1b0b434f4e544f534f2e434f4daa11300fa003020101a10830061b0461707024 KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5) krb5_tok_id: KRB5_ERROR (0x0003) Kerberos krb-error pvno: 5 msg-type: krb-error (30) stime: May 20, 2025 10:25:50.000000000 Eastern Daylight Time susec: 907315 error-code: eRR-MODIFIED (41) realm: CONTOSO.COM sname name-type: kRB5-NT-PRINCIPAL (1) sname-string: 1 item SNameString: app$ We are sending an AP-REQ to the server and it is returning KRB5KRB_AP_ERR_MODIFIED. As we talked about earlier, an AP modified error indicates that the target couldn't decrypt the Kerberos Ticket. Why is that? Chatting with our intern friend from scenario 1, they aren't doing mass resets of the machine account password. Chatting back with the ticket filer, we got a little more context around what work was done. "We upgrade to a new Linux NAS that supported AES-SHA1 so we could disable RC4". Nice! But wait, what was in our ticket cache for the service? #1> Client: Administrator @ CONTOSO.COM Server: CIFS/app.contoso.com @ CONTOSO.COM // If we don't have RC4 enabled, why are we using it? KerbTicket Encryption Type: RSADSI RC4-HMAC(NT) Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize Start Time: 5/20/2025 10:13:16 (local) End Time: 5/20/2025 20:13:16 (local) Renew Time: 5/27/2025 10:13:16 (local) Session Key Type: AES-256-CTS-HMAC-SHA1-96 Cache Flags: 0 Kdc Called: DC1.contoso.com The ticket encryption type clearly shows RC4. If the NAS has RC4 disabled, why are we using it? Let’s look at the configured msds-SET values. PS C:\> Get-ADComputer -Filter 'Name -eq "DC1" -or Name -eq "APP" -or Name -eq "M1"' -Properties "msds-SupportedEncryptionTypes" | % { Write-Host $("{0}`t0x{1:X2}" -f $_.Name,$_.'msds-SupportedEncryptionTypes') } DC1 0x1C M1 0x18 APP 0x00 What the heck is that? Why does the APP account show 0 for the msds-SET? Moreover, if the value is 0, why are we using RC4 for ticket encryption? After some research we found the following KB5021131: How to manage the Kerberos protocol changes related to CVE-2022-37966. More specifically from the KB: After installing the Windows updates that are dated on or after November 8, 2022, the following registry key is available for the Kerberos protocol: DefaultDomainSupportedEncTypes ... Windows domain controllers use this value to determine the supported encryption types on accounts in Active Directory whose msds-SupportedEncryptionType value is either empty or not set. A computer that is running a supported version of the Windows operating system automatically sets the msds-SupportedEncryptionTypes for that machines account in Active Directory. This is based on the configured value of encryption types that the Kerberos protocol is allowed to use. For more information, see Network security: Configure encryption types allowed for Kerberos. If an msds-SET is empty or is not set, then will we apply this value? With a default value of 0x27 it would be the following etypes based on MS-KILE: Supported Encryption Types Bit Flags DES-CBC-CRC, DES-CBC-MD5, RC4-HMAC, AES256-CTS-HMAC-SHA1-96 session keys. Ah... That explains it. But with that in mind what do we do about this? Do we: Update the DefaultDomainSupportedEncTypes to include AES-SHA1? Update the msds-SET for the account to reflect its true supported etypes? Well... In our case, we still have some legacy apps that cannot use AES-SHA1 so the safest option would be to set the msds-SET for the account. After updating the msds-SET to 0x18 (AES-SHA1) in AD, things worked great! Wrapping things up I know that we covered lots of content on the msds-SET but it is important to understand when working with Kerberos. In the future I'd like to break down other Kerberos protocol work like delegation and the PKINIT extension but that is going to have to be another day. With all of this in mind, I hope you have enjoyed some insight into the Kerberos protocol and its configuration. See y'all next time! BONUS! If you are curious about the changes to Kerberos etype selection on Windows 11 24H2 or Windows Server 2025, I’ve published a blog post going over the changes here: What's the deal with Kerb3961? | Microsoft Community HubIntroduction to Network Trace Analysis 3: TCP Performance
A lot of variables affect TCP performance but not all of them are easily configurable. Let’s start with our absolute constraint. The speed of light. Unfortunately, we need to abide by the pesky laws of physics. Let’s say I am communicating with a server on the exact opposite site of the earth 6,357km away.Introduction to Network Trace Analysis 5: SMB? Sounds good to me!
Let's start off with the question, what is this whole SMB thing anyway? SMB is a network file system protocol. This means that it can allow Machine A to read and write files on Machine B. This protocol serves as the backbone of much of the Enterprise Windows Ecosystem. For example, did you know that the group policy SYSVOL is an SMB share? Pretty cool right? In recent history, there have been tons of improvements to SMB.Introduction to Network Trace Analysis 4: DNS (it's always DNS)
Howdy everyone! I'm back to talk about one of my favorite causes of heartache, the domain name system (DNS). This will be our first foray into an application layer protocol. The concept of DNS is simple enough, but it can lead to some confusing situations if you don't keep its function in mind. No time to waste, let's get going!Introduction to Network Trace Analysis Part 0: Laying the Groundwork
Nearly all modern applications use the network to some degree and knowing how your specific technology interacts with the network will help you understand not only the portion that uses the network but also how your application is designed. If you can perform a basic network trace it can help direct further analysis and prevent at least a few trips down rabbit holes (I'm looking at you DNS).
