Field Extraction for Cisco Meraki

%3CLINGO-SUB%20id%3D%22lingo-sub-1707385%22%20slang%3D%22en-US%22%3EField%20Extraction%20for%20Cisco%20Meraki%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1707385%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20everyone!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20was%20having%20some%20issues%20with%20integrating%20Cisco%20Meraki%20for%20a%20customer.%20I%20did%20some%20google-fu%20and%20found%20an%20answer%20here.%20I%20now%20have%20an%20rsyslogd%20setup%20going%20to%20a%20unique%20meraki.log%20and%20that%20going%20into%20a%20custom%20table.%20We%20have%20about%203%20other%20firewall%20types%20going%20into%20the%20%22syslog%22%20table%2C%20so%20I%20figured%20this%20may%20help%20keep%20things%20organised%2C%20anyway.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EUsing%20the%20SophosXGFirewall%20function%20as%20a%20template%20I%20started%20to%20make%20a%20new%20Function%20Table%20for%20Cisco%20Meraki.%20It's%20working%20for%2099%25%20of%20the%20things%2C%20except%20for%20some%20fields%20which%20are%20encased%20in%20single-ticks%20(apostrophe%2C%20')%20--%20these%20are%20all%20for%20the%20Wireless%20Access%20Point%20logs%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-cpp%22%3E%3CCODE%3E%2F%2F%20EXAMPLE%20LOGS%0A%2F%2F%20Sep%2024%2009%3A21%3A00%20172.16.X.Y%20%201600935669.652242117%20Firewall02%20flows%20allow%20src%3D172.16.A.B%20dst%3D34.Q.R.S%20mac%3DXX%3AXX%3AXX%3AXX%3AXX%3AYY%20protocol%3Dtcp%20sport%3D58553%20dport%3D8383%0A%2F%2F%20Sep%2024%2008%3A59%3A53%20172.16.X.Y%20%201600934403.287827233%20Firewall01%20urls%20src%3D172.16.A.B%3A54234%20dst%3D34.Q.R.S%3A8383%20mac%3DXX%3AXX%3AXX%3AXX%3AXX%3AEE%20request%3A%20UNKNOWN%20https%3A%2F%2FWEBSITE-A%0A%2F%2F%20Sep%2024%2009%3A00%3A18%20172.16.Z.Y%20%201600934428.299732488%20Firewall02%20events%20type%3Dwpa_auth%20radio%3D'1'%20vap%3D'4'%20client_mac%3D'YY%3AYY%3AYY%3AYY%3AYY%3AAA'%20client_ip%3D'172.16.P.P'%20aid%3D'1016793020'%0A%2F%2F%20Sep%2024%2009%3A44%3A08%20172.16.X.Y%20%201600937057.757341105%20Firewall02%20urls%20src%3D172.16.Z.Y%3A54405%20dst%3D34.Q.R.S%3A80%20mac%3DAA%3AAA%3AAA%3AAA%3AAA%3ACC%20agent%3D'SXL%2F3.1'%20request%3A%20GET%20http%3A%2F%2FWEBSITE-B%0AMeraki_CL%0A%7C%20extend%20Meraki_IP%20%3D%20extract(%40'%5CS%2B%5Cs%2B%5Cd%2B%5Cs%2B%5CS%2B%5Cs%2B(%5CS%2B)'%2C%201%2C%20RawData)%2C%0ADevice_Name%20%3D%20extract(%40'%5CS%2B%5Cs%2B%5Cd%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B(%5CS%2B)'%2C%201%2C%20RawData)%2C%0ALog_Type%20%3D%20extract(%40'%5CS%2B%5Cs%2B%5Cd%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B(%5CS%2B)'%2C%201%2C%20RawData)%2C%0AAction%20%3D%20extract(%40'%5CS%2B%5Cs%2B%5Cd%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2B%5CS%2B%5Cs%2Bflows%5Cs%2B(%5CS%2B)'%2C%201%2C%20RawData)%2C%0ASrc_IP%20%3D%20extract(%40'src%3D%5C%22%3F(%5B%5Cw%5C.%5D%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0AClient_IP%20%3D%20extract(%40'client_ip%3D(%5B%5Cw%5C.%5D%2B)'%2C%201%2C%20RawData)%2C%0AClient_MAC%20%3D%20extract(%40'client_mac%3D(%5B%5Cw%5C%3A%5D%2B)'%2C%201%2C%20RawData)%2C%0ASrc_MAC%20%3D%20extract(%40'mac%3D%5C%22%3F(%5B%5Cw%5C%3A%5D%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0ADst_IP%20%3D%20extract(%40'dst%3D%5C%22%3F(%5B%5Cw%5C.%5D%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0AProtocol%20%3D%20extract(%40'protocol%3D%5C%22%3F(%5Cw%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0ASrc_Port%20%3D%20extract(%40'sport%3D%5C%22%3F(%5Cd%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0ADst_Port%20%3D%20extract(%40'dport%3D%5C%22%3F(%5Cd%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0AEvent_Type%20%3D%20extract(%40'type%3D%5C%22%3F(%5CS%2B)%5C%22%3F'%2C%201%2C%20RawData)%2C%0AUser_Agent%20%3D%20extract(%40'agent%3D(.*%5B%5Erequest%5D)request'%2C%201%2C%20RawData)%2C%0AURL%20%3D%20extract(%40'request%3A%20%5CS%2B%5Cs%2B(.*)'%2C1%2C%20RawData)%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOK%2C%20simple%20enough.%20The%20problem%20is%20the%20regex%20and%20something%20with%20the%20extract()%20function.%20Some%20of%20the%20items%20are%20encased%20in%20%22%5C'%22%20and%20not%20%22%5C%22%22%26nbsp%3B%20--%20when%20I%20change%20the%20regex%20for%20Client_IP%20and%20Client_MAC%20to%20extract(%40'client_mac%3D%5C'(%5Bw%5C%3A%5D)%5C''%2C%201%2C%20RawData)%20it%20breaks%20completely.%20Is%20this%20a%20bug%20of%20some%20sort%20or%20is%20there%20some%20syntactical%20wizardry%20I'm%20missing%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1707385%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3Ecisco%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EExtract%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3Emeraki%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESentinel%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Highlighted
Occasional Contributor

Hi everyone!

 

I was having some issues with integrating Cisco Meraki for a customer. I did some google-fu and found an answer here. I now have an rsyslogd setup going to a unique meraki.log and that going into a custom table. We have about 3 other firewall types going into the "syslog" table, so I figured this may help keep things organised, anyway.

 

Using the SophosXGFirewall function as a template I started to make a new Function Table for Cisco Meraki. It's working for 99% of the things, except for some fields which are encased in single-ticks (apostrophe, ') -- these are all for the Wireless Access Point logs

 

 

// EXAMPLE LOGS
// Sep 24 09:21:00 172.16.X.Y  1600935669.652242117 Firewall02 flows allow src=172.16.A.B dst=34.Q.R.S mac=XX:XX:XX:XX:XX:YY protocol=tcp sport=58553 dport=8383
// Sep 24 08:59:53 172.16.X.Y  1600934403.287827233 Firewall01 urls src=172.16.A.B:54234 dst=34.Q.R.S:8383 mac=XX:XX:XX:XX:XX:EE request: UNKNOWN https://WEBSITE-A
// Sep 24 09:00:18 172.16.Z.Y  1600934428.299732488 Firewall02 events type=wpa_auth radio='1' vap='4' client_mac='YY:YY:YY:YY:YY:AA' client_ip='172.16.P.P' aid='1016793020'
// Sep 24 09:44:08 172.16.X.Y  1600937057.757341105 Firewall02 urls src=172.16.Z.Y:54405 dst=34.Q.R.S:80 mac=AA:AA:AA:AA:AA:CC agent='SXL/3.1' request: GET http://WEBSITE-B
Meraki_CL
| extend Meraki_IP = extract(@'\S+\s+\d+\s+\S+\s+(\S+)', 1, RawData),
Device_Name = extract(@'\S+\s+\d+\s+\S+\s+\S+\s+\S+\s+(\S+)', 1, RawData),
Log_Type = extract(@'\S+\s+\d+\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)', 1, RawData),
Action = extract(@'\S+\s+\d+\s+\S+\s+\S+\s+\S+\s+\S+\s+flows\s+(\S+)', 1, RawData),
Src_IP = extract(@'src=\"?([\w\.]+)\"?', 1, RawData),
Client_IP = extract(@'client_ip=([\w\.]+)', 1, RawData),
Client_MAC = extract(@'client_mac=([\w\:]+)', 1, RawData),
Src_MAC = extract(@'mac=\"?([\w\:]+)\"?', 1, RawData),
Dst_IP = extract(@'dst=\"?([\w\.]+)\"?', 1, RawData),
Protocol = extract(@'protocol=\"?(\w+)\"?', 1, RawData),
Src_Port = extract(@'sport=\"?(\d+)\"?', 1, RawData),
Dst_Port = extract(@'dport=\"?(\d+)\"?', 1, RawData),
Event_Type = extract(@'type=\"?(\S+)\"?', 1, RawData),
User_Agent = extract(@'agent=(.*[^request])request', 1, RawData),
URL = extract(@'request: \S+\s+(.*)',1, RawData)

 

 

OK, simple enough. The problem is the regex and something with the extract() function. Some of the items are encased in "\'" and not "\""  -- when I change the regex for Client_IP and Client_MAC to extract(@'client_mac=\'([w\:])\'', 1, RawData) it breaks completely. Is this a bug of some sort or is there some syntactical wizardry I'm missing?

 

Thanks!

 

1 Reply
Highlighted

So I did a quick fix to the regex to work around this, but I'm still worried that in the future I may need to anchor something off a single apostrophe, and I won't be able to:

 

Client_IP = extract(@'client_ip=\S([\w\.]+)\S', 1, RawData),
Client_MAC = extract(@'client_mac=\S([\w\:]+)\S', 1, RawData),