Forum Discussion

Kbprogrammer's avatar
Kbprogrammer
Copper Contributor
Jun 29, 2023

Linux Arc Azure Machine Agent (AMA) agent failing to send heartbeats and logs to analytics workspace

I'm attempting to deploy the Linux Azure Arc agent on an Ubuntu (22.04) host to collect log data for ingestion into a Sentinel Log Analytics workspace. The VM is hosted outside of Azure in AWS and firewall is open to send outbound traffic to any port. I ran the first step of adding the Arc agent via: "Azure Arc" -> "Servers" ->"+Add" and downloaded and ran the bash installation script and can confirm it connects.

Then I've added the AMA agent via the "extensions" page.

Next, I've created a DCR rule, associating my arc AMA agent with Syslog source data:

And set the destination to two of my Log analytics workspaces. 

 

However; whenever I go to that log analytics workspace I never see that any AMA agent is registered with my workspace. No logs come through either.

I've checked that I have rsyslog listening on port 514 and have confirmed that I see the azuremonitor.conf configs inside of my /etc/rsyslog.d directory.  

 

Also, can confirm that rsyslog and amacoreagent are both listening on the correct ports.

To humor myself, I've also tried sending just a simple syslog messages as well as more complex messages via netcat and outside of the host with a few other syslog tools and I never get a Syslog message..

 

echo "<34>1 2023-07-01T08:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8" | nc -q0 127.0.0.1 514
 

Any ideas? 

 

2 Replies

    • Kbprogrammer's avatar
      Kbprogrammer
      Copper Contributor
      Jul 02, 2023

      Kidd_Ip Agreed. I took a look at: /var/lib/GuestConfig/arc_policy_logs/gc_agent.log and I see this in the logs.

       

      [2023-07-02 15:07:55.409] [PID 433] [TID 672] [Pull Client] [ERROR] [d4817f84-7e6c-42b4-8b3f-cec925205ae4] [/__w/1/s/src/dsc/gc_pullclient/dsc_pull_client.cpp:362] Failed to update assignments Error : Failed to get regional end_point info. Status Code '404'. Error Message 'Agent service endpoint returned is null or empty'.

       

      Also see:


      [2023-07-01 14:10:26.585] [PID 2684] [TID 2758] [Pull Client] [ERROR] [d0e9fdb1-b90d-4547-8846-0d47ff0603d2] [/__w/1/s/src/dsc/gc_pullclient/dsc_pull_client.cpp:362] Failed to update assignments Error : Failed to get the compute meta_data from the url : http://localhost:40342/metadata/instance?api-version=2019-03-11. Status Code '503'. Error Message '{"error":"service_unavailable","error_description":"Service not available. Check Agent log for details.","error_codes":[503],"timestamp":"2023-07-01 14:10:26.541137096 +0000 UTC m=+6.583236427","trace_id":"","correlation_id":"14a6f9a5-a63b-41c3-94a4-38bf91f87f53"}

      Looks like an issue trying to pull connection details so I curled the URL in question:

      curl http://localhost:40342/metadata/instance?api-version=2019-03-11

       

      It fails with: 

       

      {"error":"bad_request_102","error_description":"Required metadata header not specified","error_codes":[400],"timestamp":"2023-07-02 15:18:59.596985834 +0000 UTC m=+87255.848410309","trace_id":"","correlation_id":"0f54476e-39c7-44e7-9694-5d0abeffffcc"}

       

      However; if you add the metadata header in the request it will return the correct response..

       

      curl -H Metadata:true http://localhost:40342/metadata/instance?api-version=2019-03-11

       

      Going to keep digging but I'm unsure if there's a setting in the config to enable setting that header?

       

       

Resources