Using cURL for isolation of Web related issues
Why Use cURL? Here are some scenarios where cURL can be particularly helpful: Lack of a Latest Browser for Local Testing: When customers do not have access to the latest browser versions, cURL can act as a lightweight alternative. No GUI Environments: Ideal for environments like Azure App Services (PAAS Servers) or Linux servers where GUI-based testing isn’t possible. Client Isolation: Browsers often behave differently due to enforced policies or extensions, making it harder to isolate issues. cURL provides a clean, policy-free testing environment. Detailed Diagnostics: It offers a detailed view of request-response cycles, including headers, status codes, and payloads. Customized Requests: Supports various HTTP verbs, custom headers, and even protocols beyond HTTP, allowing more flexible testing. The syntax of cURL commands and options may change in the future. Always refer to the cURL documentation for the most up-to-date information. curl --help Everything curl - everything curl curl - How To Use Curl is Available by Default Most modern operating systems include cURL by default. On Windows, use 'cmd' since in powershell curl is added as alias for Invoke-WebRequest (Microsoft.PowerShell.Utility) - PowerShell | Microsoft Learn curl --version curl 8.10.1 (Windows) libcurl/8.10.1 Schannel zlib/1.3 WinIDN Release-Date: 2024-09-18 Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s smb smbs smtp smtps telnet tftp Features: alt-svc AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets A work-around is to invoke curl as 'curl.exe' to prevent PowerShell from treating it as an alias. Verbose Mode (-v) Verbose mode provides a detailed breakdown of the request-response cycle. This can be invaluable for identifying where a request fails. Some details include: DNS resolution TCP and TLS details HTTP Request/Response Headers and Body Protocol specific detailed messages curl https://google.com -v * Host google.com:443 was resolved. * IPv6: (none) * IPv4: X.X.X.X * Trying X.X.X.X:443... * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server accepted http/1.1 * Connected to google.com (X.X.X.X) port 443 * using HTTP/1.x > GET / HTTP/1.1 > Host: google.com > User-Agent: curl/8.10.1 > Accept: */* > * Request completely sent off * schannel: remote party requests renegotiation * schannel: renegotiating SSL/TLS connection * schannel: SSL/TLS connection renegotiated < HTTP/1.1 301 Moved Permanently < Location: https://www.google.com/ < Content-Type: text/html; charset=UTF-8 <trucated> < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="https://www.google.com/">here</A>. </BODY></HTML> * Connection #0 to host google.com left intact Sometimes the request doesn't even make it to the intended server and the output of curl's verbose mode clearly indicate the step at which failed (like DNS, TCP or TLS). Below is an example of DNS failure: curl http://vm9 -v * Could not resolve host: vm9 * shutting down connection #0 curl: (6) Could not resolve host: vm9 Below is an example of TCP connection failure (Blocked by Windows Firewall): curl http://vm2:8088 -v * Host vm2:8088 was resolved. * IPv6: (none) * IPv4: X.X.X.X * Trying X.X.X.X:8088... * connect to X.X.X.X port 8088 from 0.0.0.0 port 50020 failed: Timed out * Failed to connect to vm2 port 8088 after 21052 ms: Could not connect to server * closing connection #0 curl: (28) Failed to connect to vm2 port 8088 after 21052 ms: Could not connect to server Below is an example of HTTP Request Parsing failure. The server is supposed to add CRLF between HTTP response headers and body - RFC 9112: HTTP/1.1. Since it is not present, the clients assume that the first line of body is another header which violate the syntax of key:value (header without colon) and close the connection. The browsers do not really show detailed information without browser traces. curl http://localhost:8548/ -v * Host localhost:8548 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:8548... * Trying 127.0.0.1:8548... * Connected to localhost (127.0.0.1) port 8548 * using HTTP/1.x > GET / HTTP/1.1 > Host: localhost:8548 > User-Agent: curl/8.10.1 > Accept: */* > < HTTP/1.1 200 OK < Server: Microsoft-IIS/10.0 < Date: Wed, 31 Jul 2024 17:44:50 GMT < Connection: close < Pragma: cache < Content-type: text/html < <HTML xmlns:v="urn:schemas-microsoft-com:vml"> * Header without colon * closing connection #0 curl: (8) Header without colon SSL/TLS Troubleshooting - Part 1 The verbose output includes SSL handshake details, such as server certificate verification, trust chain, SAN, and revocation status. curl https://signalr.alterego.com -v * Host signalr.alterego.com:443 was resolved. * IPv6: (none) * IPv4: 127.0.0.1 * Trying 127.0.0.1:443... * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * schannel: SNI or certificate check failed: SEC_E_WRONG_PRINCIPAL (0x80090322) - The target principal name is incorrect. * closing connection #0 curl: (60) schannel: SNI or certificate check failed: SEC_E_WRONG_PRINCIPAL (0x80090322) - The target principal name is incorrect. More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the webpage mentioned above. In this example the Server Certificate SAN is not matching the hostname. SSL/TLS Troubleshooting - Part 2 Curl also allows to print variables from SSL Handshake like the certificate payload. On Windows, the certificate blob can be saved into a .cer file to inspect the attributes of certificate (like Subject Alternative Name Extension). You can other options to review the certificate blob like an online tool. -k will disable certificate validation checks. -w will write out variables - [available variables] curl https://signalr.alterego.com -k -v -w "%{certs}" * Host signalr.alterego.com:443 was resolved. * IPv6: (none) * IPv4: 127.0.0.1 * Trying 127.0.0.1:443... <trucated> * Subject: CN=localhost * Issuer: CN=localhost * Version: 2 * Serial Number: 3c:92:85:35:af:eb:c4:84:47:77:a9:6e:83:88:f7:14: * Signature Algorithm: sha256WithRSAEncryption * Start Date: 2022-08-05 15:25:44 GMT * Expire Date: 2027-08-05 00:00:00 GMT * Public Key Algorithm: rsaEncryption * RSA Public Key (2048 bits) * rsa(n): cb:aa:6d:03:4e:07:b6:b8:cf:f7:e<trucated> * rsa(e): 0x10001 * Signature: 7a:a6:c6:17:5f:93:34:bd:26<trucated> * Cert: -----BEGIN CERTIFICATE----- MIIC7DCCAdSgAwIBAgIQPJKFNa/rxIRHd6lug4j3FDANBgkqhkiG9w0BAQsFADAU MRIwEAYDVQQDEwlsb/<truncated> -----END CERTIFICATE----- SSL/TLS Troubleshooting - Part 3 We can also set with explicit TLS versions while making the request. This can help to understand the behavior of sever against different TLS versions. For example below, the server has TLS 1.1 disabled. curl https://vm2.alterego.cc --tls-max 1.1 -v * Host vm2.alterego.cc:443 was resolved. * IPv6: (none) <truncated> * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * Recv failure: Connection was reset * schannel: failed to receive handshake, SSL/TLS connection failed * closing connection #0 curl: (35) Recv failure: Connection was reset More TLS related options, you can use "curl --help tls". -1, --tlsv1 TLSv1.0 or greater --tlsv1.0 TLSv1.0 or greater --tlsv1.1 TLSv1.1 or greater --tlsv1.2 TLSv1.2 or greater --tlsv1.3 TLSv1.3 or greater --tls-max <VERSION> Maximum allowed TLS version Proxies Proxies/Intermediate devices can introduce unexpected behavior in responses. cURL helps identify and isolate proxy-related issues. -w "%{certs}" - will can show certificates and we can see if it is from the actual server/proxy --noproxy '*' - will disable proxy With Proxy Curl provides great indication if the proxies are involved (CONNECT Request details) and using the certificate dump we can also identify if the server certificate was provided by actual destination vs a TLS terminating device like proxy/WAF (notice the certificate subject which clearly indicates the certificate was provided by Proxy device). curl -v -k https://google.com -w "%{certs}" * Trying 127.0.0.1:8888... * CONNECT tunnel: HTTP/1.1 negotiated * allocate connect buffer * Establish HTTP proxy tunnel to google.com:443 > CONNECT google.com:443 HTTP/1.1 > Host: google.com:443 > User-Agent: curl/8.10.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection Established < FiddlerGateway: Direct < StartTime: 02:12:03.172 < Connection: close < * CONNECT phase completed * CONNECT tunnel established, response 200 * schannel: disabled automatic use of client certificate * ALPN: curl offers http/1.1 * ALPN: server did not agree on a protocol. Uses default. * Subject: OU=Created by http://www.fiddler2.com, O=DO_NOT_TRUST, CN=google.com * Issuer: OU=Created by http://www.fiddler2.com, O=DO_NOT_TRUST, CN=DO_NOT_TRUST_FiddlerRoot * Version: 2 * Serial Number: 6c:18:7b:e5:fa:d0:36:89:4a:3f:50:a1:bd:8e:80:6e: * Signature Algorithm: sha256WithRSAEncryption Without Proxy Even though a proxy might be enforced via OS settings or variables like "http/s_proxy", the using --noproxy '*' bypasses the proxy connection. curl --noproxy '*' -v -k https://google.com -w "%{certs}" * Host google.com:443 was resolved. <trucated> * schannel: renegotiating SSL/TLS connection * Subject: CN=*.google.com * Issuer: C=US, O=Google Trust Services, CN=WR2 * Version: 2 * Serial Number: 00:fc:12:5b:bd:5b:36:ea:6b:12:3b:0a:55:49:88:4c:9f: * Signature Algorithm: sha256WithRSAEncryption * Start Date: 2024-12-02 08:35:57 GMT * Expire Date: 2025-02-24 08:35:56 GMT Use different HTTP Verbs, Headers and Send Body Customize HTTP Sometimes you might want to use other HTTP verbs or customize headers. For example, testing how the server behaves for a CORS preflight request In below examples. the "google.com" origin is not allowed whereas bing.com is allowed. -X specifies HTTP verb -H allows to specify different headers -d allows to specify HTTP Body curl -X OPTIONS http://localhost:8334 -H "Origin: https://bing.com" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: Content-Type" -v * Host localhost:8334 was resolved. <truncated> > Host: localhost:8334 > User-Agent: curl/8.10.1 > Accept: */* > Origin: https://bing.com > Access-Control-Request-Method: GET > Access-Control-Request-Headers: Content-Type > < HTTP/1.1 204 No Content < Vary: Origin < Server: Microsoft-IIS/10.0 < Access-Control-Allow-Origin: https://bing.com < X-Powered-By: ASP.NET < Date: Wed, 22 Jan 2025 07:55:11 GMT curl -X OPTIONS http://localhost:8334 -H "Origin: https://google.com" -H "Access-Control-Request-Method: GET" -H "Access-Control-Request-Headers: Content-Type" -v * Host localhost:8334 was resolved. <truncated> > Accept: */* > Origin: https://google.com > Access-Control-Request-Method: GET > Access-Control-Request-Headers: Content-Type > < HTTP/1.1 204 No Content < Vary: Origin < Server: Microsoft-IIS/10.0 < X-Powered-By: ASP.NET < Date: Wed, 22 Jan 2025 07:55:30 GMT < * Connection #0 to host localhost left intact Below is an example to get a token using OAuth 2.0 client credentials flow on the Microsoft identity platform - Microsoft identity platform | Microsoft Learn Note - Windows cmd uses '^' for continuation of commands and other shells might have different continuation symbols. curl -X POST https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/token ^ -H "Host: login.microsoftonline.com" ^ -H "Content-Type: application/x-www-form-urlencoded" ^ -d "grant_type=client_credentials&client_id=MICROSOFT-APP-ID&client_secret=MICROSOFT-APP-PASSWORD&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default" Authentication Curl also supports Authentication Mechanisms like NTLM, Kerberos, Basic etc. Authentication in Curl curl http://vm2:8133 -v --negotiate * Host vm2:8133 was resolved. * IPv6: (none) * IPv4: 10.160.56.6 * Trying 10.160.56.6:8133... * Connected to vm2 (10.160.56.6) port 8133 * Server auth using Negotiate with user '' > GET / HTTP/1.1 > Host: vm2:8133 > Authorization: Negotiate YIIH <truncated> < HTTP/1.1 200 OK < Content-Type: text/html < Last-Modified: Mon, 28 Mar 2022 15:53:03 GMT < Accept-Ranges: bytes < ETag: "4045c2e8bb42d81:0" < Server: Microsoft-IIS/10.0 < WWW-Authenticate: Negotiate oYG1 <truncated> Load Test You might a find a need to make multiple requests reproduce some problem. With the use of curl's range convention [start-end] together with writeout it can print detailed results of the test. curl -s -w "\nRequest: %{url_effective} Total Time: %{time_total}s HTTP Code: %{http_code}\n" "http://google.com/?test=[1-100]" -o nul Request: http://google.com/?test=1 Total Time: 0.069207s HTTP Code: 301 Request: http://google.com/?test=2 Total Time: 0.748296s HTTP Code: 301 Request: http://google.com/?test=3 Total Time: 0.047916s HTTP Code: 301 Request: http://google.com/?test=4 Total Time: 0.153930s HTTP Code: 301 Request: http://google.com/?test=5 Total Time: 0.053448s HTTP Code: 301 Note that the calls are sequential, if you wish to run concurrent requests, open curl multiple sessions. FTP Curl has great support for FTP including below: Active and Passive Mode Implicit and Explicit FTPs File download and directory listing. Command line FTP - everything curl '-v' shows detailed command exchange and diagnosis. Use -u for specifying users - Command line FTP - everything curl. By default, uses an anonymous user. The direction of brackets ('>' and '<' indicate whether the command/response is sent by client of server respectively). Most connectivity issues in FTP are usually related to data connection (Two connections - everything curl). The curl commands shown below will trigger a need of Data connection to show directory listing. Active Mode: [--ftp-port] - Enables ACTIVE Mode. Provides options to specify the address (- (hyphen) same as control channel) [--disable-eprt] - Disables IpV6 curl ftp://<servername>:<port> -v -p - curl ftp://localhost:21 -v -P - * Host localhost:21 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:21... * Connected to localhost (::1) port 21 < 220 Microsoft FTP Service > USER anonymous < 331 Anonymous access allowed, send identity (e-mail name) as password. > PASS ftp@example.com < 230 User logged in. > PWD < 257 "/" is current directory. * Entry path is '/' * Request has same path as previous transfer > EPRT |2|::1|54989| * [FTP] [PORT] perform, DATA connection established < 200 EPRT command successful. * Connect data stream actively > TYPE A < 200 Type set to A. > LIST < 125 Data connection already open; Transfer starting. * Maxdownload = -1 * Preparing for accepting server on data port * Checking for server connect * Ready to accept data connection from server * Connection accepted from server 10-26-24 12:32PM 696 iisstart.htm * abort upload * Remembering we are in dir "" < 226 Transfer complete. * Connection #0 to host localhost left intact Passive Mode * [--disable-epsv]- Disables Ipv6 * Passive mode is enabled by default curl ftp://<servername>:<port> -v curl ftp://localhost:21 -v * Host localhost:21 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:21... * Connected to localhost (::1) port 21 < 220 Microsoft FTP Service > USER anonymous < 331 Anonymous access allowed, send identity (e-mail name) as password. > PASS ftp@example.com < 230 User logged in. > PWD < 257 "/" is current directory. * Entry path is '/' * Request has same path as previous transfer > EPSV * Connect data stream passively < 229 Entering Extended Passive Mode (|||59401|) * Connecting to ::1 (::1) port 59401 * Trying [::1]:59401... * Connected 2nd connection to ::1 port 59401 > TYPE A < 200 Type set to A. > LIST < 125 Data connection already open; Transfer starting. * Maxdownload = -1 10-26-24 12:32PM 696 iisstart.htm FTPs FTPs or FTP over TLS is a secure FTP protocol which is available in IIS. This is not be confused with SFTP which uses SSH for FTP. FTP Clients - Part 2: Explicit FTPS versus Implicit FTPS | Microsoft Learn Implicit FTPs (Control port 990): curl --ssl-reqd ftps://<server name>/ -u anonymous:anonymous MS-FTPS: Control Connection Negotiation with Implicit FTPS | Microsoft Learn Explicit FTPs (any other control port than 990 with SSL configured) FTPS - Everything curl curl --ssl-reqd ftp://<servername>:port -u anonymous:anonymous Final Thoughts cURL is an incredibly versatile tool that can simplify and speed up the isolation of web-related issues. Whether you are debugging SSL/TLS connections, testing custom HTTP requests, or bypassing proxies, FTP tests, cURL provides the flexibility and precision needed for effective troubleshooting. Next time you’re faced with a challenging web application issue, consider reaching for cURL to peel back the layers and uncover the root cause.A fatal error occurred when attempting to access the SSL server credential private key: 0x8009030d
First published on MSDN on Apr 28, 2017 Recently, I have assisted a Premier customer who installed a new certificate on Windows Server 2008 R2 but was unable to bind the certificate to the Website hosted on IIS.Should I worry about Timer_ConnectionIdle entries in HTTPERR Log?
First published on MSDN on Apr 28, 2017 While helping customers troubleshooting HTTP-related error, a common question that we face: What is Timer_ConnectionIdle (as we see a lot of them in the HTTPERR log)?, and is this something to worry about?Here is an excerpts from the HTTPERR log:2016-07-05 18:45:09 172.
