tds
3 TopicsLessons Learned #554: I Have Used BCP for Years — But How Does It Actually Work?
BCP is one of those SQL Server tools that many of us have been using for years. It is simple, reliable, and extremely fast when we need to move large amounts of data. We execute it, and millions of rows can be loaded into SQL Server or Azure SQL Database. But after using BCP many times, I had a question that stayed in my mind: Why is BCP so fast? And that question led me to another one: What does BCP actually send to SQL Server? Is it simply executing thousands or millions of INSERT statements behind the scenes? And finally: Could I create my own BCP-like application in C++, .NET, Java, or another programming language and adapt it to my own requirements? That curiosity is what motivated this article. The answer is quite interesting. BCP is not simply a faster way to execute INSERT statements. It uses a different model for moving data. A Very Simple BCP Example A typical import might look like this: bcp MyDatabase.dbo.Customers in customers.csv -S myserver.database.windows.net -d MyDatabase -c -t"," -T The bcp utility bulk copies data between SQL Server and a data file in a user-specified format. Microsoft also exposes bulk-copy functionality programmatically through SQL Server drivers. At a very high level, we normally think of the process as: File -> BCP -> SQL Server. But there is much more happening in between. My First Question: Is BCP Sending INSERT Statements? Imagine that we need to load one million rows. A traditional application could repeatedly execute something like: INSERT INTO dbo.Customers ( CustomerId, CustomerName, Amount ) VALUES ( @CustomerId, @CustomerName, @Amount ); Even when the connection is reused and the statement is parameterized, we are still performing a very large number of individual database operations. BCP works differently. And this is probably the most important concept in this article: BCP is not a faster way to send millions of INSERT statements. It avoids sending those INSERT statements in the first place. Instead, BCP establishes a bulk data stream with SQL Server. Going One Level Lower: TDS SQL Server clients communicate with the database engine using TDS — Tabular Data Stream. TDS carries SQL requests, result sets, metadata, errors, authentication information, bulk data, and other messages between the client and SQL Server. For bulk loading, TDS defines a dedicated bulk-load stream. At a simplified level, it looks like this: Conceptually, SQL Server first receives metadata describing the incoming row shape, followed by the row stream and a completion token. This is a completely different model from repeatedly submitting: INSERT INSERT INSERT INSERT INSERT SQL Server is told what the incoming data looks like, and then rows are streamed continuously through the connection. There Is an INSERT BULK Operation This was one of the details I found most interesting. When we execute bcp.exe, we do not manually write an INSERT BULK statement. However, the bulk-load protocol requires the client to identify the destination and incoming structure before sending the actual row stream. A simplified sequence looks like this: The important point is that the rows themselves are not represented as complete SQL statements. They are part of a structured bulk stream. What Is COLMETADATA? Suppose our destination table is: CREATE TABLE dbo.Customers ( CustomerId int, CustomerName varchar(100), Amount decimal(10,2) ); Before SQL Server can interpret the incoming row data, it needs information about the shape and types of the columns. Conceptually, the metadata describes something similar to: Column 1 INT Column 2 VARCHAR Length = 100 Column 3 DECIMAL Precision = 10 Scale = 2 Then the row stream follows: ROW 1001 Juan 42.50 ROW 1002 Pedro 18.75 ROW 1003 Jose 91.00 The real TDS representation is binary and structured, not human-readable like this, but the mental model is useful:BCP sends structured row data through the bulk protocol instead of generating SQL text for every record. From a CSV File to SQL Server Imagine the source file contains: 1001,Juan,42.50 1002,Pedro,18.75 1003,Jose,91.00 and we execute: bcp MyDatabase.dbo.Customers in customers.csv -S myserver -c -t"," -T Because we are using character mode, the source contains character representations. BCP reads the file, recognizes the fields according to the specified format, and feeds the values into the bulk-copy path. For example, the character value "1001" ultimately has to become an INT, and "42.50" has to become a DECIMAL(10,2). So BCP is fast, but it is not magic. Data still has to be interpreted and converted into the destination SQL Server data types. What About Memory? This was another thing I wanted to understand. Suppose I need to import a 100-GB file. Does BCP need 100 GB of memory? No. A better mental model is a streaming pipeline. The complete file does not need to reside in memory at the same time. Buffers can be filled, transmitted, reused, filled again, and transmitted again. The amount of data being transferred is not the same as the amount of memory required to transfer it. A Row Is Not Necessarily a Network Packet Sending one row through the BCP API does not necessarily mean that one network packet is immediately sent for that row. The bulk-copy implementation can accumulate rows while filling network packets. Conceptually: Row 1 ----\ Row 2 -----\ Row 3 ------> Network packet ---> SQL Server Row 4 -----/ Row 5 ----/ This is much more efficient than treating every row as an independent network operation. The command-line utility also exposes a packet-size option through -a packet_size. Larger is not automatically better; the optimal value depends on the environment, row size, driver, network, and workload. Then I Discovered the BCP API This was probably my favorite part of the investigation. BCP is not only a command-line utility. Microsoft exposes bulk-copy functionality programmatically through the ODBC bulk-copy API. Among the available functions are: bcp_init() bcp_bind() bcp_sendrow() bcp_batch() bcp_done() That means we can create our own bulk loader instead of launching bcp.exe. Understanding bcp_bind() Imagine that our application contains: int CustomerId; char CustomerName[100]; With the BCP API, those variables can be associated with destination columns. The application can then populate the same variables repeatedly and call bcp_sendrow() to feed another row into the bulk stream. CustomerId = 1; strcpy(CustomerName, "Customer A"); bcp_sendrow(hdbc); CustomerId = 2; strcpy(CustomerName, "Customer B"); bcp_sendrow(hdbc); The application is not building and submitting a new INSERT INTO... statement each time. It keeps populating values and feeding rows into the existing bulk-copy pipeline. The Source Does Not Even Need to Be a File Once we understand the API, another interesting possibility appears: the source can be anything our application knows how to read. For example, the source could be a REST API, a message stream, another database, generated data, or an in-memory structure. bcp.exe is therefore one implementation of the bulk-copy concept, not the only way to use it. Can I Create My Own BCP in Another Programming Language? Yes. The exact interface depends on the language and driver. C / C++ The ODBC BCP API gives low-level access using functions such as: bcp_init bcp_bind bcp_sendrow bcp_batch bcp_done .NET For .NET applications, Microsoft provides SqlBulkCopy : using var connection = new SqlConnection(connectionString); await connection.OpenAsync(); using var bulkCopy = new SqlBulkCopy(connection); bulkCopy.DestinationTableName = "dbo.Customers"; bulkCopy.BatchSize = 10000; await bulkCopy.WriteToServerAsync(reader); Java The Microsoft JDBC Driver provides SQLServerBulkCopy : SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(connection); bulkCopy.setDestinationTableName("dbo.Customers"); bulkCopy.writeToServer(resultSet); The important concept is that the same bulk-copy model can be consumed directly by applications. Batches: Another Piece of the Puzzle There is another important concept: transaction batching. Using the BCP API, an application can send multiple rows and then call: bcp_batch(); The command-line utility exposes the same general idea through -b batch_size. Batching can affect transaction duration, rollback scope, transaction-log behavior, error recovery, and throughput. Batch size is not the same thing as memory buffer size. If we specify -b 100000, we are primarily defining a transactional boundary. We are not saying that BCP must hold exactly 100,000 rows in RAM. What Happens When the Stream Arrives at SQL Server? Successfully delivering rows to SQL Server is only part of the operation. On the server side, a simplified path looks like this: TDS bulk stream-> Decode metadata and rows -> Bulk-load processing -> Storage Engine -> Data pages -> Transaction log SQL Server still needs to store those rows. Depending on the destination, that can involve data-type processing, page allocation, transaction logging, index maintenance, constraint validation, identity handling, triggers, locking, and dirty data pages. So Why Is BCP So Fast? After looking at the mechanism, I do not think there is one single magic optimization. BCP is fast because several efficiencies work together. Traditional row-by-row processing BCP / Bulk Copy Many individual SQL operations Continuous bulk stream Repeated execution overhead Bulk-oriented processing Potentially many client/server interactions Persistent streaming pipeline SQL statement representation Structured row representation Small network operations possible Rows can be packed efficiently into packets Frequent transaction boundaries possible Controlled batching Generic DML execution pattern Bulk-copy-oriented path For a few rows, the difference might not matter much. Multiply those savings by millions of rows, and the result becomes significant. References Microsoft Learn — bcp utility Microsoft Learn — Import and export bulk data using bcp Microsoft Open Specifications — TDS Bulk Load BCP Microsoft Learn — bcp_bind Microsoft Learn — bcp_sendrow Microsoft Learn — SqlBulkCopy Microsoft Learn — SQLServerBulkCopy530Views0likes0CommentsTLS 1.3 in SQL Server 2025: What It Really Changes for Security
Why TLS 1.3 matters TLS (Transport Layer Security) is the protocol that encrypts traffic between clients and servers. For many years, most SQL Server environments have relied on TLS 1.2, which dates back to 2008. TLS 1.3, finalized in 2018, is a significant evolution. It brings three main benefits: Better performance With fewer roundtrips and faster negotiation, secure connections are established much more quickly in practice Stronger security Old and weak algorithms are removed (RC4, SHA1, MD5, DES, 3DES…) Perfect Forward Secrecy (PFS) is mandatory. Even if a private key is compromised later, past sessions remain protected Simpler cipher suites: less configuration complexity, fewer chances to misconfigure Easier compliance Recommended by NIST Strongly encouraged by PCIDSS 4.0 Fewer exceptions to justify during audits What SQL Server 2025 adds SQL Server 2022 introduced TLS 1.3 through the new TDS 8.0 (Tabular Data Stream) protocol. And SQL Server 2025 makes it broadly usable across the SQL Server ecosystem, this is not limited to client connections only. Components that can use TLS 1.3 Client connections (ODBC, OLE DB, .NET) SQL Server Agent bcp utility sqlcmd utility Always On Availability Groups Always On Failover Cluster Instances Log Shipping Transactional replication Merge replication Peer-to-peer replication Snapshot replication Linked Servers PolyBase SQL Writer Service Prerequisites Valid certificates on all endpoints Selfsigned certificates are acceptable for testing In production, use certificates from a trusted CA or an internal PKI Updated drivers ODBC Driver 18+ for SQL Server OLE DB Driver 19+ for SQL Server Microsoft.Data.SqlClient 5.0+ Older drivers will either fall back to older TLS versions or fail to connect SQL Server configuration You can enforce encryption on all connections (Force Encryption) You can configure TDS 8.0 as the minimum protocol version (Force Strict Encryption) How Copilot can help I'm setting up log shipping between two SQL Server 2025 instances with TLS 1.3 enabled. What certificate requirements should I verify ? The MSSQL extension with GitHub Copilot can assist you when planning and rolling out TLS 1.3. Auditing the current state Analyze my current SQL Server encryption configuration. What TLS version are my connections using ? Are there any connections using deprecated protocols ? Copilot can generate and explain the queries that inspect the relevant DMVs and interpret the results. Generating configuration scripts Generate a script to configure SQL Server 2025 to require TLS 1.3 for all incoming connections, including the certificate configuration steps. Diagnosing compatibility issues My application fails to connect after enabling TDS 8.0. The error mentions "SSL Provider". What should I check and how do I fix it ? Reviewing linked servers Review my linked server configurations and identify which onesmight have issues with TLS 1.3 requirements. Documenting the changes Generate documentation for our security team explainingthe TLS configuration changes we're implementing for PCI-DSS compliance. It is possible that the AI generated content is incorrect. You remain responsible for reviewing, validating, and approving it before any use. Do not rely on this output without thorough human verification. Not intended for production use. Things to watch out for Linked Servers to legacy systems You may have linked servers that point to: Older SQL Server versions (2016 and earlier) ODBC sources that do not support TLS 1.2/1.3 Thirdparty servers with outdated TLS stacks If you enforce TLS 1.3 everywhere, these connections can fail. Possible approaches: Upgrade the remote systems Use temporary exceptions (not recommended longterm in production) Introduce a proxy or gateway that can handle protocol translation Replication to older versions Transactional or merge replication to subscribers running SQL Server 2019 or earlier may require mixed configurations. Legacy applications Applications using: .NET Framework with older System.Data.SqlClient versions Old ODBC/OLE DB drivers Thirdparty tools that are not updated may need to be upgraded or reconfigured before you enforce TLS 1.3. Suggested deployment strategy These steps are indicative; adapt them to your environment and change management process. Phase 1: Audit Inventory all inbound and outbound connections Identify client types and driver versions Document linked servers and replication paths Phase 2: Testing Enable TLS 1.3 in a staging or preproduction environment Test all critical applications and administrative tools Identify and resolve compatibility issues Phase 3: Progressive rollout Enable TLS 1.3 as an option without forcing it Monitor which connections still negotiate older TLS versions Once all key systems are compatible, enforce TLS 1.3 References TDS 8.0 - SQL Server Configure TLS 1.3 on SQL Server 2022 and later - SQL Server Encrypt Connections by Importing a Certificate - SQL Server Database Engine: Breaking Changes - SQL Server 2025 Certificate Requirements for SQL Server - SQL ServerWireshark walkthrough: observing an Azure SQL Database TDS connection on the wire
One of the most effective ways to truly understand how Azure SQL Database connectivity works is to observe the connection at the packet level. A short Wireshark capture clearly shows the interaction between TCP, TLS, TDS, and the Azure SQL gateway / redirect flow. This section walks through what you should expect to see in a healthy Azure SQL Database connection. 1. Capture setup (high level) Capture traffic on the client side, while reproducing the connection attempt (for example, using sqlcmd, SSMS, or your application). Typical display filters to start with: tcp.port == 1433 If Redirect mode is involved, you may later see additional TCP ports (for example, 11xxx–11999 or a specific dynamic port like 6188). 2. TCP three‑way handshake (transport layer) The first packets are standard TCP: SYN → client initiates connection SYN‑ACK → Azure SQL gateway responds ACK → connection established At this point: The destination IP is the Azure SQL Database gateway The destination port is 1433 This confirms basic network reachability to the gateway. 3. TDS PRELOGIN message (application layer) Immediately after the TCP handshake, Wireshark shows the TDS Pre‑Login packet. Key observations: Protocol: TDS Message type: PRELOGIN This packet contains no credentials It negotiates: TDS version Encryption capability MARS (Multiple Active Result Sets) Federated authentication flags (if applicable) In Wireshark, this often appears as: Tabular Data Stream Packet Type: Pre-Login This is the first SQL‑aware packet on the wire. 4. TLS handshake (security layer) What happens next depends on the TDS version and encryption mode: TDS 7.x (older behavior) PRELOGIN is visible in clear text TLS handshake occurs after PRELOGIN TDS 8.0 / Encrypt=Strict (modern Azure SQL behavior) TLS handshake occurs immediately after TCP PRELOGIN itself is encrypted Wireshark shows: Client Hello Server Hello Certificate exchange In this case: You will not see readable TDS fields unless TLS decryption is configured This is expected and by design 5. LOGIN and authentication phase After TLS is established: Client sends LOGIN7 message Authentication occurs (SQL auth, Azure AD token, managed identity, etc.) All sensitive data is encrypted In Wireshark: Packets are marked as TLS Application Data TDS payload is no longer human‑readable A successful login is followed by: ENVCHANGE tokens Session setup responses 6. Redirect behavior (very important in Azure SQL) If the server is using Redirect connection policy, Wireshark shows a distinctive pattern: Initial connection: TCP → 1433 (gateway) Server responds with routing information Client opens a second TCP connection: New destination IP New dynamic port (for example: 6188) In Wireshark you will see: A new TCP handshake Same SQL hostname, different port The original 1433 connection may close or go idle This is the most common source of confusion: Firewalls allow 1433 ✅ Dynamic redirect ports ❌ blocked Result: login timeout or pre‑login handshake failure 7. Query execution and result sets Once connected to the final database endpoint: Client sends SQL batches or RPC calls Server responds with result sets Traffic continues over the redirected connection In Wireshark: All packets appear as encrypted TLS application data Packet sizes correlate with query result size Long‑running queries show sustained data flow 8. What Wireshark helps you diagnose quickly Using this walkthrough, Wireshark can immediately answer: ✅ Is the client reaching the Azure SQL gateway? ✅ Is TLS handshake completing? ✅ Is Redirect mode in effect? ✅ Is a second TCP connection attempted? ✅ Which port is failing? ✅ Is the failure network‑level or authentication‑level? This makes it invaluable for troubleshooting: Login timeouts Pre‑login handshake errors Redirect / firewall issues TLS / driver encryption mismatches Key takeaway When troubleshooting Azure SQL connectivity: Always expect more than one TCP connection 1433 is only the entry point Redirect mode + TDS + TLS explains 90% of “mysterious” connection failures Once you see the flow in Wireshark, Azure SQL connectivity stops being a black box.516Views0likes0Comments