tds
2 TopicsLessons Learned #554: I Have Used BCP for Years — But How Does It Actually Work?
BCP is one of those SQL Server tools that many of us have been using for years. It is simple, reliable, and extremely fast when we need to move large amounts of data. We execute it, and millions of rows can be loaded into SQL Server or Azure SQL Database. But after using BCP many times, I had a question that stayed in my mind: Why is BCP so fast? And that question led me to another one: What does BCP actually send to SQL Server? Is it simply executing thousands or millions of INSERT statements behind the scenes? And finally: Could I create my own BCP-like application in C++, .NET, Java, or another programming language and adapt it to my own requirements? That curiosity is what motivated this article. The answer is quite interesting. BCP is not simply a faster way to execute INSERT statements. It uses a different model for moving data. A Very Simple BCP Example A typical import might look like this: bcp MyDatabase.dbo.Customers in customers.csv -S myserver.database.windows.net -d MyDatabase -c -t"," -T The bcp utility bulk copies data between SQL Server and a data file in a user-specified format. Microsoft also exposes bulk-copy functionality programmatically through SQL Server drivers. At a very high level, we normally think of the process as: File -> BCP -> SQL Server. But there is much more happening in between. My First Question: Is BCP Sending INSERT Statements? Imagine that we need to load one million rows. A traditional application could repeatedly execute something like: INSERT INTO dbo.Customers ( CustomerId, CustomerName, Amount ) VALUES ( @CustomerId, @CustomerName, @Amount ); Even when the connection is reused and the statement is parameterized, we are still performing a very large number of individual database operations. BCP works differently. And this is probably the most important concept in this article: BCP is not a faster way to send millions of INSERT statements. It avoids sending those INSERT statements in the first place. Instead, BCP establishes a bulk data stream with SQL Server. Going One Level Lower: TDS SQL Server clients communicate with the database engine using TDS — Tabular Data Stream. TDS carries SQL requests, result sets, metadata, errors, authentication information, bulk data, and other messages between the client and SQL Server. For bulk loading, TDS defines a dedicated bulk-load stream. At a simplified level, it looks like this: Conceptually, SQL Server first receives metadata describing the incoming row shape, followed by the row stream and a completion token. This is a completely different model from repeatedly submitting: INSERT INSERT INSERT INSERT INSERT SQL Server is told what the incoming data looks like, and then rows are streamed continuously through the connection. There Is an INSERT BULK Operation This was one of the details I found most interesting. When we execute bcp.exe, we do not manually write an INSERT BULK statement. However, the bulk-load protocol requires the client to identify the destination and incoming structure before sending the actual row stream. A simplified sequence looks like this: The important point is that the rows themselves are not represented as complete SQL statements. They are part of a structured bulk stream. What Is COLMETADATA? Suppose our destination table is: CREATE TABLE dbo.Customers ( CustomerId int, CustomerName varchar(100), Amount decimal(10,2) ); Before SQL Server can interpret the incoming row data, it needs information about the shape and types of the columns. Conceptually, the metadata describes something similar to: Column 1 INT Column 2 VARCHAR Length = 100 Column 3 DECIMAL Precision = 10 Scale = 2 Then the row stream follows: ROW 1001 Juan 42.50 ROW 1002 Pedro 18.75 ROW 1003 Jose 91.00 The real TDS representation is binary and structured, not human-readable like this, but the mental model is useful:BCP sends structured row data through the bulk protocol instead of generating SQL text for every record. From a CSV File to SQL Server Imagine the source file contains: 1001,Juan,42.50 1002,Pedro,18.75 1003,Jose,91.00 and we execute: bcp MyDatabase.dbo.Customers in customers.csv -S myserver -c -t"," -T Because we are using character mode, the source contains character representations. BCP reads the file, recognizes the fields according to the specified format, and feeds the values into the bulk-copy path. For example, the character value "1001" ultimately has to become an INT, and "42.50" has to become a DECIMAL(10,2). So BCP is fast, but it is not magic. Data still has to be interpreted and converted into the destination SQL Server data types. What About Memory? This was another thing I wanted to understand. Suppose I need to import a 100-GB file. Does BCP need 100 GB of memory? No. A better mental model is a streaming pipeline. The complete file does not need to reside in memory at the same time. Buffers can be filled, transmitted, reused, filled again, and transmitted again. The amount of data being transferred is not the same as the amount of memory required to transfer it. A Row Is Not Necessarily a Network Packet Sending one row through the BCP API does not necessarily mean that one network packet is immediately sent for that row. The bulk-copy implementation can accumulate rows while filling network packets. Conceptually: Row 1 ----\ Row 2 -----\ Row 3 ------> Network packet ---> SQL Server Row 4 -----/ Row 5 ----/ This is much more efficient than treating every row as an independent network operation. The command-line utility also exposes a packet-size option through -a packet_size. Larger is not automatically better; the optimal value depends on the environment, row size, driver, network, and workload. Then I Discovered the BCP API This was probably my favorite part of the investigation. BCP is not only a command-line utility. Microsoft exposes bulk-copy functionality programmatically through the ODBC bulk-copy API. Among the available functions are: bcp_init() bcp_bind() bcp_sendrow() bcp_batch() bcp_done() That means we can create our own bulk loader instead of launching bcp.exe. Understanding bcp_bind() Imagine that our application contains: int CustomerId; char CustomerName[100]; With the BCP API, those variables can be associated with destination columns. The application can then populate the same variables repeatedly and call bcp_sendrow() to feed another row into the bulk stream. CustomerId = 1; strcpy(CustomerName, "Customer A"); bcp_sendrow(hdbc); CustomerId = 2; strcpy(CustomerName, "Customer B"); bcp_sendrow(hdbc); The application is not building and submitting a new INSERT INTO... statement each time. It keeps populating values and feeding rows into the existing bulk-copy pipeline. The Source Does Not Even Need to Be a File Once we understand the API, another interesting possibility appears: the source can be anything our application knows how to read. For example, the source could be a REST API, a message stream, another database, generated data, or an in-memory structure. bcp.exe is therefore one implementation of the bulk-copy concept, not the only way to use it. Can I Create My Own BCP in Another Programming Language? Yes. The exact interface depends on the language and driver. C / C++ The ODBC BCP API gives low-level access using functions such as: bcp_init bcp_bind bcp_sendrow bcp_batch bcp_done .NET For .NET applications, Microsoft provides SqlBulkCopy : using var connection = new SqlConnection(connectionString); await connection.OpenAsync(); using var bulkCopy = new SqlBulkCopy(connection); bulkCopy.DestinationTableName = "dbo.Customers"; bulkCopy.BatchSize = 10000; await bulkCopy.WriteToServerAsync(reader); Java The Microsoft JDBC Driver provides SQLServerBulkCopy : SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(connection); bulkCopy.setDestinationTableName("dbo.Customers"); bulkCopy.writeToServer(resultSet); The important concept is that the same bulk-copy model can be consumed directly by applications. Batches: Another Piece of the Puzzle There is another important concept: transaction batching. Using the BCP API, an application can send multiple rows and then call: bcp_batch(); The command-line utility exposes the same general idea through -b batch_size. Batching can affect transaction duration, rollback scope, transaction-log behavior, error recovery, and throughput. Batch size is not the same thing as memory buffer size. If we specify -b 100000, we are primarily defining a transactional boundary. We are not saying that BCP must hold exactly 100,000 rows in RAM. What Happens When the Stream Arrives at SQL Server? Successfully delivering rows to SQL Server is only part of the operation. On the server side, a simplified path looks like this: TDS bulk stream-> Decode metadata and rows -> Bulk-load processing -> Storage Engine -> Data pages -> Transaction log SQL Server still needs to store those rows. Depending on the destination, that can involve data-type processing, page allocation, transaction logging, index maintenance, constraint validation, identity handling, triggers, locking, and dirty data pages. So Why Is BCP So Fast? After looking at the mechanism, I do not think there is one single magic optimization. BCP is fast because several efficiencies work together. Traditional row-by-row processing BCP / Bulk Copy Many individual SQL operations Continuous bulk stream Repeated execution overhead Bulk-oriented processing Potentially many client/server interactions Persistent streaming pipeline SQL statement representation Structured row representation Small network operations possible Rows can be packed efficiently into packets Frequent transaction boundaries possible Controlled batching Generic DML execution pattern Bulk-copy-oriented path For a few rows, the difference might not matter much. Multiply those savings by millions of rows, and the result becomes significant. References Microsoft Learn — bcp utility Microsoft Learn — Import and export bulk data using bcp Microsoft Open Specifications — TDS Bulk Load BCP Microsoft Learn — bcp_bind Microsoft Learn — bcp_sendrow Microsoft Learn — SqlBulkCopy Microsoft Learn — SQLServerBulkCopy530Views0likes0CommentsWireshark walkthrough: observing an Azure SQL Database TDS connection on the wire
One of the most effective ways to truly understand how Azure SQL Database connectivity works is to observe the connection at the packet level. A short Wireshark capture clearly shows the interaction between TCP, TLS, TDS, and the Azure SQL gateway / redirect flow. This section walks through what you should expect to see in a healthy Azure SQL Database connection. 1. Capture setup (high level) Capture traffic on the client side, while reproducing the connection attempt (for example, using sqlcmd, SSMS, or your application). Typical display filters to start with: tcp.port == 1433 If Redirect mode is involved, you may later see additional TCP ports (for example, 11xxx–11999 or a specific dynamic port like 6188). 2. TCP three‑way handshake (transport layer) The first packets are standard TCP: SYN → client initiates connection SYN‑ACK → Azure SQL gateway responds ACK → connection established At this point: The destination IP is the Azure SQL Database gateway The destination port is 1433 This confirms basic network reachability to the gateway. 3. TDS PRELOGIN message (application layer) Immediately after the TCP handshake, Wireshark shows the TDS Pre‑Login packet. Key observations: Protocol: TDS Message type: PRELOGIN This packet contains no credentials It negotiates: TDS version Encryption capability MARS (Multiple Active Result Sets) Federated authentication flags (if applicable) In Wireshark, this often appears as: Tabular Data Stream Packet Type: Pre-Login This is the first SQL‑aware packet on the wire. 4. TLS handshake (security layer) What happens next depends on the TDS version and encryption mode: TDS 7.x (older behavior) PRELOGIN is visible in clear text TLS handshake occurs after PRELOGIN TDS 8.0 / Encrypt=Strict (modern Azure SQL behavior) TLS handshake occurs immediately after TCP PRELOGIN itself is encrypted Wireshark shows: Client Hello Server Hello Certificate exchange In this case: You will not see readable TDS fields unless TLS decryption is configured This is expected and by design 5. LOGIN and authentication phase After TLS is established: Client sends LOGIN7 message Authentication occurs (SQL auth, Azure AD token, managed identity, etc.) All sensitive data is encrypted In Wireshark: Packets are marked as TLS Application Data TDS payload is no longer human‑readable A successful login is followed by: ENVCHANGE tokens Session setup responses 6. Redirect behavior (very important in Azure SQL) If the server is using Redirect connection policy, Wireshark shows a distinctive pattern: Initial connection: TCP → 1433 (gateway) Server responds with routing information Client opens a second TCP connection: New destination IP New dynamic port (for example: 6188) In Wireshark you will see: A new TCP handshake Same SQL hostname, different port The original 1433 connection may close or go idle This is the most common source of confusion: Firewalls allow 1433 ✅ Dynamic redirect ports ❌ blocked Result: login timeout or pre‑login handshake failure 7. Query execution and result sets Once connected to the final database endpoint: Client sends SQL batches or RPC calls Server responds with result sets Traffic continues over the redirected connection In Wireshark: All packets appear as encrypted TLS application data Packet sizes correlate with query result size Long‑running queries show sustained data flow 8. What Wireshark helps you diagnose quickly Using this walkthrough, Wireshark can immediately answer: ✅ Is the client reaching the Azure SQL gateway? ✅ Is TLS handshake completing? ✅ Is Redirect mode in effect? ✅ Is a second TCP connection attempted? ✅ Which port is failing? ✅ Is the failure network‑level or authentication‑level? This makes it invaluable for troubleshooting: Login timeouts Pre‑login handshake errors Redirect / firewall issues TLS / driver encryption mismatches Key takeaway When troubleshooting Azure SQL connectivity: Always expect more than one TCP connection 1433 is only the entry point Redirect mode + TDS + TLS explains 90% of “mysterious” connection failures Once you see the flow in Wireshark, Azure SQL connectivity stops being a black box.516Views0likes0Comments