Bulk Insert causing duplicates in target table
We use ETL tool(C# based) which uses Bulkcopy method to do bulk insert to target table. Recently we are facing 2 Issues in our daily loads 1. Target table with no primary key - the load returns success but it inserts 5x times of the same record. It loads same records 5 times. 2. Target table with Primary Key - The load returns failure with primary key constraint but it inserts the rows into target table. DBA Team is unable to find anything in there logs ,So I am confused why this error occurs. This occurs daily in 10% of the jobs and once you rerun after 2-3 hrs the same job finishes successfully without above mentioned issues. Please suggest how can I debug this issue.Lesson Learned #479:Loading Data from Parquet to Azure SQL Database using C# and SqlBulkCopy
In the realm of big data and cloud computing, efficiently managing and transferring data between different platforms and formats is paramount. Azure SQL Database, a fully managed relational database service by Microsoft, offers robust capabilities for handling large volumes of data. However, when it comes to importing data from Parquet files, a popular columnar storage format, Azure SQL Database's native BULK INSERT command does not directly support this format. This article presents a practical solution using a C# console application to bridge this gap, leveraging the Microsoft.Data.SqlClient.SqlBulkCopy class for high-performance bulk data loading.Lesson Learned #391: Data Loading - Parallel BULK INSERTs with JDBC and Azure Blob Storage
Today, we got a service request that our customer is running in multiple threads using prepared statement. In this article, I would like explore a Java code snippet used to test a connection to a SQL Server database and execute load bulk data using the BULK INSERT statement. We will explain the purpose and advantages of using the BULK INSERT statement for large-scale data inserts. Additionally, I will examine each part of the code in detail and explain its functionality.
Access Firewall Protected (Select Vnet) Azure Storage from Azure SQL Database
i have a storage account which is firewall protected and Azure SQL. Both are in same tenant /subscription/resource group. I am unable to access the Blob from Azure SQL(Bulk import). I have tried "Resource instances" feature. But its not working. Can you anyone guide me how to solve this?Access Firewall Protected (Select Vnet) Azure Storage from Azure SQL Database
I have a storage account which is firewall protected and Azure SQL. Both are in same tenant /subscription/resource group. I am unable to access the Blob from Azure SQL(Bulk import). I have tried "Resource instances" feature. But its not working. Can you anyone guide me how to solve this?Fail to OPENROWSET insert geometry data from parquet
Hi! I'm trying to insert geometry data (WKB) from a parquet file to a data base table. I manage to get it working for small geometries but it's failing for lager geometriers. Here is the code: INSERT db.s.t SELECT geometry::STGeomFromWKB(a.geom,1) FROM OPENROWSET ( BULK 'path_to.parquet', FORMAT = 'PARQUET', DATA_SOURCE = 's3_sorce' ) AS a The failure: String or binary data would be truncated while reading column of type 'VARBINARY(8000)'. Check ANSI_WARNINGS option.... Seems as if the geometry type is not recognized. Is there a way around this?
