Fail to OPENROWSET insert geometry data from parquet
Hi! I'm trying to insert geometry data (WKB) from a parquet file to a data base table. I manage to get it working for small geometries but it's failing for lager geometriers. Here is the code: INSERT db.s.t SELECT geometry::STGeomFromWKB(a.geom,1) FROM OPENROWSET ( BULK 'path_to.parquet', FORMAT = 'PARQUET', DATA_SOURCE = 's3_sorce' ) AS a The failure: String or binary data would be truncated while reading column of type 'VARBINARY(8000)'. Check ANSI_WARNINGS option.... Seems as if the geometry type is not recognized. Is there a way around this?Lesson Learned #479:Loading Data from Parquet to Azure SQL Database using C# and SqlBulkCopy
In the realm of big data and cloud computing, efficiently managing and transferring data between different platforms and formats is paramount. Azure SQL Database, a fully managed relational database service by Microsoft, offers robust capabilities for handling large volumes of data. However, when it comes to importing data from Parquet files, a popular columnar storage format, Azure SQL Database's native BULK INSERT command does not directly support this format. This article presents a practical solution using a C# console application to bridge this gap, leveraging the Microsoft.Data.SqlClient.SqlBulkCopy class for high-performance bulk data loading.Lesson Learned #391: Data Loading - Parallel BULK INSERTs with JDBC and Azure Blob Storage
Today, we got a service request that our customer is running in multiple threads using prepared statement. In this article, I would like explore a Java code snippet used to test a connection to a SQL Server database and execute load bulk data using the BULK INSERT statement. We will explain the purpose and advantages of using the BULK INSERT statement for large-scale data inserts. Additionally, I will examine each part of the code in detail and explain its functionality.
Bulk Insert causing duplicates in target table
We use ETL tool(C# based) which uses Bulkcopy method to do bulk insert to target table. Recently we are facing 2 Issues in our daily loads 1. Target table with no primary key - the load returns success but it inserts 5x times of the same record. It loads same records 5 times. 2. Target table with Primary Key - The load returns failure with primary key constraint but it inserts the rows into target table. DBA Team is unable to find anything in there logs ,So I am confused why this error occurs. This occurs daily in 10% of the jobs and once you rerun after 2-3 hrs the same job finishes successfully without above mentioned issues. Please suggest how can I debug this issue.Access Firewall Protected (Select Vnet) Azure Storage from Azure SQL Database
I have a storage account which is firewall protected and Azure SQL. Both are in same tenant /subscription/resource group. I am unable to access the Blob from Azure SQL(Bulk import). I have tried "Resource instances" feature. But its not working. Can you anyone guide me how to solve this?Access Firewall Protected (Select Vnet) Azure Storage from Azure SQL Database
i have a storage account which is firewall protected and Azure SQL. Both are in same tenant /subscription/resource group. I am unable to access the Blob from Azure SQL(Bulk import). I have tried "Resource instances" feature. But its not working. Can you anyone guide me how to solve this?
