Forum Discussion
APS to Synapse Migration (Datetime datatype issue)
Hi Everyone,
I am seeking help to fix the issue below.
I followed the steps from github for migrating APS to Synapse.
Link: https://github.com/microsoft/AzureSynapseScriptsAndAccelerators
Source Date values:
Sample External table Script:
CREATE EXTERNAL TABLE [DIM].[account]
(
[ID] [bigint] NOT NULL,
[EffectiveFrom] [datetime] NOT NULL,
[EffectiveTo] [datetime2] NOT NULL,
[ValidFrom] [datetime] NOT NULL,
[ValidTo] [datetime2] NOT NULL
)
WITH (DATA_SOURCE = [datasource_abfss], LOCATION = N'/locxxxxxxx', FILE_FORMAT = [parquet_file_format], REJECT_TYPE = VALUE, REJECT_VALUE = 0 )
GO
SELECT TOP (100) [ID]
,[EffectiveFrom]
,[EffectiveTo]
,[ValidFrom]
,[ValidTo]
FROM [DIM].[account]
GO
Facing challenges for reading datetime columns
A Parquet file was unable to be exported from an ADLS2 blob into an External table.
Here are the Errors:
HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopSqlException: Arithmetic overflow error converting timestamp to data type DATETIME.
*******************************************************************************************************************
2nd try to read from parquet file using Auto generated scripts:
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/**',
FORMAT = 'PARQUET'
) AS [result]
Error:
Started executing query at Line 1
Error handling external file: 'Inserting value to batch for column type DATETIME2 failed. Invalid argument provided.'. File/External table name: ''https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/account.snappy.parquet'.
Your assistance in advance is greatly appreciated !