Forum Discussion

Santhosh0579's avatar
Santhosh0579
Copper Contributor
Jun 07, 2023

APS to Synapse Migration (Datetime datatype issue)

Hi Everyone,

I am seeking help to fix the issue below.

I followed the steps from github for migrating APS to Synapse.

Link: https://github.com/microsoft/AzureSynapseScriptsAndAccelerators

 

Source Date values:

 

Sample External table Script:

CREATE EXTERNAL TABLE [DIM].[account]
(
[ID] [bigint] NOT NULL,
[EffectiveFrom] [datetime] NOT NULL,
[EffectiveTo] [datetime2] NOT NULL,
[ValidFrom] [datetime] NOT NULL,
[ValidTo] [datetime2] NOT NULL
)
WITH (DATA_SOURCE = [datasource_abfss], LOCATION = N'/locxxxxxxx', FILE_FORMAT = [parquet_file_format], REJECT_TYPE = VALUE, REJECT_VALUE = 0 )
GO

SELECT TOP (100) [ID]
,[EffectiveFrom]
,[EffectiveTo]
,[ValidFrom]
,[ValidTo]
FROM [DIM].[account]
GO

 

Facing challenges for reading datetime columns

A Parquet file was unable to be exported from an ADLS2 blob into an External table.

Here are the Errors:

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopSqlException: Arithmetic overflow error converting timestamp to data type DATETIME.

*******************************************************************************************************************

2nd try to read from parquet file using Auto generated scripts:

SELECT

    TOP 100 *

FROM

    OPENROWSET(

        BULK 'https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/**',

        FORMAT = 'PARQUET'

    ) AS [result]

 

Error: 

Started executing query at Line 1

Error handling external file: 'Inserting value to batch for column type DATETIME2 failed. Invalid argument provided.'. File/External table name: ''https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/account.snappy.parquet'.

 

Your assistance in advance is greatly appreciated !

 

No RepliesBe the first to reply

Resources