Forum Widgets
Latest Discussions
validating record count at SQL server database tables with migrated azure data lake gen2
we are migrating out project from onpermise to azure , so on-premisedatabase is the SQL server that we are using and azure datalake gen2 is the storage location where store data currently and so far we are currently validating record count of each table manually from sql server database tables and similarly we write pyspark code in databricks to write those data as parquet file and we validate record count from pyspark manually every timewhich is time consuming is that possible execute this process to make it automated in order to save time ? can this be done by using pyspark code ?
APS to Synapse Migration (Datetime datatype issue)
Hi Everyone, I am seeking help to fix the issue below. I followed the steps from github for migrating APS to Synapse. Link:https://github.com/microsoft/AzureSynapseScriptsAndAccelerators Source Date values: Sample External table Script: CREATE EXTERNAL TABLE [DIM].[account] ( [ID] [bigint] NOT NULL, [EffectiveFrom] [datetime] NOT NULL, [EffectiveTo] [datetime2] NOT NULL, [ValidFrom] [datetime] NOT NULL, [ValidTo] [datetime2] NOT NULL ) WITH (DATA_SOURCE = [datasource_abfss], LOCATION = N'/locxxxxxxx', FILE_FORMAT = [parquet_file_format], REJECT_TYPE = VALUE, REJECT_VALUE = 0 ) GO SELECT TOP (100) [ID] ,[EffectiveFrom] ,[EffectiveTo] ,[ValidFrom] ,[ValidTo] FROM [DIM].[account] GO Facing challenges for reading datetime columns A Parquet file was unable to be exported from an ADLS2 blob into an External table. Here are the Errors: HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopSqlException: Arithmetic overflow error converting timestamp to data type DATETIME. ******************************************************************************************************************* 2nd try to read from parquet file using Auto generated scripts: SELECT TOP 100 * FROM OPENROWSET( BULK 'https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/**', FORMAT = 'PARQUET' ) AS [result] Error: Started executing query at Line 1 Error handling external file: 'Inserting value to batch for column type DATETIME2 failed. Invalid argument provided.'. File/External table name: ''https://xxxxx.dfs.core.windows.net/srcblob/DIM_account/account.snappy.parquet'. Your assistance in advance is greatly appreciated !
Enhance Data Migration Assistant
When you try to access an Azure target DB there are only two azure options: - Azure Directory Integrated Auth - Azure Directory Password Auth Now that also MFA is afforded in Azure this optionshould be integrated in here also! So there should be a solution for selecting the possible Auth-providers like in SQL-Management Studio. Best regards.
DMA - CREATE button is disabled with SQL DB Managed Instance as TARGET option.
Hi Everyone, Looking for some guidance re SQL DB migration through DMA tool. When I select "target server type" as "Azure SQL Database Managed Instance" the CREATE button remains disabled (cant click). If I choose any other option for "target server type" CREATE button does gets active. It is just when I select the ..managed instance.. as the target server type option CREATE does not become enabled. I have attached the screenshot. Please advise. This is my first post here, if you see I should be posting in some other forum please guide. Thank you!Azure Migrate :: online migration only available to Managed Instance
Hello, I found out that Azure Migrate provides online migration to Managed Instance only. Azure SQL DB are not covered. Has it always been like that or is that something new? The only documentation I find online is about Transactional Replication to Azure SQL DB and is something created by a normal user, nothing official: https://www.youtube.com/watch?v=7QI1VjbS0YI&t=476s&ab_channel=RajasekharReddyBolla Does anyone know how to do online replication to Azure SQL DB?
