Microsoftmaxha There could be multiple possibilities that could make the logstash pipeline running slow. Recently, I've migrated about 40TB+ from ELK to ADX without any issues.
Few things to check it out:
1. Parallelize the pipeline runs on multiple instances (VMs) to speed up the migration process. In the pipeline, you could provide Kibana query based on document creation date or any other property that would limit to a subset of a data.
2. Sometimes, network latency could add a bottleneck. Does your ELK and ADX are in the same region?
3. Based on my experiments with the ADX logstash plugin, it is pretty fast and reliable. The ADX output plugin runs in batches and streams each batch to ADX instance from the backend. Try checking out if there is any latency occurring in reading the data from the ELK instance. To check on this, use a simple file output plugin and check how fast the data is read from ELK and writing to a file.
From my migration experience, it was about 20-25M records every hour and parallelizing the pipeline runs on multiple instances could speed up the process.
Let me know how it goes.
