Batch enpoint / AutoML BatchScoring

[JimmyLab]

Hello Everyone. 
I am currently strugling with the excecution of a Data factory pipeline that calls a ML model that I made on AutoML.

This is the pipeline model that im using
(I pretty much I followed the steps of this doc
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-batch-azure-data-factory?view=az...)

But at the time I try to excecute a 'parquet file' , it doesn't read the data.

Now, apart from this, I followed a tutorial on how to make a AutoML classification model.
https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-first-experiment-automated-ml?view...

When I did this, I noticed that the blob containing this file, made two extra folders, in this case, data was readable and my model was excecuted succesfull.

As I tried to solve the issue I moved the parquet file to said folder. Data was readable, but it still gave me "error".

This is how the parquet looks if uploaded from it's default position in the blob storage

The error in both Batch Scoring outputs look almost the same:

[execution failed. User process 'python' exited with status code 42. Please check log file 'user_logs/std_log_1.txt' for error details. Error: Traceback (most recent call last):
File "driver/amlbi_main.py", line 275, in <module>
main()
File "driver/amlbi_main.py", line 226, in main
sys.exit(exitcode_candidate)
SystemExit: 42]

[Execution failed. User process 'python' exited with status code 42. Please check log file 'user_logs/std_log_1.txt' for error details. Error: Traceback (most recent call last):
File "driver/amlbi_main.py", line 275, in <module>
main()
File "driver/amlbi_main.py", line 226, in main
sys.exit(exitcode_candidate)
SystemExit: 42]

In 'User Logs'

[Azure Machine Learning Batch Inference Start

[2024-04-02 15:54:32.614574] No started flag set. Skip creating started flag.

Azure Machine Learning Batch Inference End

Cleaning up all outstanding Run operations, waiting 300.0 seconds

2 items cleaning up...

Cleanup took 0.18171906471252441 seconds

Traceback (most recent call last):

  File "driver/amlbi_main.py", line 275, in <module>

    main()

  File "driver/amlbi_main.py", line 226, in main

    sys.exit(exitcode_candidate)

SystemExit: 42]

And In Logs - Job_error

[

The process002 on node 1 failed with error: Traceback (most recent call last):

  File "/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/driver/azureml_user/parallel_run/payload_thread.py", line 428, in call_init

    self.scoring_module.call_init()

  File "/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/driver/azureml_common/parallel_run/stderr_converter.py", line 29, in wrapper

    res = method(self, *args, **kwargs)

  File "/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/driver/azureml_user/parallel_run/score_module.py", line 322, in call_init

    result = self.init()

  File "/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/scoring_file_v_2_0_0.py", line 46, in init

    model = joblib.load(model_path)

  File "/azureml-envs/azureml-automl/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 650, in load

    with open(filename, 'rb') as f:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/4d856a4a-700f-4f51-97a2-649973585f27_score_model/model.pkl'

]

I'd  like to undertand:
1.why is the parquet readable if it is taken from this said folder that automatically created from the classification csv model?
2. why the BatchScoring error keeps saying "No such file or directory: '/mnt/azureml/cr/j/17cbff753b2343c58e0564f6ad455dab/exe/wd/4d856a4a-700f-4f51-97a2-649973585f27_score_model/model.pkl'" if I never gave it that certain file or directory (I couldn't find it manually) ? 

Thank you so much for taking your time  to read this.