Taxanomy metadata is not copied, even though I am reusing the terms

I have two document libraries.
Both have two different site columns:

Site Column A refers to Metadata Termset A

Site Column B refers to Metadata Termset B

Some very important terms are reused via "reuse" in the termstore and are shared between Termset A and Termset B.

I am now tagging some documents with those shared terms.

But when I move or copy the files between the libraries the term values get lost. Even though the term id is the same in both.

Any idea how I could fix that behaviour?