Sharing sharepoint site with guest user - site reloads and asks for credentials again

We have 2 O365 tenants, A and B, the users of tenant B are guests/external users in tenant A and are shown as (name)#EXT#@DomainA.onmicrosoft.com

B can login to the Sharpoint portal of A, they can briefly see the content but after 3 seconds they get kicked out and asked for password, which doesn't work. We tried resetting their password in B but that doesn't work, all users of B have this issue.

We also noticed that, even for the users of A in their own tenant, the Sharepoint site reloads after about 3 seconds but the user remains logged in.

Why does this happen? Is this a bug or a configuration issue?