Can Rich Returns impact Shopify's new Site Speed Score in a negative way?

No. We don't make changes to your store or any of your templates from our side that would have an effect on your site's loading speed.