I'm confused.
Wasn't your return address on the package?
As far as not getting eBay involved, as long as the buyer is bright enough to use Confirmation of Delivery, he will have proof that he returned and you received the shipment.
He can do that even as part of opening a Dispute much later.
He has 30 days from Delivery to open a Dispute.
When you get it back, if you mark the transaction as Cancelled, your FVF will be refunded.