You should only be charging taxes to Canadian buyers, and of course only if you are registered to remit taxes.
There are no Canadian federal or provincial taxes payable by foreign customers.
I am unclear about how taxes are calculated when the Canadian customer lives in a province that does not collect HST, but basically it is based on the postal code. I'll let someone who does add tax (I absorb it into my asking price) tell you more about that.