You need to add a Salesforce Remote Site Setting corresponding to the Domain portion of your Authentication Provider’s Authorize Endpoint URL and/or Token Endpoint URL, if they have different Domains then you will need to add a Remote Site Setting for each.
For example, Sharepoint Online has a different domain for the Authorize Endpoint URL required than for the Token Endpoint URL:
Authorize Endpoint URL: https://acme.sharepoint.com/_layouts/15/OauthAuthorize.aspx
Token Endpoint URL: https://accounts.accesscontrol.windows.net/acme.onmicrosoft.com/tokens/OAuth/2
For this setup, you would need to add TWO Remote Site Settings (by going to Setup > Remote Site Settings, and creating two separate Remote Site Settings, one for each of the following domains:
Remote Site Setting 1 URL: https://acme.sharepoint.com
Remote Site Setting 2 URL: https://accounts.accesscontrol.windows.net