Oy!
Seems like there’s a limit to pagination of 2,000.
So, that basically means that I can get a max 4000 records so long as the 2000 records doesn’t cause an apex heap size error. In my case it does. 
My workaround is to use the following as a framework in my snippet.
Workaround for offset 2000 limit on SOQL query
