Jeffrey, for my knowledge you can't do that as far as your pivot chart is linked to the pivot table, without deleting fields in both. IMHO, the easiest way to break that link is to create the pivot chart independently from pivot table. Most probably your data is in data model (default option when you create pivot table), if so insert power chart from Power Pivot menu, without creating another pivot table. If you don't work with data model for any reason, second pivot table could be created in separate worksheet (which you may hide later on) to create the power chart on it.
One option would be to make a copy of the Pivot table and remove the 2 unwanted columns for your chart, you could then use Slicers to control both Pivot Tables at the same time (assuming you have Excel 2010 or later).
i.e. one pivot table contains all the data you want to see and one controls your chart