I assume that when you say you swapped the speakers over, the fault moved to the other speaker.
Mucky pots are usually crackly when you rotate them, and it's unusual to only affect one side.
Definitely worth poking about for dry joints with the iron, as Jocko suggests. Maybe you disturbed something while fiddling with the pot?