On Facebook, mouse over the the right side of a challenge next to where it says "Accept," "Play," "Their Turn," or "Nudge." You'll see a little grey wastebasket appear which you can click to end the game with that player. 

For iOS and Android, on a specific player's challenge, drag your finger across the screen from right to left. A red "Delete" button will appear- tap that to delete the challenge or match.