Don't act on ButtonReleases when we didn't see the corresponding ButtonPress