I added my ID to a remote system out of curiosity. I later deleted it, and was no longer able to connect to the system because it said I was on the blacklist, which I was not. Deleting a whitelist entry should mean just that; it should have no impact/relevence to the blacklist.
v13.1.3206 on Ubuntu, both ends.