Have the locks rekeyed.
It will keep from entering whenever he wants to.
And if he comes by, or has an agent come by and they can't get in, and he tries to give you s@#$, tell him that until he provides you with a copy of the lease, he doesn't get a new key.
Put it in a letter once you have...