How can you possibly justify retaining the rights to something the client has paid for. By default he pays for all the work you do, including the copyright unless you have already registered this copyright with the authorities.

If the client pays you to design something from scratch, including the source file, he has paid you for all of it, not just a piece of it, which is rendered useless if you no longer work for him.

You guys know what you are doing when you agree to design and build a site for somebody, the client doesn't. This issue should not be kept in the smallprint of a jargon filled contract that the client probably doesn't understand.

It is underhand to suggest keeping the source file if he paid you to write it. (If you added an existing source file that you had already written and have registered it, that is a different story).