Do you think I should be a little bit more firm with him?

Even if you DID hurt his feelings who cares. You get reject'd sometimes, that's life. That's no reason to put someone down. It's not like you were doing that to him when you said no. You should let him know what it is you want && don't want from him. But don't feel like it's YOUR fault he cannot handle rejection well..