Should I say yes to him?

Why are you on Yahoo Answers asking this..you need to learn to make your own decisions without the help of people on the internet. Obviously he isn't very important to you if you're going to say no just because people are going to say things.