I miss him terribly?

Just go with your heart, you shouldn't have to listen to what other people say. They should be happy for you.