How would you respond?

He is setting boundaries, just like you've asked him to. Except where he draws the boundaries is different from where you want him to draw them. He might want to continue a relationship with you, but him and his stbx go way, way back, they share a daughter, and in his judgment his friendship with the ex is justified. As much as you'd like to think that you have a say, you really don't. If the situation is too uncomfortable for your liking, I suggest you look for a man with less baggage.