Natwar Singh has decided to quit to save his party the "embarassment". But his family is clearly unhappy over being made a scapegoat.