The former prime minister started the day being heckled by four protesters and finished it by being booed as he left