Aggressive drivers when trying to cross Mare Street to access Tudor Road

More than once I have been waiting at Warburton Road as it meets Mare Street for a break in traffic so that I can cross to Tudor Road (which is filtered for motor traffic so a nice way to reach Victoria Park) when a car behind me has been very agressive. On one occasion I was "nudged" by the driver literally driving into my back wheel.

I think the aggression stems from:
a) drivers using this route precisely because they are in a hurry / rat running to avoid the traffic lights on Mare Street, so not being willing to be held up
b) drivers not understanding "why doesn't he go?" when it would have been possible for me to turn left (as the driver wants to). I need enough clear space on both sides of the road to go straight across.