Cutting back privet hedge / restoring fence

At some point this winter I intend to replace the old rotten fence and posts. As you can see on the photo, the far post is well and truly buried in the privet. I will need to cut the hedge back but I'm not sure when the best time to do it would be? I don't want to kill the hedge. I will be quite happy for it grow back to where it is now?

G,

 

I'd just do it, privet is tough and a hard cutback wont do it any real harm.