Referee Darren Drysdale has been charged with improper conduct over his confrontation with Ipswich midfielder Alan Judge.
The incident happened in the closing stages of Ipswich’s goalless draw against Northampton at Portman Road on Tuesday night.
After Judge had been given a yellow card, the Ipswich player approached the referee.
A confrontation between the two briefly ensued, with Drysdale appearing to move his head towards Judge, before Ipswich’s Flynn Downes and Jack Lankester and Northampton defender Lloyd Jones stepped in and calmed the situation.
There was a red card shortly afterwards, with Downes picking up a second yellow card.
Drysdale apologised to both Judge and Ipswich for his actions on Wednesday afternoon but the FA has now charged him with improper conduct.
However, Judge insists an apology is unnecessary and doesn't even want action taken against Drysdale.
He told the club's website: "Referees have a tough job to do and it was heat of the moment stuff that happens in football. The photo makes it look worse than it was and to be honest, the matter was finished with from my end as soon as I walked off the pitch.
"I wasn’t looking for the referee to apologise; I wasn’t looking for him to be charged. There was never going to be a complaint from me and I made that clear.
“Like I said, for me - it was finished with straight away."
Drysdale has been removed from taking charge of the League Two fixture between Southend and Bolton on Saturday.
An FA statement read: “Darren Drysdale has been charged with a breach of FA Rule E3 following last night’s EFL League One game between Ipswich Town FC and Northampton Town FC.
“It is alleged that the match official’s behaviour during the 90th minute of the game amounts to improper conduct and he has until Thursday 4 March 2021 to provide a response.”
Drysdale had said in his earlier statement, released by the Professional Game Match Officials Limited before he was charged by the FA: “I fully understand that it is important for us as referees to maintain our composure throughout the game and always engage with players in a professional manner.
“I’m sorry that I did not do that last night and I can only apologise to Alan and Ipswich Town.”