The Belgian won 6-2 3-6 6-1 in one hour and 44 minutes and will now meet her doubles partner for a place in the last four.
"I tried too much, I think, in the second set," Mertens told wtatennis.com.
"In the third set I just tried to step up my game a little bit, a little bit more aggressive, make some balls, just to try to build a point and then go for it."
Sofia Kenin kicked off action from the Camellia Group with a 6-4 6-4 victory over fellow American Alison Riske.
The 20-year-old broke Riske in the fifth game of the opener, and did the same in the third game of the second set to take victory in their first-ever meeting.
"I just feel like I played really well, I served well," said Kenin, who dropped just 12 points in 10 service games and won 84 per cent of the points played behind her first serve.
"Ali is a tough player, she's had a really good year and it wasn't easy for me to play her.
"We're really good friends, but I just tried to put it behind me."
With Kenin now top of the Camellia group, the American duo must play Czech Karolina Muchova to determine who advances into the semi-final.
Number 10 seed Dayana Yastremska made a superb tournament bow in a brutal hitting battle against Donna Vekic.
Yastremska saved two set points in the first set before overcoming number eight seed Vekic 7-6 (6) 6-2.
Vekic is unable to progress out of the Azalea Group having already lost to Kiki Bertens, but Yastremska could write a remarkable story this week after being a last-minute call-up.
"I didn't practice for five days and I completely relaxed," said Yastremska, 19, who had returned home to the Ukraine after an early exit in Moscow last week.
"But I just don't put pressure on myself. I realised that I didn't prepare so good, so I just accepted it and I played how I played."
