Watson lost the epic match on her 18th double fault as the second-seeded Mladenovic secured a 7-6 (7/4) 6-7 (5/7) 7-5 victory in three hours and 32 minutes.
The match - the longest WTA encounter of 2017 - featured 15 breaks of serve with the Frenchwoman just edging home, converting eight of the 22 break points she had, while Watson won seven of 14.
Mladenovic will face Kirsten Flipkens in the third round.
Top seed Mirjana Lucic-Baroni made comfortable progress to the quarter-finals with a 6-4 6-4 win over Shelby Rogers.
Jelana Ostapenko beat Madison Brengle 7-5 6-4, while Christina McHale got the better of fellow American Taylor Townsend 6-1 7-6 (7/5).
Flipkens advanced after Ajla Tomljanovic - who returned to the court this week after more than a year out - was forced to retire after losing the first set 7-5.
Monica Puif saw off Daniela Hantuchova 6-4 6-4, while there were also wins for Andrea Petkovic and Lesia Tsurenko as seven of the top eight seeds advanced to the last eight.
Ashleigh Barty's comeback gathered more pace with victory over Miyu Kato to reach the quarter-finals of the Malaysian Open in Kuala Lumpur.
The 20-year-old Australian stepped away from the sport in 2014, trying her hand at professional cricket, before returning to tennis last summer.
Barty reached the third round of the Australian Open to climb to 158 in the rankings, and will rise again after this week.
She was totally dominant against Kato, who upset second seed Carla Suarez Navarro in round one, winning 6-0 6-3 in just 58 minutes.
Fifth seed Duan Ying-ying made it four Chinese players into the last eight with a battling 6-0 4-6 7-5 win over Sara Sorribes Tormo, joining Han Xinyun, Wang Qiang and Zhang Kai-lin in the quarter-finals.
But fourth seed Peng Shuai retired from her match against Magda Linette while trailing 5-2.
