Halep appeared to be cruising after winning the first set and going up a service break in the second, but Mertens broke back to level the second set at 4-4.
Halep responded by converting her fifth break-point chance in the next game and served out at love to clinch the match. Halep will next play Anett Kontaveit, who rallied from a set and a break down to beat 18-year-old Iga Swiatek 6-7 (4), 7-5, 7-5 and reach the quarterfinals of a major for the first time.
“Mentally,” Halep said. “I think I did a great job today.”
Kontaveit twice served for the match in the third set but finally clinched it when she broke Swiatek’s serve.
“It was such a battle today. So happy I toughed it out,” the 28th-seeded Kontaveit said. “I was just trying to stay as tough as I could, fight for every point.”
No. 4-seeded Halep reached the Australian Open final in 2018 and won her first major title at the French Open that year.
Muguruza won the 2016 French Open and 2017 Wimbledon. She is unseeded at a Grand Slam tournament for the first time since the 2014 French Open, when she was ranked 35th and also made the quarterfinals.
She came into the Australian Open after withdrawing from her quarterfinal match at the Hobart International due to a virus. But Muguruza said she is feeling better each day.
The Associated Press contributed to this report.