Romania’s Halep, who won the Madrid title in 2016 and 2017, is still yet to reach a WTA Tour final this season and this was her earliest exit in the Spanish capital since a 2015 first-round loss.
Halep had sailed through the first two rounds on the Madrid clay without dropping a set but she ended up second best in a match which yielded 15 breaks of serve.
Halep served 10 double faults in the match, Mertens nine.
Halep, a two-time Grand Slam champion and former world number one, was twice a break up in the final set at 3-1 and 4-3 but finally succumbed after just over two and a half hours of play.
In the quarter-finals, 13th seed Mertens will meet her doubles partner, Belarusian seventh seed Aryna Sabalenka, who dispatched American Jessica Pegula 6-1, 6-2 in just 52 minutes.
World number one Ashleigh Barty will meet ninth seed Petra Kvitova in the quarters, while eighth seed Belinda Bencic is up against Spaniard Paula Badosa, ranked 62 in the world.
The final last-eight tie will see Anastasia Pavlyuchenkova play Karolina Muchova, who needed four match points to beat Maria Sakkari 6-0, 6-7 (9/11), 7-5.
The Austrian, playing for the first time since a surprise loss to Lloyd Harris in Dubai, broke his 91st-ranked opponent’s serve four times and did not face a break point.