With just seconds to go, it looked like the two sides would have to settle for a share of the spoils before Hugo Lloris' poor clearance found the midfielder in the centre circle.
With Lloris nowhere near his goal, Toivonen hit the ball first-time into an open net to send Sweden top of the group, leapfrogging their opponents.
A stunning first-half goal from Olivier Giroud looked to have been enough to earn France a deserved point after Jimmy Durmaz's equaliser, but Toivonen's brilliant effort meant the hosts took control of the group for the first time.
Didier Deschamps' side held a three-point lead over their opponents going into the tie in Stockholm, having emerged victorious when the two sides met in Paris last November.
They started off on the front foot, with Dimitri Payet looking dangerous on the left flank and Antoine Griezmann causing problems through the middle.
The two combined to set up Moussa Sissoko in the 12th minute, but the Spurs midfielder's shot was deflected and collected comfortably by the goalkeeper.
Payet volleyed over the bar soon after as France hit their stride early, with Sweden's best efforts coming from distance and failing to trouble Lloris.
The game kicked into life in the 37th minute as a deflected cross made its way out to Giroud on the corner of the box.
The Arsenal strike teed the ball up for himself, before firing an unstoppable volley with the outside of his foot into the top corner for his 27th international goal, moving him level with the absent Karim Benzema.
However Les Blues were not ahead for long.
Following some smart play down the left flank, the France defence failed to clear a cross and Durmaz found himself free at the back post, before firing a smart first-time shot past Lloris into the bottom corner two minutes before the break.
France started the second half brightly, with Griezmann having a header well saved by Robin Olsen after some good wing-play by Sissoko.
Paul Pogba curled narrowly past the far post moments later, before the hosts hit back with chances for Jacob Johansson and Durmaz.
Lloris was called into action to deny the latter, before acrobatically tipping over a free-kick from 25 yards.
However the France captain turned villain with just seconds remaining as he attempted to bring the ball out of defence before a horrible miskick found Toivonen, who expertly dispatched into an empty goal to claim a big win.
