An horrendous defensive blunder from Leicester centre-half Patrik Gerrbrand gifted Southampton's Kenwyne Jones the only goal of the match and the Saints a fourth-round FA Cup victory at the Walkers Stadium.
But it was the visitors who had been under pressure early on as Stephen Hughes chanced his arm in the sixth minute, only for his left-foot shot to go wide.
Moments later, after Richard Stearman was cautioned by referee Nigel Miller following a foul on Marian Pahars, the Latvian headed Matt Oakley's free-kick wide at the other end.
Next, Leicester defender Paddy McCarthy went close with two headers which sailed just over.
Then, with 10 minutes to the break, Southampton's David Prutton was forced to limp out of the action after a block-tackle on Paddy Murphy left the former clutching his ankle.
After treatment Prutton tried to rejoin the game but soon went down again and was replaced by Simon Gillett, making his Saints debut.
Darren Potter, the Liverpool midfielder signed on loan until the end of the season, and Jim Brennan, the left-back who was a free-transfer acquisition from Norwich, were also making their inaugural appearances for the south coast club.
Into the second half and both sides ran out unchanged with Saints first on the offensive, this time through Nathan Dyer, who made tracks along the right flank before crossing deep to the back post.
But although Pahars was quick to control the ball and then make space for a shot, his effort on the turn was blasted high over City goalkeeper Rab Douglas's crossbar.
At the other end, Elvis Hammond broke into the Saints penalty area and looked odds-on to score as he bore down on exposed Bartosz Biakowski, only for the goalkeeper to pull of a fine save low down to his left.
With the hour mark still not passed the hosts threatened again as Alan Maybury found himself free inside the Southampton penalty area.
But with team-mate Iain Hume unmarked and screaming for the ball to be played to him on the penalty spot, the Irishman opted to go for glory himself and blazed a right-foot shot wide and over from an acute angle.
Hume was next to race clear after beating Biakowski to a high-bouncing ball but the allowed the keeper to regain his line and save the eventual shot.
Kelly introduced Mark De Vries as a 67th-minute replacement for Hammond but it was the visitors who were next on the front foot as Dyer raced clear along the right channel and centred for Dexter Blackstock, only for the sliding striker to force the ball wide.
But then, in the first minute added time, Gerrbrand failed to deal with a high-bouncing ball as substitute Jones - on in the 84th minute for Dexter Blackstock - seized his chance to lash the ball low past Douglas.
Teams:
Leicester Douglas, McCarthy, Gerrbrand, Johansson, Stearman,Hughes, Gudjonsson, Smith, Maybury, Hume, Hammond (de Vries 67).
Subs Not Used: O'Grady, Henderson, Wesolowski, Kisnorbo.
Booked: Stearman.
Southampton Bialkowski, Baird, Lundekvam, Potter, Brennan,Dyer, Oakley, Higginbotham, Prutton (Gillett 36), Pahars,Blackstock (Jones 83).
Subs Not Used: Smith, Fuller, Cranie.
Booked: Blackstock.
Goals: Jones 90.
Att: 20,427
Ref: N Miller (Durham).