Lithuania rode their luck in Tbilisi but ran out winners thanks to second-half goals from Andrius Ksanavicius and Mindaugas Kalonas.
Ksanavicius put the visitors ahead after 52 minutes at the Boris Paichadze Stadium.
But they could not seal the points until Kalonas struck deep in stoppage-time to give the Lithuanians their fifth and final win of the campaign.
Dato Kvirkvelia had the first effort on goal for the home side after four minutes with a 30-yard shot which Lithuania goalkeeper Zydrunas Karcemarskas parried away.
From the resulting corner Karcemarskas was called into action again to save a David Siradze header.
Levan Kenia could have put the hosts ahead in the eighth minute but his close-range header was blocked by defender Marius Stankevicius.
Edgaras Jankauskas had the visitors' first effort on 26 minutes but his 30-yard effort was off target.
Georgia hit the post when Zurab Menteshashvili's inswinging free-kick from the edge of the box narrowly evaded Kakha Kaladze and Lasha Salukvadze.
Two minutes later Kenia played in Siradze whose effort from the left of the penalty area was well saved by Karcemarskus with his legs.
Karcemarskus was in action again in the 37th minute to save Kenia's effort from the edge of the area.
Vidas Alunderis could have put Lithuania ahead but his header from Mantas Savenas' corner was saved by Georgia keeper George Makaridze.
In first-half stoppage time Alunderis found Jankauskas but he headed wide when well-placed in the penalty area.
Two minutes after the interval Levan Mchedlidze had his effort blocked by Tomas Zvirgzdauskas. The same player then saw his header saved by Karcemarskus on 48 minutes.
Lithuania went in front a minute later when Ksanavicius' effort from the edge of the box beat Makaridze and ended up in the bottom-right corner of the net.
Stankevicius could have doubled the visitors' advantage in the 73rd minute but he volleyed Kalonas' corner over the bar.
Mchedlidze hit a brilliant left-footed overhead kick against the inside of the post on 88 minutes as the home side went in search of an equaliser.
But Velicka broke free down the right in the sixth minute of stoppage time and squared for Kalonas who finished low into the corner of the net to put the result beyond doubt.
Georgia (0) 0 Lithuania (0) 2
Georgia Makaridze, Salukvadze, Asatiani, Kaladze, Kvirkvelia, Kankava, Tskitishvili, Menteshashvili (Martsvaladze 33), Kenia, McHedlidze, Siradze (Gelashvili 80).
Subs Not Used: Kvaskhvadze, Shashiashvili, Kvakhadze,
Dekanosidze, Sanaia.
Lithuania Karcemarskas, Stankevicius, Zvirgzdauskas, Skerla, Paulauskas, Dedura, Alunderis, Ksanavicius, Savenas (Kalonas 67), Danilevicius (Morinas 76), Jankauskas (Velicka 56).
Subs Not Used: Grybauskas, Labukas, Klimavacius, Papeckys.
Booked: Jankauskas, Karcemarskas, Kalonas.
Goals: Ksanavicius 52, Kalonas 90.
Ref: Aleksandar Stavrev (FYR Macedonia).