Southampton, who went down to 10-men in the 30th minute after Yan Valery was shown his second yellow, bounced back after falling behind 2-1 in extra time by levelling 63 second later through Mohammed Elyounoussi.
Long turned from provider to goalscorer with James Ward-Prowse picking out the striker for an easy tap in goal from the middle of the six-yard box in the 102nd minute of action.
Southampton took the lead after just eight minutes when Nathan Redmond was the benefactor of a poor back pass that Nathan Tella read precisely before threading a pass through, which the former calmly dispatched beyond the keeper.
Valery — having already been booked inside the opening three minutes — was then sent off for a second yellowafter bringing down former Saints teammate Michael Obafemi, who was looking to break on goal.
Jack Stephens failed to double the Southampton advantage in the 61st minute when he missed a golden opportunity from a free header at the back post. The goal would’ve given the visitors breathing room in what had been a fairly benign Cup tie.
Swansea found their equaliser 14 minutes from full-time with Joel Piroe pouncing on a loose ball in the box following a build up move that saw Korey Smith link up with Kyle Naughton.
Fraser Forster then denied a late chance for Swansea to break the deadlock with a strong foot after Piroe was sprung one-on-one with the Saints goalkeeper.
The Cup tie came to life when Saints fell behind 2-1 from Jan Bednarek’s unfortunate own goal, which was the product of a Olivier Ntcham cross.
But Elyounoussi had an instantly reply for Saints, firing in the equaliser to make it 2-2 after taking one touch from Long’s cross before putting his right-foot through the ball.
Long, who last scored for Saints in a 2-0 win over Aston Villa on February 22, 2022, then made it 3-2 following a great combination play with James Ward-Prowse with the striker left free in the box.
Swansea City (4-3-3): Hamer; Manning, Cooper (Ntcham 62), Bennett, Naughton; Walsh (Grimes 61), Downes, Smith (Williams 114); Cullen (Dhanda 85), Obafemi (Fulton 91), Piroe
Unused subs: Benda (GK), Bidwell, Williams, Patrick Williams, Abdulai
Goals: Piroe (77), Bednarek (95 OG)
Bookings: Bennett (43), Cooper (45+1), Walsh (59), Dhanda (90+3)
Southampton (4-4-2): Forster; Perraud, Bednarek, Stephens, Valery; Tella (Elyounoussi 77), Romeu, Ward-Prowse, S. Armstrong (Diallo 65); Broja (Long 78), Redmond (A. Armstrong 65)
Unused subs: Caballero (GK), Olaigbe, Lyanco, Simeu, Chauke
Goals: Redmond (8), Elyounoussi (96), Long (102)
Bookings: Valery (3, 30), Stephens (90+3)