Greg Van Avermaet (BMC Racing) came home for second place, with Michal Kwiatkowski (Omega Pharma-QuickStep) in third.
Nibali was part of a much reduced peloton after a tough day of racing that took in nine categorised climbs across Yorkshire.
The final, steep climb of Jenkin Road in Sheffield did not disappoint, serving to split the race apart and force the overall contenders to show their hand very early in the race.
Chris Froome (Sky) led over the top of the hill, but it was pre-stage favourite Peter Sagan (Cannondale) that everyone was watching on the final run-in to the finish.
This allowed Nibali to get away, as everyone else seemed reluctant to give Sagan - the strongest sprinter - a free ride to the line.
Nibali maintained a crucial two-second gap to head up the general classification going into Monday's third stage from Cambridge to London. It's the first time that Nibali has worn the coveted maillot jaune in his career.
Stage one winner and overnight race leader Marcel Kittel (Giant-Shimano) was one of a large number of riders dropped on the succession of climbs in the second half of the stage, and finished around 20 minutes down on Nibali.
Nibali leads Sagan by two seconds overall, with Van Avermaet in third at the same time. Froome finished 19th on the stage to move up to fifth overall, also two seconds behind Nibali.
Sagan takes the lead in the points classification, with Cyril Lemoine (Cofidis) moving into the lead in the king of the mountains classification after being in an early escape group.
Mark Cavendish (Omega Pharma-QuickStep) was the day's only absentee, after being forced to withdraw from the race due to a shoulder injury sustained as a result of a crash on stage one.
Tour de France 2014, stage two: York to Sheffield, 201km
1. Vincenzo Nibali (Ita) Astana in 5-08-36
2. Greg Van Avermaet (Bel) BMC Racing at 2 secs
3. Michal Kwiatkowski (Pol) Omega Pharma-QuickStep
4. Peter Sagan (Svk) Cannondale
5. Tony Gallopin (Fra) Lotto-Belisol
6. Michael Albasini (Swi) Orica-GreenEdge
7. Andrew Talansky (USA) Garmin-Sharp
8. Bauke Mollema (Ned) Belkin
9. Tejay van Garderen (USA) BMC Racing
10. Romain Bardet (Fra) Ag2r at same time
19. Chris Froome (GBr) Sky at same time
23. Geraint Thomas (GBr) Sky at 16 secs
68. Simon Yates (GBr) Orica-GreenEdge at 4-34
Overall classification after stage two
1. Vincenzo Nibali (Ita) Astana in 9-52-43
2. Peter Sagan (Svk) Cannondale at 2 secs
3. Greg Van Avermaet (Bel) BMC Racing at 2 secs
4. Michael Albasini (Swi) Orica-GreenEdge at 2 secs
5. Chris Froome (GBr) Sky at 2 secs
6. Bauke Mollema (Ned) Belkin at 2 secs
7. Jurgen Van Den Broeck (Bel) Lotto-Belisol at 2 secs
8. Alberto Contador (Spa) Tinkoff-Saxo at 2 secs
9. Tejay van Garderen (USA) BMC Racing at 2 secs
10. Jakob Fuglsang (Den) Astana at 2 secs
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1