Contador had been suffering from a fever overnight and started the tough mountain stage with uncertain form, particularly as he was also still nursing injuries from crashes earlier in the race.
After trying to get into the day's early escape group on first climb over the Port de la Bonaigua, Contador dropped back to the bunch and then repeatedly visited the team car before abandoning.
The Spaniard won the Tour in 2007 and 2009 and had been one of the front runners for this year's race. Contador fell on his right side in day one in a crash with Luke Rowe (Sky) and Brent Bookwalter (BMC Racing).
Contador crashed again on day two, that time on his left side. At the start of stage nine, he sat in 20th place overall at 3-12 minutes behind race leader Chris Froome (Sky).
Yesterday, Contador lost 1-41 minutes on Froome. "My body isn't responding as I would have liked," he said. "These crashes are catching up with me.
"It's terrible to have crashed like this twice in the first week and to have lost so much time after working so hard to be here. It's cycling, danger is everywhere. Believe me, I did the best I can."
The team confirmed that Roman Kreuziger will lead the team in the classification battle for the rest of the race.
"Contador had a fever this morning and that was when we knew it would be over," Sports Director Sean Yates said. "We will now work for Roman Kreuziger in the overall and Sagan for the green jersey."
Contador will need to re-build now to race with Spain in the Rio de Janeiro Olympics. He is also due to race the Vuelta a España.
Next year, he will ride with American WorldTour team Trek-Segafredo sources told Cycling Weekly.
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1