Cavendish was brought down after colliding with world champion Peter Sagan in the final 250m of the stage and suffered a deep cut to his hand as well as the shoulder injury.
Sagan, who appeared to elbow the British sprinter as he passed him on the right hand side next to the barriers, was later disqualified for his part in the incident.
"Mark suffered a fracture to the right scapula," Dimension Data team doctor Adrian Rotunno said in a team statement on Thursday night.
"Fortunately, no surgery is required at this stage, and most importantly, there is no nerve damage. He's been withdrawn from the race for obvious reasons, and we'll continue monitoring him over the coming days."
The Manxman's Dimension Data team had cast immediate doubt on his continued participation as he went off to hospital, straight after the stage, with team principal Doug Ryder saying "it didn't look good."
But now x-rays have confirmed that the 32-year-old will have to miss the rest of the Tour, which will come as a major blow after he worked his way back from glandular fever, which kept him out of racing for months this season, to be fit for the Tour.
"I'm obviously massively disappointed to get this news about the fracture," Cavendish said.
"The team was incredible today. They executed to perfection what we wanted to do this morning.
"I feel I was in a good position to win, and to lose that and even having to leave the Tour, the race I've built my career around, is really sad.
"I wish the best of luck to me teammates for thereat of the race. Now I'm looking froward to watching the race on TV, seeing my team fly the flag for South Africa and raise awareness for Qhubeka."
With Cavendish now withdrawn from the Tour, and Sagan disqualified, two of the highest profile sprinters have now left the race with just for days gone.
Many of their rivals and former sprinters of the pro peloton had sympathy for Sagan's disqualification, while Cavendish himself said he was "not angry, just confused" about the incident and the elbow from Sagan.
His team boss Ryder, was more damning about Sagan's move at the end of the stage however.
“It was ridiculous, that wasn’t racing as it should be,” Ryder said.
“At that speed, there are instincts that happen. I don’t think anybody does anything deliberately. It happened, and I don’t think it’s right.
“To be disqualified … that affects him and his race, but it doesn’t help us in terms of Mark and his career. He’s a legend in the sport and that’s really disappointing.
“You can see all the guys moving over and Peter’s elbow moved off the bar after. Sprinter are sprinters but the deviation is pretty drastic around there.”
The Tour de France continues on Wednesday's stage five without Cavendish and Sagan towards it's first summit finish, on a 160.5km route to the Le Planche de Belles Filles.
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1