The jury disqualified Peter Sagan after he drifted into Manx cyclist in the final 150 metres of the stage to Vittel. Cavendish with his right arm in a sling had to go to the hospital.
"His shoulder does not look good so we'll have to wait and see, we can only hope," team principal Doug Ryder said when asked if he thought Cavendish could continue.
"He's a tough guy but that was a hard crash. We can only hope but I'm not sure."
Cavendish left the bus and was forced to take his anti-doping test first before his x-rays.
"It doesn't look good, we have to wait for the x-rays, but let's hope that it's nothing that severe, but it didn't look that great from my perspective. But I'm not a doctor."
The jury decided later after Ryder spoke that Sagan should go home. Sagan won stage three on Monday and won the green points jersey in the last five editions of the Tour de France.
The incident happened with Cavendish, the Tour de France's most successful sprinter. Cavendish counts 30 stage wins and is nearly the record held by Eddy Merckx at 34. He fought to recover from glandular fever just to be on the start line for the 2017 edition.
"It was ridiculous, that wasn't racing as it should be," Ryder said.
"At that speed, there are instincts that happen. I don't think anybody does anything deliberately. It happened, and I don't think it's right.
"To be disqualified ... that affects him and his race, but it doesn't help us in terms of Mark and his career. He's a legend in the sport and that's really disappointing.
"You can see all the guys moving over and Peter's elbow moved off the bar after. Sprinter are sprinters but the deviation is pretty drastic around there."
Thank you for reading 20 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1