Martin (Israel Start-Up Nation) lost over six minutes to the pink jersey of Egan Bernal (Ineos Grenadiers) and slipped out of the top 10 in the general classification on the road to Montalcino.
The Irishman spoke to GCN after stage 11: "I told my wife this morning that I won't crash, so for me personally cycling is not worth the risk.
"I had guys crashing all around me on the first section [of gravel] so I just did my own pace, I nearly came back but my licence is road cycling so it's not my thing. Fair play to the guys who are at the front, but I just didn't want to take the risk today and that's it.
"The cycling fan in me thinks this is a beautiful stage but yeah it was on an unpredictable road surface and I don't have much experience on that particular terrain. No excuses just didn't feel like taking the risks."
Martin now sits 18th overall after dropping 10 places in the overall standings, leaving him 7-06 down on Bernal in pink.
Martin said: "Personally, I was a bit too relaxed. Everyone was battling for positions; I got a few pushes and lost my head for a while. My team-mates did amazing by bringing me back to the second peloton, from there we nearly bridged the gap to the front."
Stage 12 heads back to the Apennines where the race takes on another tough hill stage that takes on four categorised climbs over the 212km stage from Siena to Bagno di Romagna and plenty more days of racing to gain back some time.
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1