The Australian had been caught up in a collision with a motorcycle which stopped in front of him, along with Bauke Mollema (Trek-Segafredo) and race leader Chris Froome (Team Sky).
While Porte fared better than his former teammate Froome, who was forced to run up the climb for a short period after his bike was broken in the incident, the BMC rider still ended up losing time in the provisional standings to riders who had been behind him on the course, and lambasted the crowd control shortly after the finish.
"The crowd was all over the road and the motorbike just stopped right in front of us and we had no where to go but straight over the top of the motorbike,” Porte said.
“It was just a mess. Froome was on my wheel and was straight into me.
“I don't know what they're going to do but they need to do something about it. It's not fair. One minute we're 23 seconds in front and the next thing for something so silly, everyone's back on us.
“If you can't control the crowds what can you control?
"It's not really the motorbikes, it's the crowds in your face the whole time, pushing riders. At the top there it was just crazy."
Fortunately for Porte, the race jury decided to award him and Froome the same time as former breakaway companion Mollema on the line, 5-05 behind stage winner Thomas De Gendt.
Porte had provisionally finished 5-24 back and Froome at 6-45, which would have seen Adam Yates (Orica-BikeExchange) move into the overall lead should it have stood.
Former Sky rider Porte now sits in 11th place in GC, 2-22 of the overall lead, having lost much of that time with an unfortunate puncture on stage two of the race.
Despite the race jury awarding him the same time for the stage, Porte still keen to get across the difficulties he felt the fans put the riders in on the climb to Chalet Reynard, 6km from the summit of Ventoux.
"It's the decision they [the race jury] had to take. It's already out of control. I agree that you come to the race, you have a good time but you don't need to be running beside the riders, you don't need to be hitting riders, pushing riders.
“Things have got to change and I can't believe there weren't barriers there. At the end of the day I've trained so hard for this and yeah okay now I get the same time as Mollema, but I also crashed and now I'm sore.
“Tomorrow's a crucial [time trial] stage as well and it remains to be seen how I'll pull up.
“We love the fans and 99 percent of them are brilliant but why do some of them need to take their selfies and run along beside us?
“There's passion and there's stupidity and it's not such a fine line between them."
Thank you for reading 20 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1