Thomas added to his race lead on the Col du Portet by dropping rivals on Wednesday, but while he sprinted for third place bonuses seconds, a fan leaned over to grab or hit his right arm.
"At the time I didn't know what it was, I thought it might have been an accident," Thomas said.
"We are coming to race our bikes, that's all we want to do. The whole peloton wants to do it safely. That's a bit too much.
"Just stay at home, don't come out and try to affect the race like that. I could've quite have easy fallen and lost time, or certainly not gained the time I did. Just do what you normally do [at home]."
Thomas heard that the man seen in the photos tried to do the same thing with Nairo Quintana (Movistar) when he was riding to his stage win.
"I think it was too much to drink and a bit of an idiot," added Thomas.
"I definitely felt it. It didn't feel like a grab, but like a hit. I was going quite fast past him and it certainly moved me off my line and it was a bit of a shock. Luckily it was OK."
Race organisers ASO, along with the police, reportedly are trying to identify the man, but they have yet to update the situation.
The incident follows several others over the years and in 2018 for Team Sky. Some have punched, some have thrown urine; one hit four-time winner Chris Froome on the Alpe d'Huez climb last week. Another was spotted spitting towards him climbing the Col du Portet.
"Just try to think about what you are doing," Team Sky sports director Nicolas Portal said. "Would you do that in the streets in a city? No. Don't do it at a sporting event where kids are watching. It's not nice.
"If he crashed, broke his collarbone, it would have been done and finished just because someone doesn't like this guy.
"It's a big problem, I'm not try to lesson it. It's been three weeks with things happening. It's like we are almost used to it, and that's bad."
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1