The 18-year-old won a gold medal at the World Athletics Championships in Berlin on Wednesday.
She was virtually unknown until last month, when she set a new record for the 800m in the African Junior Championships.
That victory also meant she had improved her personal best by seven seconds in less than nine months - and she began to attract huge attention.
While the athlete, who comes from a rural village in Limpopo province, has shrugged off speculation over her gender, others have taken the matter to heart.
The South African government, political parties, unions and ordinary citizens have expressed outrage at the tests ordered by the International Association of Athletics Federation (IAAF).
A number of "support" groups have been created by well-wishers on social networking site Facebook since news broke of the IAAF's plans.
One group, called In Support of Caster Semenya and All African Women, had more than 1,000 members by Thursday afternoon.
Its creator, Thabiso Theffo, said he made the group "to show support for the running sensation who has received unfair treatment".
Regional and national radio stations also rallied behind Ms Semenya and held debates throughout the day calling for people's views.
"I think what is happening is stupid," said Hadija Haruna, a freelance journalist.
"Why didn't they didn't check her before she won last month?"
Johannesburg residents were keen to register their annoyance.
Oscar Ramphele compared the Semenya affair with the case of disabled South African athlete Oscar Pistorius, who was initially banned from the 2008 Olympics after officials ruled his prosthetic legs gave him an unfair advantage.
"They should have spoken to her first before talking to the media. Pistorius went through more or less similar humiliation," Mr Ramphele said.
University student Ntombifuthi said the matter should have been handled in private.
"They shouldn't be judging her beforehand, it's not fair," she said.
'Bogus and groundless'
And the official response has been no less strident.
The women's ministry said: "Semenya's outstanding success despite negative media speculation about her gender demonstrates her determination and resolve to succeed."
The ruling African National Congress urged the country to back "our golden girl".
Youth bodies have also strongly defended Ms Semenya.
The ANC youth wing said it "condemns with contempt those who are questioning her gender".
"Caster is one of South Africa's diligent female athletes," the party said in a statement.
The Young Communist League of South Africa said the IAAF's actions were racially motivated.
"This smacks of racism of the highest order," the party said.
"It represents a mentality of conforming feminine outlook within the white race, and that as long as it does not fall within this race or starve and paint itself in order to look like the white race it therefore is not feminine."
The Congress of South African Trade Unions said queries raised about Ms Semenya's gender were "bogus and groundless".