Police used tear gas and stun grenades to disperse 150 Palestinian protesters who had thrown rocks at non-Muslims who entered the al-Aqsa mosque compound.
The site, known to Jews as the Temple Mount, is sacred to both religions.
Israeli police said the visitors were foreign tourists, but Palestinians said they were Jewish extremists.
"At a time when (the US administration) is trying to bridge the divide... Israel is deliberately escalating tensions in Jerusalem," said Palestinian peace negotiator Saeb Erekat.
"We've seen this before, and we know what the consequences are," he added.
In the past the al-Aqsa/Temple Mount compound has been a flashpoint for Israel-Palestinian violence, notably after the visit of then Israeli opposition leader Ariel Sharon in 2000.
Protests were quelled by Sunday afternoon, with dozens of police patrolling the streets of the walled Old City and blocking some of its gates.
Different versions
There are conflicting accounts as to who was the initial target of Muslim anger.
Palestinian sources said about 15 people from the Temple Mount Guardians group managed to enter the compound and performed acts of worship in contravention of agreements putting the compound under Muslim control.
At first, Israeli police confirmed this, but later issued a clarification saying the group was in fact made up of non-Jewish French tourists.
Police said Palestinian worshippers had started protesting at immodest clothing worn by the visitors.
Other accounts say the tourists were mistaken for members of a large group of religious and right-wing Jews which had gathered at one gate of the compound to press for entry.
Mr Erekat accused the Israeli authorities of escorting hardline Jewish settlers from the Israeli-occupied West Bank, "whose presence is deliberately designed to provoke a reaction", around the mosque.
At least 10 Palestinians were injured in the clashes and several Israeli police were lightly hurt.
The Arab League expressed "extreme anger... at the premeditated aggression" at the Israeli security forces for allowing "Zionist extremists" into al-Aqsa.
The incident happened hours before the start of Yom Kippur, the holiest day in the Jewish religious calendar, and Israeli security officials were unavailable for further comment.
What is to be done between now and 2SS? | September 17, 2017 |
The settlers will rise in power in Israel's new government | March 14, 2013 |
Israeli Apartheid | March 14, 2013 |
Israel forces launch arrest raids across West Bank | March 14, 2013 |
This Court Case Was My Only Hope | March 14, 2013 |
Netanyahu Prepares to Accept New Coalition | March 14, 2013 |
Obama may scrap visit to Ramallah | March 14, 2013 |
Obama’s Middle East trip: Lessons from Bill Clinton | March 14, 2013 |
Settlers steal IDF tent erected to prevent Palestinian encampment | March 14, 2013 |
Intifada far off | March 14, 2013 |