Police used tear gas and stun grenades to disperse 150 Palestinian protesters who had thrown rocks at non-Muslims who entered the al-Aqsa mosque compound.
The site, known to Jews as the Temple Mount, is sacred to both religions.
Israeli police said the visitors were foreign tourists, but Palestinians said they were Jewish extremists.
"At a time when (the US administration) is trying to bridge the divide... Israel is deliberately escalating tensions in Jerusalem," said Palestinian peace negotiator Saeb Erekat.
"We've seen this before, and we know what the consequences are," he added.
In the past the al-Aqsa/Temple Mount compound has been a flashpoint for Israel-Palestinian violence, notably after the visit of then Israeli opposition leader Ariel Sharon in 2000.
Protests were quelled by Sunday afternoon, with dozens of police patrolling the streets of the walled Old City and blocking some of its gates.
Different versions
There are conflicting accounts as to who was the initial target of Muslim anger.
Palestinian sources said about 15 people from the Temple Mount Guardians group managed to enter the compound and performed acts of worship in contravention of agreements putting the compound under Muslim control.
At first, Israeli police confirmed this, but later issued a clarification saying the group was in fact made up of non-Jewish French tourists.
Police said Palestinian worshippers had started protesting at immodest clothing worn by the visitors.
Other accounts say the tourists were mistaken for members of a large group of religious and right-wing Jews which had gathered at one gate of the compound to press for entry.
Mr Erekat accused the Israeli authorities of escorting hardline Jewish settlers from the Israeli-occupied West Bank, "whose presence is deliberately designed to provoke a reaction", around the mosque.
At least 10 Palestinians were injured in the clashes and several Israeli police were lightly hurt.
The Arab League expressed "extreme anger... at the premeditated aggression" at the Israeli security forces for allowing "Zionist extremists" into al-Aqsa.
The incident happened hours before the start of Yom Kippur, the holiest day in the Jewish religious calendar, and Israeli security officials were unavailable for further comment.
Front Page Features