On a ridge high above the West Bank town of Nablus, a cluster of red roofs and neat white concrete homes mark the Israeli settlement of Elon Moreh, its security patrols on alert for attackers like the lone Palestinian gunman who mowed down four residents during the second intifada.

Elon Moreh was founded in 1979, but the name is mentioned in Genesis as the site of the Patriarch Abraham's first sojourn in the promised land. It is one of 133 "official" Jewish settlements in the occupied West Bank and East Jerusalem that are the focus of intensifying US-led attempts to restart the moribund Middle East peace process.

It has not taken Barack Obama long to discover that in the high-octane politics of the Arab-Israeli conflict, few issues are as explosive as this: Binyamin Netanyahu, the Likud prime minister, is arguing hard over the precise meaning of a settlement "freeze" or the "natural growth" of the existing population. And the president, who singled out settlements for stark criticism in his Cairo speech in June, is being painted as an enemy of Israel.

For Palestinians, the settlements and the 500,000 Israelis who now live in them form an impassable political and physical obstacle to peace – a seemingly permanent feature of a biblical landscape that has been transformed in the 42 years since the 1967 war.

Unless the settlements are removed, they insist, there cannot be a viable Palestinian state in the West Bank, already peppered with hundreds of army checkpoints and divided into disconnected enclaves crisscrossed by settler-only roads, closed military areas – and Israel's controversial separation barrier. Many believe it is already too late.

Obama's grand hopes for a peace agreement based on two independent states living side by side could founder on the rock of settlement expansion.

Even US-backed Arab states such as Saudi Arabia say they cannot, as Obama wants, "normalise" relations with the Jewish state unless settlement activity stops. Egypt and Jordan, which already have peace treaties with Israel, agree.

Israelis themselves are divided. The settlement of Elon Moreh, explains ultra-Orthodox community activist Sarah Gelbard, "is the holiest place on Earth, a place that Hashem [God] himself chose for his holiest people. It can't be tampered with, given away, torn apart or anything else."

Some secular Israelis support settlement, but only for strategic reasons – in the Jordan valley or on the Golan Heights, captured from Syria.

Only Israelis on the far left advocate full withdrawal to the 1967 borders. Most hope for a deal under which big "urban" settlements such as Ma'aleh Adumim and Ariel, near or straddling the pre-war "green line" or inside the "security wall", would be incorporated into Israel in exchange for land in the Negev: that was the plan at Camp David in 2000 before the second intifada erupted. According to one recent poll, 66% of Israeli Jews want to hold on to East Jerusalem. Unlike the West Bank it was annexed to Israel and is now home to 200,000 Jews in suburbs that encircle Arab areas. But under international law it too is considered occupied territory.

Even Palestinians who still believe in a two-state solution see the Israeli settlements as the continuation of a century of colonisation, dispossession and disaster. The only certainty has been that whether their lands were purchased, conquered or expropriated, they have nearly always been lost for ever.

Settlements have often been billed as "an appropriate Zionist response" to pressure for concessions. In the 1970s and 1980s the Gush Emunim (Bloc of the Faithful) movement proved adept at manipulating first Labour and later Likud governments which feared confronting the religious nationalists – who were seen as zealous new "pioneers".

That remained true after 1993, when Yasser Arafat signed the Oslo agreement with Yitzhak Rabin but without securing a settlement freeze. The number of settlements and settlers doubled in the decade that followed. In 1998 Ariel Sharon, then in charge of US-brokered talks with Arafat, urged settlers "to move, run and grab as many hilltops as they can … because everything we take now will stay ours. Everything we don't grab will go to them." In 2004, as prime minister, he persuaded George Bush to endorse the same "new realities on the ground" that Obama is now challenging.

No wonder, says a senior western diplomat in Tel Aviv, that "Palestinians above all point to settlements as proof that even when the Israelis are talking peace, annexation is the real agenda".