It's April 1988, about five in the morning, 40km outside Kabul in Afghanistan. I'm taking shelter in a scrape in the rock, flattening my cheek against the cold surface, semi-automatic gunfire and the concussion of departing mortars beating in my ears. In theory, I'm making a documentary about young Soviet army conscripts in Afghanistan. In reality, I've been marooned on this "zastava", or mountain outpost, for days. The 17-year-old kids, who are the heroes of our documentary, fire back at the attacking mujahideen, in the grip of a kind of hyper-bravado. I, on the other hand, have leapt from my makeshift sleeping bag to cower in what passes for cover on this bare outcrop. "Why am I here?" I ask myself pointlessly, and not for the first time. "Aren't there safer assignments I could pursue, where nights are spent between soft sheets? Why am I obsessed with war?"

A quarter of a lifetime later, I'm still exploring that obsession, trying to bring to the screen what is, without doubt, the most ambitious, agonising and creatively troublesome film I've ever undertaken. The Promise, which screens on Channel 4 from 6 February for four weeks, attempts in drama to come to an understanding of the most dangerous and intractable war of our age – the conflict between Arab and Jew in the Middle East – as seen through the eyes of two outsiders, a British teenager and her grandfather. Erin Matthews, an 18-year-old just beginning her gap year, travels to Israel with her Jewish schoolfriend, Eliza. Eliza, who has dual nationality, has been summoned back to Israel for military service. Erin goes with her for moral support, taking a diary written 60 years before by her grandfather, Len. Fresh from the second world war and the airborne assault on Germany, Sergeant Len Matthews has been unexpectedly posted – like 100,000 other British troops – to keep the peace in what was then called Palestine. As Erin reads his diary, we travel back in time to witness, with Len, the war at the birth of the state of Israel. And as Erin reads, she becomes curious about the disputed country beyond the comfort of Eliza's seaside home. She starts to retrace her grandfather's steps, beginning a journey through modern-day Israel and the occupied territories that will see her solve the mystery of why Len's life was destroyed by the few months he spent in that troubled land.

War attracts nothing so much as cliche. Perhaps the greatest is that the first casualty of war is truth. For example, to my knowledge there are at least three convincing and apparently well-documented explanations of the killings that took place in the Arab village of Deir Yassin, one of the emblematic events of the bloody war of 1948. If we were to tiptoe into the minefield that is Middle-East politics, we had better get our facts right. For four years, a team of six researchers picked away at the story of Len and Erin in our two time frames, 1945-48 and today. We tracked down and interviewed over 80 veterans of the British Mandate in Palestine (Britain was the colonial power until 1948), studied archives from the period at the Imperial War Museum, the Airborne Forces Museum at Duxford and at the public record office in Kew, where thousands of declassified intelligence reports from the period can still be found and read. We unearthed unpublished photographs and accounts of the perilous journey undertaken by Palestinian Arabs in 1948, fleeing their homes in the face of the advancing Jewish forces. We spoke to Israeli academics who had interviewed Jewish women used to befriend British soldiers to covertly extract intelligence from them. And we spoke to their controllers, the underground fighters of the Irgun Tsvai Leumi, who fought to a standstill a proud British army fresh from victory in a world war.

For the present-day story we interviewed Israeli Jewish boys and girls, conscripted at 18 in defence of their country. We tracked down children of the same age from overseas, members of the International Solidarity Movement, who had confronted Israeli bulldozers to protect the homes of Palestinians in the occupied territories. We drew on testimony from Combatants for Peace, Breaking the Silence and other organisations concerned with the uneasy and undeclared truce in Israel today. On my own research trips to the region I located and visited the site of the massacre at Deir Yassin, finding the former Arab village still intact but, incredibly, now being used as a high-security hospital for mentally ill patients. I stood in the death cell where Jewish fighters condemned by the British Mandate government for insurrection awaited their fate, visited the sites of recent suicide bombings and gazed out across Israel's protective wall, surely the most palpable and chilling symbol of division on our planet.

Our research turned up some surprising facts, counter to common knowledge. For example, for many years I had believed that the Israeli military had invented the strategy of destructive reprisals against the families of insurgents. If a Palestinian blows him or herself up in an Israeli city, the Israeli Defence Force will locate the family home of that bomber and bulldoze it. How strange then to discover, as we pored over records of tactics in Mandate Palestine, that the British used exactly the same techniques against the Irgun, part-precursors of the present-day Israeli military, in 1946. If British interests were attacked by a Jewish "terrorist", the home of that terrorist would be dynamited, as a matter of policy. Why would the Jews, who demonstrably defeated the British and their entire tactical handbook, adopt exactly the same failed anti-insurgency approach as their former masters when they in turn faced an insurgency? It made no sense but, as we were to discover, nothing is simple in a land where truth has long since been co-opted as a weapon of war.

Making the drama in Israel itself also turned out to be anything but simple. At the outset, it had seemed a wise decision. Nowhere else looks quite like modern-day Israel – the topography, the architecture, the physiognomy of its diverse population. Creating Erin's story elsewhere in the Arab world would be time-consuming and costly. And where better to stage scenes set in 1940s Palestine than in the locations where the events had taken place, where some key buildings survive and others could be readily recreated from local archive and memory. English is widely spoken, period weapons and vehicles abound, there's a thriving film industry. It ought to have been straightforward. In practice, it was anything but. When I dramatised events from the Bosnian war for Leigh Jackson's Warriors, I faked them in the Czech Republic. Scenes for my drama about Somalia and Liberia were recreated in Kenya and Ghana. I did Iraq in Morocco, Pakistan in India, even Belfast was carefully remounted in the streets of Leeds and Bradford. Never before had I attempted to dramatise a conflict in the land in which it was taking place, using ex-combatants and reservists as actors and extras, local technicians as crew, shooting events still raw in the memory in the places in which they had occurred. Scenes that look achievable on paper take on a lively extra dimension when you have real Israelis and Palestinians playing your roles.

One particularly difficult scene calls for an actor playing an IDF commander to use a Palestinian civilian as a human shield while moving through a dangerous area in Gaza. We had detailed research supporting the event we were depicting and, by chance, an Israeli soldier had been found guilty in the courts for using exactly these tactics in the week we were to shoot the scene. None of these justifications made the sequence any easier to achieve in the cockpit of unresolved animosities that is Israel today. The first actor I cast walked out during rehearsals, explaining politely that, although he knew these things happened, recreating such an event in a scene with Palestinian actors wasn't something he was able to do. I recast the part, outlining in over-elaborate detail to the talented substitute actor we chose what the scene would involve. When he agreed, I privately assumed he was a committed liberal, out of sympathy with Israeli military policy. But when it came to staging the scene, in the predominantly Arab town of Ramle with Palestinian actors playing opposite him, it became clear that he had recent military experience in the occupied territories. Eventually, he revealed that he was an officer in the Israeli army reserves, spending a weekend a month in uniform. When I asked why, if that was true, he had been prepared to accept the role he said: "These things happen. We need to confront them." And confront them he did, in one of the most distressing and powerful scenes in the film.

In episode three of The Promise, Erin travels to Hebron in the occupied West Bank. We used her visit as an opportunity to restage a scene from our research, where a Jewish settler faces off angrily against an Arab resident. The actors involved wanted to be photographed together at the end of what was an unremittingly aggressive confrontation. "The image you'll never see in The Promise," said the Jewish actor as she posed arm-in-arm with her Arab fellow actor. Later she told me that, in a long career on stage and screen in Israel, this was the first time she had ever acted with a "real Palestinian". It had taken the arrival of a foreign film crew, not realising the magnitude of what it was they were asking, to bring this thing about.

So what have they taught me, my seven years engaged with the inciting conflict of our terrorist-obsessed age? The most striking thing I'm left with is a question: how did we get from there to here? Like most British soldiers we interviewed, arriving in Palestine from the war in Europe, Len Matthews felt only sympathy for the Jewish plight. Having seen the ovens of Bergen-Belsen, his heart tells him that Jews deserve a place of safety, almost at any price. In 1945, that view was shared by most of the world. In the era inhabited by Erin, his granddaughter, just 60 years later, Israel is isolated, loathed and feared in equal measure by its neighbours, finding little sympathy outside America for its uncompromising view of how to defend its borders and secure its future. How did Israel squander the compassion of the world within a lifetime? That's the question The Promise sets out to explore. Its other purpose is to act as a reminder to all of us Brits who shake our heads and mutter "not our problem". As the departing colonial power, Britain was charged with seeing both communities to independence in good order. In Palestine, as in so many other examples of our rapid retreat from empire, we left chaos, political confusion, bloodshed and war. It turns out that it is our problem, at least in part, and we should take some responsibility for it.