The battle happened in Nuristan province in the remote east of the country when military outposts were attacked, a Nato statement said.
The Taliban said it carried out the attack. Reports say local officials including a police chief were captured.
Violence has escalated in the east as insurgents relocate from the south.
My heart goes out to the families of those we have lost and to their fellow soldiers who remained to finish the fight... Both the US and Afghan soldiers fought bravely together
Col Randy George
US area commander
In a statement, Nato's International Security Assistance Force (Isaf) said that tribal militia launched attacks on the foreign and Afghan military outposts from a mosque and a nearby village.
The attack is thought to have taken place in the Kamdesh district of Nuristan, and lasted several hours.
About 300 militants attacked one outpost at the foot of a hill, before turning their fire on a US base on higher ground, attacking from two sides, a provincial police chief said.
ANALYSIS
Martin Patience, BBC News, Kabul
Prior to this deadly clash, US soldiers at the outposts that saw fierce fighting were due to be redeployed elsewhere in the country in the coming weeks.
According to officials, even after Saturday's assault, that remains the case.
Part of the reason is that the US Gen Stanley Mc Chrystal - who heads the mission in Afghanistan - wants to focus on providing security in key towns and villages across the country.
The hope is that the Afghan government supported by the West will then be able to deliver services to the people.
That means essentially "closing" a string of smaller bases.
As this latest attack showed, these small bases are particularly vulnerable when swarmed by huge numbers of insurgents.
One Nato spokesman called it a "complex attack in a difficult area".
US jets carried out air strikes in response.
"Coalition forces effectively repelled the attack and inflicted heavy enemy casualties while eight Isaf and two ANSF [Afghan National Security Forces] members were killed," the Nato statement said.
It was the worst loss coalition troops have suffered since August 2008, when 10 French troops were killed in an ambush in Kabul province.
A Taliban spokesman, Zabihullah Mujahid, said the movement was behind the attack.
According to AP news agency, Mr Mujahid also said some 35 Afghan police officers had been taken into Taliban custody, and their fate would be decided by a council.
Provincial governor Jamaluddin Badar confirmed that some officials including a local police chief had been captured.
Dangerous province
WORST SINGLE ATTACKS ON FOREIGN FORCES
Sept 2009: Six Italian soldiers die in suicide bomb in Kabul
Aug 2008: Ten French troops killed in ambush in Sarobi, east of Kabul
July 2008: Nine US soldiers die in militant siege in Wanar, bordering Nuristan and Wanar provinces
Nov 2007: Six US soldiers and three Afghan troops killed in ambush
July 2007: Six Canadian soldiers and Afghan interpreter die when vehicle hits IED in Kandahar province
May 2007: Five US, one UK, one Canadian soldier die in hostile attack on helicopter in Helmand province
June 2005: Sixteen US soldiers die in attack on helicopter in Konar province
It is not the first time coalition forces have suffered damaging attacks in this region, says the BBC's Martin Patience in Kabul.
Nine US soldiers were killed in a single incident last year when more than 100 fighters breached a US outpost in the village of Wanat on the border of Nuristan and Kunar provinces.
The incident, which is still being investigated, was the biggest American loss of life in battle in Afghanistan since operations began in 2001, and forced US and Afghan soldiers to abandon the village.
The province's mountainous terrain makes it easier for insurgents to sneak up and launch attacks, but more difficult for military forces to access the area, our correspondent says.
Nuristan has for decades been a crossing point for fighters entering the country from Pakistan, he adds.
US anxiety
The security situation in northern and eastern Afghanistan has deteriorated since the beginning of the year.
An insecure security situation has been exacerbated by political uncertainty
The instability has been exacerbated by political uncertainty created by August's presidential poll, which has been marred by widespread fraud allegations.
The commander of the more than 100,000 Nato and US forces in the country, US Gen Stanley McChrystal, has described the situation as "serious" and is believed to have requested up to 40,000 additional troops.
But US President Barack Obama - who has already sent thousands of extra troops to the country - says strategy in Afghanistan must be agreed before a decision can be made on whether to bolster military forces further
