Protoaurignacian teeth
Upper Palaeolithic modern human infant skull from Sungir (left) and the Neandertal infant skull of Roc de Marsal (right). Shara E. Bailey, Center for the Study of Human Origins, Department of Anthropology, New York University, New York, USA

A longstanding debate about a mystery species living in Europe 42,000 years ago has been settled, with DNA evidence from a tooth showing it was that of a modern human, not Neanderthal.

The Proto-Aurignacian culture, known for its simple ornaments and bladelets, is known to have lived in southern Europe at this time. Evidence of which species it belonged to, however, has remained elusive – fossilised bones have never been found meaning scientists could not identify them.

However, researchers from the University of Bologna have now analysed one of two teeth samples found at excavation sites in Italy to show they were modern humans – and that they likely led to the extinction of Neanderthals in the area.

The study, published in the journal Science, notes that the Proto-Aurignacian culture first appeared about 42,000 years ago. The teeth dated to 41,000 years ago.

Neanderthals disappeared from Western Europe by about 39,260 years ago, while modern humans are believed to have arrived between 45,000 and 40,000 years ago.

Protoaurignacian teeth mystery species
Three-dimensional digital models of the lower deciduous incisor from Riparo Bombrini (left) and of the upper deciduous incisor from Grotta di Fumane. Daniele Panetta, CNR Institute of Clinical Physiology, Pisa, Italy

One of the teeth contained mitochondrial DNA and the scientists compared this with present-day humans, ancient modern humans, Neanderthals, Denisovans, a hominin from Spain and a chimpanzee. Findings showed it belonged to a modern human.

Study author Stefano Benazzi told IBTimes UK that the artefacts from the culture were "quite particular" with lots of innovations, suggesting there was something strange about it: "Most people thought they were modern humans but we didn't know ... there were no human remains. There was never proof so there were lots of debates about if they were Neanderthals or if they were modern humans because they appear quite sophisticated."

Because the DNA shows they were modern humans, this means they were living in Europe with Neanderthals 42,000 years ago – and that they probably killed them off: "It suggests modern humans were the cause of Neanderthal extinction. We don't know if it was direct or indirect. Direct might be competition for food or direct killing we don't know about that."

Protoaurignacian excavation site
Bombrini Shelter during the 2005 excavation. Fabio Negrino, Dipartimento di Antichità, Filosofia, Storia e Geografia, Università di Genova, Genova, Italy

Benazzi said there is evidence in other parts of the world where the two species overlapped of interbreeding – an avenue he thinks would be interesting to explore further.

"We couldn't find evidence of interbreeding. But maybe if in the future we found something like a bone or tooth dating to that period we could extract DNA that we could see if there was interbreeding in Europe ... [We are] continuing to excavate and maybe we will be able to find something else, but the chances are really low. Obviously we would like to find something, but we don't know."

He said that while the finding help with our understanding of modern man's entry into Europe, there is still far more to learn: "Let's say it's a narrowing of a puzzling window that developed between 45,000 and 40,000 years ago. It's not closed but we're getting closer."