Opposition candidate Maithripala Sirisena, a one-time ally of Rajapaksa who defected in November and derailed what the president thought would be an easy win, took 51.3 percent of the votes polled in Thursday's election.
Rajapaksa got 47.6 percent, according to the Election Department.
Celebratory firecrackers were set off in the capital, Colombo, after Rajapaksa accepted the victory of Sirisena, who has vowed to root out corruption and bring constitutional reforms to weaken the power of the presidency.
Sri Lanka's stock market climbed to its highest in nearly four years.
"We expect a life without fear," said Fathima Farhana, a 27-year-old Muslim woman in Colombo.
"I voted for him because he said he will create equal opportunities for all," she said of Sirisena, a soft-spoken 63-year-old from the rice-growing hinterlands of the Indian Ocean island state.
Like Rajapaksa, Sirisena is from the majority Sinhala Buddhist community but he has reached out to ethnic minority Tamils and Muslims and has the support of several small parties.
His allies say he will rebalance the country's foreign policy, which tilted heavily towards China in recent years as Rajapaksa fell out with the West over human rights and allegations of war crimes committed at the end of a drawn-out conflict with Tamil separatists in 2009.
U.S. Secretary of State John Kerry was quick to welcome the successful election and commended Rajapaksa for accepting the verdict of the nation's 15 million voters.
"I look forward to working with President-elect Sirisena as his new government works to implement its campaign platform of a Sri Lanka that is peaceful, inclusive, democratic, and prosperous," Kerry said in a statement.
Indian Prime Minister Narendra Modi telephoned Sirisena to congratulate the new leader of "a close friend and neighbour".
Sri Lanka is just off India's southern coast and has historically had mixed ties with its much larger neighbour. Rajapaksa had cold-shouldered New Delhi in recent years but Sirisena told an Indian newspaper this week that "we will revert to the old, non-aligned policy".
Sirisena is expected to be sworn in at Colombo's Independence Square at 6:00 p.m. (1230 GMT).
The results showed Rajapaksa remained popular among Sinhala Buddhists, who account for about 70 percent of the country's 21 million people, but Sirisena earned his lead with the support of the ethnic Tamil-dominated former war zone in the north and Muslim-dominated areas.
Rajapaksa won handsomely in the last election in 2010, surfing a wave of popularity months after the defeat of the Tamil Tiger rebels.
But critics say he had become increasingly authoritarian, with several members of his family holding powerful positions. Although the economy had blossomed since the end of the war, voters complained of the high cost of living.
Rajapaksa had called this election two years early, confident that the usually fractured opposition would fail to come up with a credible candidate. But he did not anticipate the emergence of Sirisena, who shared a traditional Sri Lankan dinner with him one evening and turned on him the next day.
Sirisena will lead a motley coalition of ethnic, religious, Marxist and centre-right parties, which analysts say could hamper economic reform and encourage populist policies.
He has pledged to abolish the executive presidency that gave Rajapaksa unprecedented power and hold a fresh parliamentary election within 100 days.
He has also promised a crackdown on corruption, which would include investigations into big infrastructure projects such as a $1.5 billion deal with China Communications Construction Co Ltd to build a port city.
It is not clear if the port, to be built on land reclaimed from the sea in Colombo, will be cancelled.
However, Sirisena's backers have said a casino licence given to Australian gambling tycoon James Packer's Crown Resorts Ltd will be withdrawn.
"VIOLATIONS"Sri Lanka's poll chief allayed fears about army interference in the country's close-fought presidential election Thursday.
The election commissioner, Mahinda Deshapriya, assured that inquiries had already been made and that there had been no major violation of elections laws.
He did admit to receiving complaints about security forces being deployed in the Wanni region, which had been the stronghold of separatist rebels from the Tamil minority during the country's decades long civil war, which ended in 2009.
A senior official speaking to The Anadolu Agency said that the poll chief has already verified the allegations. "The army commander has assured that this was not done," the official said.
The polls chief has said the army commander gave a firm assurance that troops had only been placed on stand-by and the election process was supported by the police, with 65,000 armed policemen deployed island-wide with instructions to "shoot in the head" in the event of any violations.
According to the Colombo-based Centre for Monitoring Election Violence, there had been over 400 incidents of election law violations in the island since the announcement of the poll on Nov. 20. The highest number of incidents has been recorded from the northern district of Jaffna.
Election monitors have expressed concern over Tamils and Muslims from the northern and eastern regions being prevented from voting, given the strong likelihood of them supporting the common presidential candidate, Maithripala Sirisena.
The voting was closed at by 4 p.m. local time with apprehensions about the integrity of the process in former war zones which are still to recover completely from the impact of 27 years of civil strife and remained heavily militarized.
On Wednesday, the first poll-related death was recorded when one of three people injured while decorating the election platform of Maithripala Sirisena, the opposition candidate, succumbed to injuries.
According to the island’s elections secretariat, the capital, Colombo, recorded the highest voter turnout at 75 percent. In the north, former conflict-affected districts of Wanni and Jaffna recorded 70 and 65 percent voter turnout, respectively.