The Treasury and the Department for Culture, Media and Sport (DCMS) are discussing a price of £9bn - up from an initial figure of £2.35bn.
Construction alone could cost £3.3bn, the government has said.
The DCMS said budget talks were still ongoing and accurate figures would be announced later.
Costs have mounted since London won the right to host the games in July 2005.
On top of soaring construction bills, an extra £2bn has been allocated as a contingency fund.
Regeneration costs of £1.8bn and a £1bn VAT bill have also been added.
And security costs have grown to at least £900m.
The Shadow Culture Secretary, Hugo Swire, said the news was "staggering", and said Chancellor Gordon Brown should take responsibility, because he signed off the original budget.
"There is no doubt that there will be real worries across the country about the huge increase in costs... it is time we got a proper explanation from Gordon Brown on what has gone on and assurances as to what the government is doing to control the ever-increasing budget," he said.
There has been some suggestion that the Treasury and the DCMS have clashed over the budget, but the DCMS has denied this.
"This is just the latest of many different figures to be quoted by the media in the last few months, claiming to be the cost of the 2012 games," a DCMS spokesman said.
"Discussions are going on across government to resolve outstanding issues and we will make an announcement about long-term budget figures when they have concluded.
"It is completely incorrect to suggest government departments are at odds on this issue. This is simply not the case."
In the wake of heavy criticism of the government's handling of the games, Culture Secretary Tessa Jowell admitted to parliament in November that the cost of the Games would rise by £900m - 40% - to £3.3bn.
But critics have continued to express wide-ranging concerns over everything from transport to the burden on the tax payer and the effect on the National Lottery's funding of good causes.
The Treasury has insisted a 60% contingency should be to be added to the construction cost.
But this figure has been resisted both by the Mayor of London Ken Livingstone and also the DCMS.
Both of them would like a much lower contingency, a view that appears to have support from CLM, the Delivery Partners.
A variety of other factors are also at play.
These include rise in commodity prices, adjustments to transport figures to reflect 2012 prices and a revised estimate for inflation on construction costs.
The land in the east end of London chosen for the site also needs decontamination and major remedial work before it can be fit for the games.
The Treasury has also decided that the Olympic Development Authority will have to pay VAT.
While VAT is in effect paid to the Treasury, the cash initially still has to be found before it is reclaimed.
The chairman of the CMS Select Committee, John Whittingdale, said it was important the British taxpayer was not asked to write a "blank cheque" for the Olympics.
He said: "I think the total is going to be a lot bigger than was initially suggested, and what is important, I think, is that the government comes out and gives a clear statement so that we can actually have some facts rather than continue to have figures bandied about which seem to rise by the week."
BBCLast Mod: 00 0000, 00:00