The Alphajets training plane crashed in late afternoon on Wednesday into a building housing a reception center for disabled people, killing one person and leaving at least three injured, according to local police.
The pilot and co-pilot managed to eject and were recovered by a rescue team.
The accident was due to "technical problem" Le Drian told French daily Le Figaro.
He said that the Alphajets plane belonged to the French Air Force's air base 705 in Tours and was on a training mission when it went down.
An experienced pilot was in control with a student sitting in the back, he added.
An investigation has been launched by military experts.