The sultan died after “a wise and triumphant march rich with generosity that embraced Oman and extended to the Arab, Muslim and entire world and achieved a balanced policy that the whole world respected”, the state-run Oman News Agency said.
His cause of death was not disclosed.
There had been concerns over his health in recent weeks and he had reportedly been in Belgium for treatment, but traveled back to Oman shortly before the new year.
Oman, on the eastern edge of the Arabian Peninsula and a key Western ally, has announced a three-day period of national mourning.
Sultan Qaboos was the longest-serving leader in the Middle East, having ruled the country since 1970 after taking over the leadership from his father in a peaceful coup.
Born in Oman in 1940, he traveled to the UK in 1958 to attend Sandhurst and joined the British Army where he was posted to the 1st Battalion The Cameronians (Scottish Rifles) and served in Germany for a year.
Sultan Qaboos had no children, and had not publicly appointed a successor. However, he will have secretly recorded his choice in a sealed letter should the royal family disagree on the succession line.
The royal family will now have three days to appoint a successor, according to a statute from 1996.
If they family fail to do so, a council of military and security officials, supreme court chiefs and heads of two assemblies will put in power the person appointed by the sultan in his letter.
Analysts worry about royal family discord and a resurgence of tribal rivalries and political instability, now a new ruler has to be chosen.