Talk about a holiday in the Gulf and everyone thinks of Dubai – understandably so, for the Emirate is by far the most popular destination in the region. But if you want to see real Arabia, as opposed to a mass of shopping malls and luxury hotels, there is a far better option: Oman.

Oman – or Muscat and Oman as it used to be called – is a sizeable country, not a tiny strip of desert. It can offer most of the things that Dubai can, save the excess, and it does so with self-confidence, grace and charm.

Example: we were in the capital, Muscat, standing in the car park of a shopping complex one evening, after a very good dinner in the courtyard in the Kargeen Caffe. We were looking for a cab to go back to the hotel.

Two young women came up and asked whether they could help. We explained the problem. Oh, one said, we'll drive you. Well, at that moment a cab appeared so we declined, but you see the point: this is a country where people welcome foreigners and where women are comfortable going up to strangers to offer help.

There were many other small courtesies, of people going out of their way to show us things, explain aspects of the society, or simply try to make sure we had a good time. It was a glimpse of the deep courtesy towards strangers that is embedded in Arab culture. There is, it was explained to us, a reason for such empathy. Until a few years ago, when oil wealth began to spread through the nation, life for most people was very tough. So if you were travelling and got into difficulties your very survival might depend on strangers. Society only worked if everybody helped each other – and they still do, even for a problem as mundane as finding a cab.

Oman runs from the Straits of Hormuz, with a brief pause for a patch of UAE territory, around the corner of the Arabian Sea, and all the way down to the Yemen border. Its coastline, scuba divers please note, measures more than 1,000 miles. In land area it is about the size of Britain, but only 2.5 million people live here. Muscat, the capital, is at the top right-hand corner and has for centuries had great strategic importance, controlling entry to the Gulf; little wonder the UK has long sought to be on good terms with the Sultan. It is also strategically significant for migrating birds, and is "the eagle capital of the world" – of which more in a moment.

The obvious place to start is Muscat. It is a sprawl, mostly built in the past 10 or 20 years and with a curious feeling of Los Angeles about it: malls, villas and freeways.

Muscat can do luxury if you want it. We stayed on the coast at The Chedi, a combination of designer chic and whatever-you-want-in-the-world service, expensive but not absurdly so for what you get. Some guests spent all their time being pampered there, but to get a feeling for this ancient trading society, go to the two old ports: Old Muscat and Muttrah.

Old Muscat used to be a trading port but now is most notable for being the location of the palace of the ruler, Sultan Qaboos Bin Said Al-Said. By palatial standards it is not particularly grand, but the process of extending it and the other public buildings, and landscaping them, has meant that much of the old town has been knocked down. Nearby Muttrah has a charming Mediterranean-style waterfront and a harbour now used mainly by cruise liners. It also has a large souk where ranks of tailors will run up a suit for you, plus a fun-fair. Muttrah's most spectacular feature, however, was floating in the harbour. This is the home-port for the Sultan's main yacht. "Yacht" seems an inadequate word as the Al Said is the size of a cross-channel ferry. In a league where size matters it is, at 508ft, the second-longest private yacht in the world. Look out to sea and you know who is running the show.

So what else, aside from going to the souk and ogling the Sultan's yacht, do you do in Muscat? Well, if you are interested in the country itself, try the Bait Adam museum, a private house that serves as a repository for the diversity of Omani culture, from coins to old prints. You can join the family for a meal as well – just book in advance for supper. The more formal Oman Museum covers the nation's culture from Bedouin traditions to colonialism.

Food can be very cheap or very good, and sometimes both. The usual Middle Eastern staples are available: lamb or chicken shawarma, falafel and Lebanese-style salads, eggs, cheese and couscous. In addition, Indian food is ubiquitous. On the coast, good, fresh fish is available.

To share the dazzling blue water that flanks Oman, there are several suitable operators along the coast, including the Oman Dive Centre, where we booked a snorkelling expedition. If you feel so inclined you can also "wadi-bash", driving in 4x4s along dry river beds. You also can visit the new mosque paid for by the Sultan. But the highlight of our trip was none of these. It was going to the rubbish dump.

If you are a vulture or an eagle, a rubbish dump is where you go for lunch. So we persuaded a taxi-driver to take us in his pristine Cadillac to the town dump. The first one was bird-less, but the driver's cousin happened to work for the municipal cleansing department. By phone, he told us that we had to go to another one 20 miles out of town. Amid grey mounds of dusty rocks we were able to see Egyptian vultures and steppe eagles. Most impressive, though, was a pair of lappet-faced vultures, which have a wingspan of 10ft and are the largest birds in Oman. The by now very dusty Cadillac then took us back to our own lunch at The Chedi.

Muscat is rewarding, but it is worth getting a glimpse of the rest of the country, because the interior of Oman is fascinating. Inland, there are ruined medieval mud villages and forts, many surrounded by date palm plantations.

You can see part of the "empty quarter", the world's largest sandy desert, which Oman shares with Saudi Arabia and Yemen, or visit the site of the Queen of Sheba's palace, where she lived 3,000 years ago.

To witness how man's dreams can decay, seek out the old ghost town of Tanuf and the fascinating ruined city of Manah. The roads into the mountains from here are constantly being improved, making high-altitude hikes a possibility. Serious hikers will want to try the Jebel Shams rim walk: the "mountain of the sun" is Oman's highest at nearly 10,000ft.

We also visited Nizwa, a couple of hours' drive inland from Muscat, via the high, rocky plateau between it and the sea. Nizwa is an oasis, as are all the towns of the interior, since without water no community can survive. It is the main city of the interior and a former capital of the whole country.

It has one of the largest forts in Oman, and a big, new souq. We went to the weekly market and watched sheep and goats being paraded round the ring before sale. The most notable sight was a Bedouin woman in a burka squatting down to test the udders of a goat and rejecting it as not a promising milker. If you do the milking you know what you are looking for.

We also went to several smaller towns on the Jebel Akhdar mountains. The range goes up to 10,000ft but the towns are on a plateau at around 6,000ft, with each dependent on a spring and the tiny amount of rain they receive each year. The spring water is distributed through each town by a network of channels, carefully organised so that each family had access to its share of the water. The astounding thing is the way a trickle of water, used carefully, can support a whole village. If that is a lesson in conservation for us all, Oman has I think a wider message. This is a culture to respect and admire – real Arabia, not the Disneyland version.